fix: unify runtime readiness publication and graceful shutdown flow (#3087)

* fix(server): unify runtime readiness and shutdown flow

* refactor(server): decouple readiness shared types

* refactor(server): keep readiness types crate-private

* refactor(server): await protocol shutdown handles

* fix(server): bypass cached startup readiness

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,22 +1,32 @@
 ---
 name: rustfs-release-version-bump
-description: Prepare a RustFS release branch like PR #2957 by bumping versioned files, aligning release assets, running required verification, and finishing with commit, push, and gh PR creation. Use when publishing a new RustFS alpha, beta, or stable release.
+description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
 ---
-
 # RustFS Release Version Bump
 
-Use this skill when the task is to prepare a new RustFS version release branch following the pattern validated in PR `#2957`.
+Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
 
-## Read first
+Validated baseline: release pattern used in PR `#2957`.
 
-- Read `AGENTS.md`.
-- Read `.github/pull_request_template.md`.
-- Inspect the current branch diff against `origin/main`.
-- Do not assume every release file should change in the same way every time.
+## Required inputs
 
-## Scope validated by PR #2957
+- Exact target version, for example `1.0.0-beta.4`.
+- Delivery scope:
+- Local only (`edit/verify`).
+- Local + git (`commit/push`).
+- Full GitHub flow (`commit/push/PR`).
 
-The `1.0.0-beta.3` release branch updated these files:
+If target version is missing or ambiguous, stop and ask before editing.
+
+## Read before editing
+
+- `AGENTS.md` (root and nearest path-specific files).
+- `.github/pull_request_template.md`.
+- Current branch status and diff against `origin/main`.
+
+## Default release file scope
+
+Treat the following file list as the default checklist for each release bump:
 
 - `Cargo.toml`
 - `Cargo.lock`
@@ -26,78 +36,89 @@ The `1.0.0-beta.3` release branch updated these files:
 - `helm/rustfs/Chart.yaml`
 - `rustfs.spec`
 
-Treat this file list as the default checklist for future release bumps. Only drop a file if the repository's current release pattern clearly says it is no longer part of the publish flow.
+Only drop a file when the current repository release process clearly no longer requires it.
 
-## Workflow
+## Hard release policy
 
-1. Confirm release intent
-- Identify the target version string exactly, for example `1.0.0-beta.4`.
-- Check whether the user wants only local commit preparation or the full `commit + push + PR` flow.
-- Compare the branch against `origin/main` and isolate only release-related edits.
-- If the target version is not explicit, stop and ask for it before editing.
+- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
+- Helm chart version mapping follows `beta.N -> 0.N.0`.
+- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
+- Do not change these rules without explicit confirmation.
 
-2. Update core Rust workspace versions
-- Bump the workspace version in `Cargo.toml`.
-- Bump all internal workspace crate versions in `Cargo.toml`.
-- Ensure `Cargo.lock` reflects the same release version for workspace packages.
-- Re-read the diff and verify there are no partial version leftovers.
+## Step-by-step workflow
 
-3. Align release assets
-- Update versioned Docker examples in `README.md` and `README_ZH.md` using the `<version>` tag form, for example `rustfs/rustfs:1.0.0-beta.4`.
-- Update `flake.nix` package version.
-- Update `helm/rustfs/Chart.yaml` `appVersion`.
-- Update `helm/rustfs/Chart.yaml` `version` using the release-chart rule:
+1. Confirm intent and isolate scope
+- Confirm target version string exactly.
+- Confirm whether user requested local-only or full GitHub flow.
+- Inspect current branch and ensure only release-related files are touched for this task.
+
+2. Update workspace versions
+- Bump `[workspace.package].version` in `Cargo.toml`.
+- Bump internal workspace crate dependency versions in `Cargo.toml`.
+- Update `Cargo.lock` so workspace package versions match target version.
+- Re-scan for partial leftovers.
+
+3. Update release assets
+- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
+- `flake.nix`: update package version to target version.
+- `helm/rustfs/Chart.yaml`:
+- `appVersion` = target version.
+- `version` follows chart mapping rule, for example:
 - `1.0.0-beta.3` -> `0.3.0`
 - `1.0.0-beta.4` -> `0.4.0`
-- Follow the same pattern for later beta releases unless the repository rule changes.
-- Update `rustfs.spec` release metadata and changelog entry.
-- Set `rustfs.spec` `Release` to the prerelease suffix without the base version, for example `beta.3` for `1.0.0-beta.3`.
-- Keep the release asset changes in a separate commit from the core Rust workspace version bump when the branch contains both.
+- `rustfs.spec`:
+- Set `Release` to prerelease suffix (example `beta.4`).
+- Add/update top changelog entry with exact format:
+- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
+- `- Update RPM package to RustFS 1.0.0-beta.4`
+- Changelog identity and time must come from current environment:
+- `git config --get user.name`
+- `git config --get user.email`
+- `date '+%a %b %d %Y'`
+- Changelog version text must match target release version exactly.
 
-4. Stop and discuss before changing release policy
-- Ask before changing the established Docker tag style away from `<version>`.
-- Ask before changing the established Helm chart version mapping away from `beta.N -> 0.N.0`.
-- Ask before changing the established `rustfs.spec` `Release` rule away from the release suffix form such as `beta.N`.
-- Ask before widening the release scope beyond the files already validated in PR `#2957`.
+4. Verify before shipping
+- Run:
+- `cargo fmt --all`
+- `cargo fmt --all --check`
+- `make pre-commit`
+- If verification passes, run `cargo clean`.
+- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
 
-5. Verify before shipping
-- Run `make pre-commit`.
-- If verification succeeds, run `cargo clean` to remove generated build artifacts before wrapping up.
-- If `make pre-commit` fails, stop and return `BLOCKED`.
+5. Commit strategy
+- Preferred split when both parts changed:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
+- `chore(release): align release assets for <version>` for docs and packaging files.
+- If user asks for one commit, use one commit.
+- Stage only intended release files; do not include unrelated working tree changes.
 
-6. Commit structure
-- Prefer two commits when the change naturally splits:
-- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`
-- `chore(release): align release assets for <version>` for docs and packaging metadata
-- If the user asks for a single commit, follow that request.
+6. Push and PR
+- Push branch:
+- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
+- Create PR with template headings unchanged:
+- `gh pr create --base main --head <branch> --title ... --body-file ...`
+- PR title/body must be English.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands and any `BLOCKED` reason clearly.
 
-7. Push and PR
-- Push the release branch with `git push -u origin <branch>` or `git push` if upstream already exists.
-- Create the PR with `gh pr create --base main --head <branch> --title ... --body-file ...`.
-- Keep the PR title and body in English.
-- Keep the `.github/pull_request_template.md` headings exactly.
-
-## Ready-to-check commands
+## Recommended check commands
 
+- `git status --short --branch`
 - `git diff --name-only origin/main...HEAD`
 - `git diff --stat origin/main...HEAD`
+- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
 - `cargo clean`
-- `git status --short --branch`
 
-## Output expectations
+## Output contract
 
-When using this skill, return:
+When using this skill, always report:
 
-- The files changed for the release bump
-- Any uncertainty that needs user confirmation before editing
-- Verification status
-- Commit messages used
-- Push status and PR URL when the GitHub flow was requested
-
-## Established release policy
-
-- Docs use Docker tags in `<version>` form, not `v<version>`.
-- `helm/rustfs/Chart.yaml` `version` follows `beta.N -> 0.N.0` based on the current release policy.
-- `rustfs.spec` `Release` follows the release suffix form, for example `beta.3`.
-- If any of these rules need to change in the future, pause and confirm before editing.
+- Target version.
+- Files changed.
+- Any assumptions or uncertainties requiring confirmation.
+- Verification result (`PASSED` or `BLOCKED`) with key evidence.
+- Commit message(s) used.
+- Push status and PR URL when GitHub flow is requested.

.agents/skills/security-advisory-lessons/SKILL.md

@@ -91,7 +91,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
 - Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
 - Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
-- License/version-like metadata endpoints should expose only coarse public data unless authenticated.
+- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
 
 ### Profiling, debug, and health endpoints
 - Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -20,7 +20,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
 - `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
 - `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-- `GHSA-x5xv-223c-8vm7`: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
+- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
 
 ### IAM import, service accounts, and privilege boundaries
 

.config/make/help.mak

@@ -4,7 +4,7 @@
 .PHONY: help
 help: ## Shows This Help Menu
 	echo -e "$$HEADER"
-	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} ; {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}' | sed -e 's/\[36m##/\n[32m##/'
+	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
 
 .PHONY: help-build
 help-build: ## Shows RustFS build help

.docker/compose/README.md

@@ -54,6 +54,46 @@ Strict mode is available when you explicitly want `/health/ready == 200` as the
 WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
 ```
 
+### Profiling + Trace Validation
+
+The profiling-focused 4-node compose keeps profiling enabled and points RustFS
+to an OTLP/HTTP collector endpoint:
+
+```bash
+docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
+```
+
+Important behavior notes:
+
+- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
+  traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
+- Startup usually produces logs and metrics first. That does not guarantee
+  visible traces yet.
+- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
+- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
+  many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
+  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
+
+Minimal trace verification flow:
+
+```bash
+# 1. Start the profiling compose with richer span visibility.
+RUSTFS_OBS_LOGGER_LEVEL=debug \
+docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
+
+# 2. Generate real request traffic after startup.
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 3. Then inspect Tempo or Jaeger.
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+If logs and metrics are present but traces are sparse, the most common cause is
+"no real request traffic yet" or "`info` level filtered nested spans", not an
+OTLP routing failure.
+
 ### (Deprecated) Minimal Observability
 
 ```bash

.docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml

@@ -0,0 +1,236 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Profiling-first 4-node local-build compose.
+#
+# Goals:
+# - force linux/amd64 runtime/build on Apple Silicon hosts;
+# - enable RustFS built-in CPU profiling;
+# - keep all tuning knobs host-overridable via env.
+#
+# Observability notes:
+# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
+#   `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
+# - Logs and metrics usually appear during startup. Traces mainly appear after
+#   real HTTP/S3/gRPC requests create spans.
+# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
+#   filters many `debug`-level nested spans. Use `debug` when validating trace
+#   richness rather than collector reachability.
+
+services:
+  node1:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node1_data_0:/data/rustfs0
+      - node1_data_1:/data/rustfs1
+      - node1_data_2:/data/rustfs2
+      - node1_data_3:/data/rustfs3
+    ports:
+      - "9000:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node2:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node2_data_0:/data/rustfs0
+      - node2_data_1:/data/rustfs1
+      - node2_data_2:/data/rustfs2
+      - node2_data_3:/data/rustfs3
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node3:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node3_data_0:/data/rustfs0
+      - node3_data_1:/data/rustfs1
+      - node3_data_2:/data/rustfs2
+      - node3_data_3:/data/rustfs3
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node4:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node4
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node4_data_0:/data/rustfs0
+      - node4_data_1:/data/rustfs1
+      - node4_data_2:/data/rustfs2
+      - node4_data_3:/data/rustfs3
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-cluster-net
+
+volumes:
+  node1_data_0:
+  node1_data_1:
+  node1_data_2:
+  node1_data_3:
+  node2_data_0:
+  node2_data_1:
+  node2_data_2:
+  node2_data_3:
+  node3_data_0:
+  node3_data_1:
+  node3_data_2:
+  node3_data_3:
+  node4_data_0:
+  node4_data_1:
+  node4_data_2:
+  node4_data_3:
+
+networks:
+  rustfs-cluster-net:
+    driver: bridge

.docker/compose/docker-compose.cluster.local-build.yml

@@ -20,13 +20,16 @@ services:
       dockerfile: Dockerfile.source
     hostname: node1
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -47,13 +50,16 @@ services:
       dockerfile: Dockerfile.source
     hostname: node2
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -74,13 +80,16 @@ services:
       dockerfile: Dockerfile.source
     hostname: node3
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -101,13 +110,16 @@ services:
       dockerfile: Dockerfile.source
     hostname: node4
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-latency-stable.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-throughput-max.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60

.docker/compose/docker-compose.cluster.perf-round1-linux.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40

.docker/compose/docker-compose.cluster.perf-round1.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30

.docker/compose/docker-compose.cluster.yaml

@@ -21,8 +21,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
+      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
     platform: linux/amd64
     ports:
       - "9000:9000" # Map port 9001 of the host to port 9000 of the container
@@ -38,8 +38,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
+      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
     platform: linux/amd64
     ports:
       - "9001:9000" # Map port 9002 of the host to port 9000 of the container
@@ -55,8 +55,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
+      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
     platform: linux/amd64
     ports:
       - "9002:9000" # Map port 9003 of the host to port 9000 of the container
@@ -72,8 +72,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
+      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
     platform: linux/amd64
     ports:
       - "9003:9000" # Map port 9004 of the host to port 9000 of the container

.docker/observability/README.md

@@ -90,6 +90,54 @@ docker compose down -v
 - **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
 - **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
 
+### Verifying RustFS Traces
+
+When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
+OTLP/HTTP base URL, for example:
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+RustFS automatically expands that base URL to:
+
+- `/v1/traces`
+- `/v1/metrics`
+- `/v1/logs`
+
+Important behavior notes:
+
+- Logs and metrics usually appear during startup, so seeing those two signals
+  first is expected.
+- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
+  startup, because request-path spans are created on demand.
+- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
+  many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
+  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
+
+Minimal validation flow:
+
+```bash
+# 1. Start this observability stack.
+docker compose up -d
+
+# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+export RUSTFS_OBS_LOGGER_LEVEL=debug
+
+# 3. Generate real request traffic.
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 4. Inspect Grafana or Jaeger.
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+If logs and metrics are present but traces are sparse, the most common cause is
+"no real request traffic yet" or "`info` level filtered nested spans", not an
+OTLP routing failure.
+
 ## Troubleshooting
 
 - **Service Health**: Check the health of services using `docker compose ps`.

.docker/observability/README_ZH.md

@@ -90,6 +90,50 @@ docker compose down -v
 -   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
 -   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
 
+### 验证 RustFS Trace
+
+当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
+OTLP/HTTP 的基础 URL，例如：
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+RustFS 会自动在该基础 URL 后补全：
+
+- `/v1/traces`
+- `/v1/metrics`
+- `/v1/logs`
+
+需要注意：
+
+- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
+- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
+- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
+  如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
+
+最小验证流程：
+
+```bash
+# 1. 启动本目录下的可观测性技术栈。
+docker compose up -d
+
+# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+export RUSTFS_OBS_LOGGER_LEVEL=debug
+
+# 3. 产生真实请求流量。
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 4. 到 Grafana 或 Jaeger 中检查。
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
+“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
+
 ## 故障排除
 
 -   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。

.docker/observability/grafana/dashboards/rustfs.json

@@ -9037,13 +9037,13 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "{__name__=~\"rustfs_(notification|audit)_.*\",job=~\"$job\"}",
+          "expr": "{__name__=~\"rustfs_(notification|audit|log_chain)_.*\",job=~\"$job\"}",
           "legendFormat": "{{__name__}}",
           "range": true,
           "refId": "A"
         }
       ],
-      "title": "Notification / Audit (All)",
+      "title": "Notification / Audit / LogChain (All)",
       "type": "timeseries"
     },
     {

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -38,3 +38,16 @@ groups:
         expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
       - record: rustfs:scanner_cycles_success:rate5m
         expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
+
+      - record: rustfs:log_chain_op_event_mismatch:rate5m
+        expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
+
+      - alert: RustFSLogChainOpEventMismatchDetected
+        expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
+        for: 10m
+        labels:
+          severity: warning
+          component: s3-log-chain
+        annotations:
+          summary: "RustFS log-chain op/event mismatch detected"
+          description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."

.github/actions/setup/action.yml

@@ -53,18 +53,19 @@ runs:
           musl-tools \
           build-essential \
           pkg-config \
-          libssl-dev
+          libssl-dev \
+          protobuf-compiler
 
     - name: Install protoc
-      uses: arduino/setup-protoc@v3
+      uses: rustfs/setup-protoc@v3.0.1
       with:
-        version: "33.1"
-        repo-token: ${{ inputs.github-token }}
+        version: "34.1"
+        repo-token: ${{ github.token }}
 
     - name: Install flatc
-      uses: Nugine/setup-flatc@v1.2.4
+      uses: Nugine/setup-flatc@v1
       with:
-        version: "25.9.23"
+        version: "25.12.19"
 
     - name: Install Rust toolchain
       uses: dtolnay/rust-toolchain@stable

.github/workflows/audit.yml

@@ -75,7 +75,7 @@ jobs:
         uses: actions/checkout@v6
 
       - name: Dependency Review
-        uses: actions/dependency-review-action@v4
+        uses: actions/dependency-review-action@v5
         with:
           fail-on-severity: moderate
-          comment-summary-in-pr: true
+          comment-summary-in-pr: always

.github/workflows/build.yml

@@ -169,7 +169,7 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-15-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -234,8 +234,7 @@ jobs:
         run: |
           mkdir -p ./rustfs/static
           if [[ "${{ matrix.platform }}" == "windows" ]]; then
-            curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300
-            if [[ $? -eq 0 ]]; then
+            if curl.exe --fail -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300; then
               unzip -o console.zip -d ./rustfs/static
               rm console.zip
             else
@@ -244,9 +243,8 @@ jobs:
             fi
           else
             chmod +w ./rustfs/static/LICENSE || true
-            curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-              -o console.zip --retry 3 --retry-delay 5 --max-time 300
-            if [[ $? -eq 0 ]]; then
+            if curl --fail -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
+              -o console.zip --retry 3 --retry-delay 5 --max-time 300; then
               unzip -o console.zip -d ./rustfs/static
               rm console.zip
             else

.github/workflows/ci.yml

@@ -204,7 +204,7 @@ jobs:
         with:
           tool: s3s-e2e
           git: https://github.com/s3s-project/s3s.git
-          rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
+          rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
 
       - name: Run end-to-end tests
         run: |

.github/workflows/e2e-s3tests.yml

@@ -40,8 +40,8 @@ on:
 
 env:
   # main user
-  S3_ACCESS_KEY: rustfsadmin
-  S3_SECRET_KEY: rustfsadmin
+  S3_ACCESS_KEY: rustfs-ci-admin
+  S3_SECRET_KEY: rustfs-ci-secret
   # alt user (must be different from main for many s3-tests)
   S3_ALT_ACCESS_KEY: rustfsalt
   S3_ALT_SECRET_KEY: rustfsalt

.github/workflows/nix-flake-update.yml

@@ -31,7 +31,9 @@ jobs:
   update-flake:
     name: Update flake.lock
     runs-on: ubuntu-latest
-    timeout-minutes: 45
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -39,6 +41,9 @@ jobs:
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@v3
 
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@v3.20.0
+
       - name: Check Nix flake inputs
         uses: DeterminateSystems/flake-checker-action@v12
 
@@ -56,7 +61,7 @@ jobs:
             nix
             automated
           commit-msg: "chore(deps): update flake.lock"
-          pr-reviewers: houseme, overtrue, majinghe
+          pr-reviewers: overtrue, majinghe
           token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
 
       - name: Log PR details

.github/workflows/nix.yml

@@ -43,7 +43,9 @@ jobs:
   nix-validation:
     name: Nix Build & Check
     runs-on: ubuntu-latest
-    timeout-minutes: 45
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6

.github/workflows/performance.yml

@@ -86,7 +86,7 @@ jobs:
 
       - name: Build with profiling optimizations
         run: |
-          RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off" \
+          RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off --cfg tokio_unstable" \
           cargo +nightly build --profile profiling -p rustfs --bins
 
       - name: Run performance profiling

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 ## 📋 Code Quality Requirements
 
-For instructions on setting up and running the local development environment, please see [Development Guide](docs/DEVELOPMENT.md).
+This guide covers the local development environment and the checks expected before contributing.
 
 ### 🔧 Code Formatting Rules
 

Cargo.toml

@@ -15,10 +15,10 @@
 [workspace]
 members = [
     "rustfs", # Core file system implementation
-    "crates/appauth", # Application authentication and authorization
     "crates/audit", # Audit target management system with multi-target fan-out
     "crates/checksums", # client checksums
     "crates/common", # Shared utilities and data structures
+    "crates/data-usage", # Shared data usage models and algorithms
     "crates/config", # Configuration management
     "crates/credentials", # Credential management system
     "crates/crypto", # Cryptography and security features
@@ -39,15 +39,16 @@ members = [
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
-    "crates/s3-common", # Common utilities and data structures for S3 compatibility
+    "crates/s3-types", # S3 event type definitions
+    "crates/s3-ops", # S3 operation definitions and mapping
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
     "crates/scanner", # Scanner for data integrity checks and health monitoring
     "crates/signer", # client signer
     "crates/targets", # Target-specific configurations and utilities
     "crates/trusted-proxies", # Trusted proxies management
+    "crates/tls-runtime", # Project-wide TLS runtime foundation
     "crates/utils", # Utility functions and helpers
-    "crates/workers", # Worker thread pools and task scheduling
     "crates/io-metrics", # Zero-copy metrics collection for performance analysis
     "crates/io-core", # Zero-copy core reader and writer implementations
     "crates/zip", # ZIP file handling and compression
@@ -59,7 +60,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.95.0"
-version = "1.0.0-beta.3"
+version = "1.0.0-beta.4"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -76,46 +77,47 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.3" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.3" }
-rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.3" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.3" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.3" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.3" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.3" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.3" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.3" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.3" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.3" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.3" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.3" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.3" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.3" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.3" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.3" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.3" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.3" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.3" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.3" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.3" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.3" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.3" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.3" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.3" }
-rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.3" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.3" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.3" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.3" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.3" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.3" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.3" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.3" }
-rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.3" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.3" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.4" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.4" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.4" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.4" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.4" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.4" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.4" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.4" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.4" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.4" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.4" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.4" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.4" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.4" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.4" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.4" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.4" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.4" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.4" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.4" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.4" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.4" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.4" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.4" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.4" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.4" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.4" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.4" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.4" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.4" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.4" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.4" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.4" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.4" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.4" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.4" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.4" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "98d3d8067efdf97d3e93cdca7b9231753c904aca" }
+mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "2bad388283bc3ce48801fc2ffcd22445eb6f3d24" }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
@@ -126,7 +128,7 @@ futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
 pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
-lapin = { version = "4.7.2", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
+lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
 hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
@@ -141,11 +143,11 @@ tokio-rustls = { version = "0.26.4", default-features = false, features = ["logg
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
 tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic = { version = "0.14.6", features = ["gzip"] }
+tonic = { version = "0.14.6", features = ["gzip", "deflate"] }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3", features = ["timeout"] }
-tower-http = { version = "0.6.10", features = ["cors"] }
+tower-http = { version = "0.6.11", features = ["cors"] }
 
 # Serialization and Data Formats
 bytes = { version = "1.11.1", features = ["serde"] }
@@ -154,11 +156,11 @@ byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
 prost = "0.14.3"
-quick-xml = "0.39.4"
+quick-xml = "0.40.1"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
-serde_json = { version = "1.0.149", features = ["raw_value"] }
+serde_json = { version = "1.0.150", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -166,9 +168,9 @@ aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
 argon2 = { version = "0.6.0-rc.8" }
 blake2 = "0.11.0-rc.6"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
-crc-fast = "1.9.0"
+crc-fast = "1.10.0"
 hmac = { version = "0.13.0" }
-jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
+jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
 openidconnect = { version = "4.0", default-features = false }
 pbkdf2 = "0.13.0"
 rsa = { version = "0.10.0-rc.18" }
@@ -189,20 +191,19 @@ time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros
 # Database
 deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
 tokio-postgres = { version = "0.7", default-features = false, features = ["runtime", "with-serde_json-1"] }
-tokio-postgres-rustls = "0.13"
+tokio-postgres-rustls = "0.14.0"
 
 # Utilities and Tools
 anyhow = "1.0.102"
 arc-swap = "1.9.1"
-astral-tokio-tar = "0.6.1"
+astral-tokio-tar = "0.6.2"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.16" }
+aws-config = { version = "1.8.17" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-sdk-s3 = { version = "1.133.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.7" }
-backtrace = "0.3.76"
+aws-smithy-types = { version = "1.4.8" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
@@ -216,7 +217,7 @@ crossbeam-deque = "0.8.6"
 crossbeam-utils = "0.8.21"
 datafusion = "53.1.0"
 derive_builder = "0.20.2"
-enumset = "1.1.12"
+enumset = "1.1.13"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
@@ -230,7 +231,7 @@ ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
 libc = "0.2.186"
 libsystemd = "0.7.2"
-local-ip-address = "0.6.12"
+local-ip-address = "0.6.13"
 memmap2 = "0.9.10"
 lz4 = "1.28.1"
 matchit = "0.9.2"
@@ -254,12 +255,12 @@ rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
-rumqttc = { package = "rumqttc-next", version = "0.33.0", features = ["websocket"] }
+rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
 redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
-s3s = { git = "https://github.com/rustfs/s3s", rev = "a3b16608df35aaeed8fff08b4988d03f4ca9445b", features = ["minio"] }
+s3s = { git = "https://github.com/rustfs/s3s", rev = "507e1312b211c3ddc214b03875d6fabd15d22ed5", features = ["minio"] }
 serial_test = "3.4.0"
 shadow-rs = { version = "2.0.0", default-features = false }
 siphasher = "1.0.3"
@@ -267,9 +268,9 @@ smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
 snafu = "0.9.0"
 snap = "1.1.1"
-starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
+starshard = { version = "2.2.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.39.0"
+sysinfo = "0.39.2"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -277,7 +278,7 @@ thiserror = "2.0.18"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
-tracing-opentelemetry = "0.32.1"
+tracing-opentelemetry = { version = "0.33" }
 tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
@@ -292,23 +293,23 @@ zip = "8.6.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
-metrics = "0.24.5"
+metrics = "0.24.6"
 dial9-tokio-telemetry = "0.3"
-opentelemetry = { version = "0.31.0" }
-opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
-opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rustls"] }
-opentelemetry_sdk = { version = "0.31.0" }
-opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
-opentelemetry-stdout = { version = "0.31.0" }
-pyroscope = { version = "2.0.3", features = ["backend-pprof-rs"] }
+opentelemetry = { version = "0.32.0" }
+opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
+opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
+opentelemetry_sdk = { version = "0.32.0", features = ["rt-tokio"] }
+opentelemetry-semantic-conventions = { version = "0.32.0", features = ["semconv_experimental"] }
+opentelemetry-stdout = { version = "0.32.0" }
+pyroscope = { version = "2.0.4", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
 suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
-rcgen = "0.14.7"
-russh = "0.60.2"
-russh-sftp = "2.1.2"
+rcgen = "0.14.8"
+russh = { version = "0.60.3", git = "https://github.com/Eugeny/russh", rev = "fc6e3ab4cd4338e94ae64e17aeed2acee9335e6b" }
+russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -324,17 +325,11 @@ jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
 # pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
 # Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
-pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
+pprof = { package = "pprof-pyroscope-fork", version = "0.1500.4", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs"]
 
-[patch.crates-io]
-# Pinned to the simon-escapecode/russh fork carrying the upstream fix at
-# https://github.com/Eugeny/russh/pull/702. Drops out when a russh release
-# containing the fix is published.
-russh = { git = "https://github.com/simon-escapecode/russh", rev = "5cac2ed84945f9b80a52b673e058f2032bbe98ec" }
-
 [profile.release]
 opt-level = 3
 

Dockerfile.decommission-local

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM rust:1.93-trixie
+FROM rust:1.95-trixie
 
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \

Dockerfile.glibc

@@ -105,8 +105,6 @@ RUN groupadd -g 10001 rustfs && \
 
 ENV RUSTFS_ADDRESS=":9000" \
     RUSTFS_CONSOLE_ADDRESS=":9001" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
     RUSTFS_CONSOLE_ENABLE="true" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \

Dockerfile.source

@@ -31,7 +31,7 @@ ARG TARGETARCH
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.93-trixie AS builder
+FROM rust:1.95-trixie AS builder
 
 # Re-declare args after FROM
 ARG TARGETPLATFORM

README.md

@@ -115,7 +115,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command

README_ZH.md

@@ -1,4 +1,4 @@
-[![RustFS](https://github.com/user-attachments/assets/1b5afcd6-a2c3-47ff-8bc3-ce882b0ddca7)](https://rustfs.com.cn)
+[![RustFS](https://repository-images.githubusercontent.com/722597620/0fa936a2-8164-4f53-867f-def4beb64b21)](https://rustfs.com.cn)
 
 <p align="center">RustFS 是一个基于 Rust 构建的高性能分布式对象存储系统。</p>
 
@@ -113,7 +113,7 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
  # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：

build-rustfs.sh

@@ -217,7 +217,7 @@ setup_rust_environment() {
     # Set up environment variables for musl targets
     if [[ "$PLATFORM" == *"musl"* ]]; then
         print_message $YELLOW "Setting up environment for musl target..."
-        export RUSTFLAGS="'--cfg tokio_unstable -C target-feature=-crt-static'"
+        export RUSTFLAGS="--cfg tokio_unstable -C target-feature=-crt-static"
 
         # For cargo-zigbuild, set up additional environment variables
         if command -v cargo-zigbuild &> /dev/null; then

crates/appauth/README.md

@@ -1,37 +0,0 @@
-[![RustFS](https://rustfs.com/images/rustfs-github.png)](https://rustfs.com)
-
-# RustFS AppAuth - Application Authentication
-
-<p align="center">
-  <strong>Application-level authentication and authorization module for RustFS distributed object storage</strong>
-</p>
-
-<p align="center">
-  <a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
-  <a href="https://docs.rustfs.com/">📖 Documentation</a>
-  · <a href="https://github.com/rustfs/rustfs/issues">🐛 Bug Reports</a>
-  · <a href="https://github.com/rustfs/rustfs/discussions">💬 Discussions</a>
-</p>
-
----
-
-## 📖 Overview
-
-**RustFS AppAuth** provides application-level authentication and authorization capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
-
-## ✨ Features
-
-- JWT-based authentication with secure token management
-- RBAC (Role-Based Access Control) for fine-grained permissions
-- Multi-tenant application isolation and management
-- OAuth 2.0 and OpenID Connect integration
-- API key management and rotation
-- Session management with configurable expiration
-
-## 📚 Documentation
-
-For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
-
-## 📄 License
-
-This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.

crates/audit/Cargo.toml

@@ -29,7 +29,7 @@ categories = ["web-programming", "development-tools", "asynchronous", "api-bindi
 rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
-rustfs-s3-common = { workspace = true }
+rustfs-s3-types = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }
 futures = { workspace = true }

crates/audit/src/entity.rs

@@ -14,7 +14,7 @@
 
 use chrono::{DateTime, Utc};
 use hashbrown::HashMap;
-use rustfs_s3_common::EventName;
+use rustfs_s3_types::EventName;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
 

crates/checksums/Cargo.toml

@@ -23,7 +23,7 @@ homepage.workspace = true
 description = "Checksum calculation and verification callbacks for HTTP request and response bodies sent by service clients generated by RustFS, ensuring data integrity and authenticity."
 keywords = ["checksum-calculation", "verification", "integrity", "authenticity", "rustfs"]
 categories = ["web-programming", "development-tools", "network-programming"]
-documentation = "https://docs.rs/rustfs-signer/latest/rustfs_checksum/"
+documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
 
 [dependencies]
 bytes = { workspace = true }

crates/common/Cargo.toml

@@ -33,12 +33,8 @@ tonic = { workspace = true }
 uuid = { workspace = true }
 chrono = { workspace = true }
 metrics = { workspace = true }
-rustfs-madmin = { workspace = true }
-rustfs-filemeta = { workspace = true }
 serde = { workspace = true }
-path-clean = { workspace = true }
 rmp-serde = { workspace = true }
-async-trait = { workspace = true }
 s3s = { workspace = true }
 tracing = { workspace = true }
 

crates/common/src/globals.rs

@@ -17,6 +17,7 @@
 use chrono::{DateTime, Utc};
 use std::collections::HashMap;
 use std::sync::LazyLock;
+use std::sync::atomic::{AtomicU64, Ordering};
 use tokio::sync::RwLock;
 use tonic::transport::Channel;
 
@@ -27,6 +28,7 @@ pub static GLOBAL_RUSTFS_ADDR: LazyLock<RwLock<String>> = LazyLock::new(|| RwLoc
 pub static GLOBAL_CONN_MAP: LazyLock<RwLock<HashMap<String, Channel>>> = LazyLock::new(|| RwLock::new(HashMap::new()));
 pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
+pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
 /// Global initialization time of the RustFS node.
 pub static GLOBAL_INIT_TIME: LazyLock<RwLock<Option<DateTime<Utc>>>> = LazyLock::new(|| RwLock::new(None));
 
@@ -88,6 +90,16 @@ pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
     *GLOBAL_MTLS_IDENTITY.write().await = identity;
 }
 
+/// Set the global outbound TLS generation.
+pub fn set_global_outbound_tls_generation(generation: u64) {
+    GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
+}
+
+/// Get the global outbound TLS generation.
+pub fn get_global_outbound_tls_generation() -> u64 {
+    GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
+}
+
 /// Evict a stale/dead connection from the global connection cache.
 /// This is critical for cluster recovery when a node dies unexpectedly (e.g., power-off).
 /// By removing the cached connection, subsequent requests will establish a fresh connection.

crates/common/src/heal_channel.rs

@@ -260,9 +260,17 @@ pub enum HealChannelCommand {
         response_tx: oneshot::Sender<Result<HealAdmissionResult, String>>,
     },
     /// Query heal task status
-    Query { heal_path: String, client_token: String },
+    Query {
+        heal_path: String,
+        client_token: String,
+        response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
+    },
     /// Cancel heal task
-    Cancel { heal_path: String },
+    Cancel {
+        heal_path: String,
+        client_token: String,
+        response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
+    },
 }
 
 /// Heal request from admin to ahm
@@ -379,7 +387,9 @@ fn heal_response_sender() -> &'static HealResponseSender {
 
 /// Publish a heal response to subscribers.
 pub fn publish_heal_response(response: HealChannelResponse) -> Result<(), broadcast::error::SendError<HealChannelResponse>> {
-    heal_response_sender().send(response).map(|_| ())
+    let sender = heal_response_sender();
+    let _ = sender.send(response);
+    Ok(())
 }
 
 /// Subscribe to heal responses.
@@ -405,14 +415,36 @@ pub async fn send_heal_request(request: HealChannelRequest) -> Result<(), String
     }
 }
 
+async fn receive_heal_channel_response(
+    response_rx: oneshot::Receiver<Result<HealChannelResponse, String>>,
+) -> Result<HealChannelResponse, String> {
+    response_rx
+        .await
+        .map_err(|e| format!("Failed to receive heal channel response: {e}"))?
+}
+
 /// Send heal query request
-pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<(), String> {
-    send_heal_command(HealChannelCommand::Query { heal_path, client_token }).await
+pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    send_heal_command(HealChannelCommand::Query {
+        heal_path,
+        client_token,
+        response_tx,
+    })
+    .await?;
+    receive_heal_channel_response(response_rx).await
 }
 
 /// Send heal cancel request
-pub async fn cancel_heal_task(heal_path: String) -> Result<(), String> {
-    send_heal_command(HealChannelCommand::Cancel { heal_path }).await
+pub async fn cancel_heal_task(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    send_heal_command(HealChannelCommand::Cancel {
+        heal_path,
+        client_token,
+        response_tx,
+    })
+    .await?;
+    receive_heal_channel_response(response_rx).await
 }
 
 /// Create a new heal request
@@ -626,5 +658,10 @@ mod tests {
         let received = receiver.recv().await.expect("should receive heal response");
         assert_eq!(received.request_id, response.request_id);
         assert!(received.success);
+
+        drop(receiver);
+        let response = create_heal_response("req-no-subscriber".to_string(), true, None, None);
+
+        publish_heal_response(response).expect("publish without subscribers should be ignored");
     }
 }

crates/common/src/internode_metrics.rs

@@ -1,166 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use metrics::{counter, gauge};
-use std::sync::{
-    Arc, LazyLock,
-    atomic::{AtomicU64, Ordering},
-};
-use std::time::{Duration, SystemTime, UNIX_EPOCH};
-
-#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
-pub struct InternodeMetricsSnapshot {
-    pub sent_bytes_total: u64,
-    pub recv_bytes_total: u64,
-    pub outgoing_requests_total: u64,
-    pub incoming_requests_total: u64,
-    pub errors_total: u64,
-    pub dial_errors_total: u64,
-    pub dial_avg_time_nanos: u64,
-    pub last_dial_unix_millis: u64,
-}
-
-#[derive(Debug, Default)]
-pub struct InternodeMetrics {
-    sent_bytes_total: AtomicU64,
-    recv_bytes_total: AtomicU64,
-    outgoing_requests_total: AtomicU64,
-    incoming_requests_total: AtomicU64,
-    errors_total: AtomicU64,
-    dial_errors_total: AtomicU64,
-    dial_total_time_nanos: AtomicU64,
-    dial_samples_total: AtomicU64,
-    last_dial_unix_millis: AtomicU64,
-}
-
-impl InternodeMetrics {
-    pub fn record_sent_bytes(&self, bytes: usize) {
-        let bytes = bytes as u64;
-        if bytes == 0 {
-            return;
-        }
-        self.sent_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_sent_bytes_total").increment(bytes);
-    }
-
-    pub fn record_recv_bytes(&self, bytes: usize) {
-        let bytes = bytes as u64;
-        if bytes == 0 {
-            return;
-        }
-        self.recv_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_recv_bytes_total").increment(bytes);
-    }
-
-    pub fn record_outgoing_request(&self) {
-        self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_requests_outgoing_total").increment(1);
-    }
-
-    pub fn record_incoming_request(&self) {
-        self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_requests_incoming_total").increment(1);
-    }
-
-    pub fn record_error(&self) {
-        self.errors_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_errors_total").increment(1);
-    }
-
-    pub fn record_dial_result(&self, duration: Duration, success: bool) {
-        let elapsed_nanos = duration.as_nanos().min(u128::from(u64::MAX)) as u64;
-        self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
-        let samples = self.dial_samples_total.fetch_add(1, Ordering::Relaxed) + 1;
-        let total = self.dial_total_time_nanos.load(Ordering::Relaxed);
-        gauge!("rustfs_system_network_internode_dial_avg_time_nanos").set(total as f64 / samples as f64);
-
-        if !success {
-            self.dial_errors_total.fetch_add(1, Ordering::Relaxed);
-            counter!("rustfs_system_network_internode_dial_errors_total").increment(1);
-        }
-
-        let now_ms = SystemTime::now()
-            .duration_since(UNIX_EPOCH)
-            .unwrap_or_default()
-            .as_millis()
-            .min(u128::from(u64::MAX)) as u64;
-        self.last_dial_unix_millis.store(now_ms, Ordering::Relaxed);
-    }
-
-    pub fn snapshot(&self) -> InternodeMetricsSnapshot {
-        let dial_samples_total = self.dial_samples_total.load(Ordering::Relaxed);
-        let dial_total_time_nanos = self.dial_total_time_nanos.load(Ordering::Relaxed);
-        let dial_avg_time_nanos = dial_total_time_nanos.checked_div(dial_samples_total).unwrap_or(0);
-
-        InternodeMetricsSnapshot {
-            sent_bytes_total: self.sent_bytes_total.load(Ordering::Relaxed),
-            recv_bytes_total: self.recv_bytes_total.load(Ordering::Relaxed),
-            outgoing_requests_total: self.outgoing_requests_total.load(Ordering::Relaxed),
-            incoming_requests_total: self.incoming_requests_total.load(Ordering::Relaxed),
-            errors_total: self.errors_total.load(Ordering::Relaxed),
-            dial_errors_total: self.dial_errors_total.load(Ordering::Relaxed),
-            dial_avg_time_nanos,
-            last_dial_unix_millis: self.last_dial_unix_millis.load(Ordering::Relaxed),
-        }
-    }
-
-    #[doc(hidden)]
-    pub fn reset_for_test(&self) {
-        self.sent_bytes_total.store(0, Ordering::Relaxed);
-        self.recv_bytes_total.store(0, Ordering::Relaxed);
-        self.outgoing_requests_total.store(0, Ordering::Relaxed);
-        self.incoming_requests_total.store(0, Ordering::Relaxed);
-        self.errors_total.store(0, Ordering::Relaxed);
-        self.dial_errors_total.store(0, Ordering::Relaxed);
-        self.dial_total_time_nanos.store(0, Ordering::Relaxed);
-        self.dial_samples_total.store(0, Ordering::Relaxed);
-        self.last_dial_unix_millis.store(0, Ordering::Relaxed);
-    }
-}
-
-pub fn global_internode_metrics() -> &'static Arc<InternodeMetrics> {
-    static GLOBAL_INTERNODE_METRICS: LazyLock<Arc<InternodeMetrics>> = LazyLock::new(|| Arc::new(InternodeMetrics::default()));
-    &GLOBAL_INTERNODE_METRICS
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn snapshot_reports_recorded_values() {
-        let metrics = global_internode_metrics();
-        metrics.reset_for_test();
-
-        metrics.record_sent_bytes(64);
-        metrics.record_recv_bytes(32);
-        metrics.record_outgoing_request();
-        metrics.record_incoming_request();
-        metrics.record_error();
-        metrics.record_dial_result(Duration::from_millis(9), true);
-        metrics.record_dial_result(Duration::from_millis(3), false);
-
-        let snapshot = metrics.snapshot();
-        assert_eq!(snapshot.sent_bytes_total, 64);
-        assert_eq!(snapshot.recv_bytes_total, 32);
-        assert_eq!(snapshot.outgoing_requests_total, 1);
-        assert_eq!(snapshot.incoming_requests_total, 1);
-        assert_eq!(snapshot.errors_total, 1);
-        assert_eq!(snapshot.dial_errors_total, 1);
-        assert_eq!(snapshot.dial_avg_time_nanos, 6_000_000);
-        assert!(snapshot.last_dial_unix_millis > 0);
-
-        metrics.reset_for_test();
-    }
-}

crates/common/src/lib.rs

@@ -13,12 +13,9 @@
 // limitations under the License.
 
 pub mod bucket_stats;
-pub mod capacity_scope;
 // pub mod error;
-pub mod data_usage;
 pub mod globals;
 pub mod heal_channel;
-pub mod internode_metrics;
 pub mod last_minute;
 pub mod metrics;
 mod readiness;

crates/common/src/metrics.rs

@@ -14,7 +14,6 @@
 
 use crate::last_minute::{AccElem, LastMinuteLatency};
 use chrono::{DateTime, Utc};
-use rustfs_madmin::metrics::{ScannerMetrics as M_ScannerMetrics, TimedAction};
 use serde::{Deserialize, Serialize};
 use std::{
     collections::HashMap,
@@ -351,6 +350,32 @@ pub struct CurrentCycle {
     pub started: DateTime<Utc>,
 }
 
+#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
+pub struct ScannerTimedAction {
+    pub count: u64,
+    pub acc_time: u64,
+    pub bytes: u64,
+}
+
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
+pub struct ScannerLastMinute {
+    pub actions: HashMap<String, ScannerTimedAction>,
+    pub ilm: HashMap<String, ScannerTimedAction>,
+}
+
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
+pub struct ScannerMetricsReport {
+    pub collected_at: DateTime<Utc>,
+    pub current_cycle: u64,
+    pub current_started: DateTime<Utc>,
+    pub cycles_completed_at: Vec<DateTime<Utc>>,
+    pub ongoing_buckets: usize,
+    pub life_time_ops: HashMap<String, u64>,
+    pub life_time_ilm: HashMap<String, u64>,
+    pub last_minute: ScannerLastMinute,
+    pub active_paths: Vec<String>,
+}
+
 impl CurrentCycle {
     pub fn unmarshal(&mut self, buf: &[u8]) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         *self = rmp_serde::from_slice(buf)?;
@@ -572,8 +597,8 @@ impl Metrics {
     }
 
     /// Build a full metrics report snapshot.
-    pub async fn report(&self) -> M_ScannerMetrics {
-        let mut m = M_ScannerMetrics::default();
+    pub async fn report(&self) -> ScannerMetricsReport {
+        let mut m = ScannerMetricsReport::default();
 
         if let Some(cycle) = self.get_cycle().await {
             m.current_cycle = cycle.current;
@@ -606,7 +631,7 @@ impl Metrics {
             {
                 m.last_minute.actions.insert(
                     metric.as_str().to_string(),
-                    TimedAction {
+                    ScannerTimedAction {
                         count: last_min.n,
                         acc_time: last_min.total,
                         bytes: last_min.size,
@@ -633,7 +658,7 @@ impl Metrics {
             {
                 m.last_minute.ilm.insert(
                     action.as_str().to_string(),
-                    TimedAction {
+                    ScannerTimedAction {
                         count: last_min.n,
                         acc_time: last_min.total,
                         bytes: last_min.size,

crates/concurrency/src/backpressure.rs

@@ -14,15 +14,17 @@
 
 //! Backpressure management
 
-use rustfs_io_core::{BackpressureMonitor as CoreBackpressureMonitor, BackpressureState};
+use rustfs_io_core::{
+    BackpressureConfig as CoreBackpressureConfig, BackpressureMonitor as CoreBackpressureMonitor, BackpressureState,
+};
 use rustfs_io_metrics::backpressure_metrics;
 use std::sync::Arc;
 use std::time::Instant;
 use tokio::io::{DuplexStream, duplex};
 
-/// Backpressure configuration
-#[derive(Debug, Clone)]
-pub struct BackpressureConfig {
+/// Facade policy for duplex-pipe watermark backpressure.
+#[derive(Debug, Clone, Copy)]
+pub struct PipeBackpressurePolicy {
     /// Buffer size in bytes
     pub buffer_size: usize,
     /// High watermark percentage
@@ -31,7 +33,7 @@ pub struct BackpressureConfig {
     pub low_watermark: u32,
 }
 
-impl Default for BackpressureConfig {
+impl Default for PipeBackpressurePolicy {
     fn default() -> Self {
         Self {
             buffer_size: 4 * 1024 * 1024, // 4MB
@@ -41,7 +43,7 @@ impl Default for BackpressureConfig {
     }
 }
 
-impl BackpressureConfig {
+impl PipeBackpressurePolicy {
     /// Calculate high watermark threshold in bytes
     pub fn high_watermark_bytes(&self) -> usize {
         (self.buffer_size as u64 * self.high_watermark as u64 / 100) as usize
@@ -51,42 +53,59 @@ impl BackpressureConfig {
     pub fn low_watermark_bytes(&self) -> usize {
         (self.buffer_size as u64 * self.low_watermark as u64 / 100) as usize
     }
+
+    /// Convert the facade policy into the reusable io-core admission-pressure config.
+    ///
+    /// The concurrency layer still owns duplex buffer sizing, but the shared
+    /// overload/admission primitive lives in `io-core`.
+    pub fn to_core_config(&self) -> CoreBackpressureConfig {
+        CoreBackpressureConfig {
+            max_concurrent: 32,
+            high_water_mark: self.high_watermark as f64 / 100.0,
+            low_water_mark: self.low_watermark as f64 / 100.0,
+            cooldown: std::time::Duration::from_millis(100),
+            enabled: true,
+        }
+    }
 }
 
 /// Backpressure manager
 pub struct BackpressureManager {
-    config: BackpressureConfig,
+    config: PipeBackpressurePolicy,
+    core_config: CoreBackpressureConfig,
     monitor: Arc<CoreBackpressureMonitor>,
 }
 
 impl BackpressureManager {
     /// Create a new backpressure manager
     pub fn new(buffer_size: usize, high_watermark: u32, low_watermark: u32) -> Self {
-        let config = BackpressureConfig {
+        Self::from_policy(PipeBackpressurePolicy {
             buffer_size,
             high_watermark,
             low_watermark,
-        };
-
-        let core_config = rustfs_io_core::BackpressureConfig {
-            max_concurrent: 32,
-            high_water_mark: high_watermark as f64 / 100.0,
-            low_water_mark: low_watermark as f64 / 100.0,
-            cooldown: std::time::Duration::from_millis(100),
-            enabled: true,
-        };
+        })
+    }
 
+    /// Create a new backpressure manager from the facade policy type.
+    pub fn from_policy(config: PipeBackpressurePolicy) -> Self {
+        let core_config = config.to_core_config();
         Self {
             config,
+            core_config: core_config.clone(),
             monitor: Arc::new(CoreBackpressureMonitor::new(core_config)),
         }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &BackpressureConfig {
+    pub fn config(&self) -> &PipeBackpressurePolicy {
         &self.config
     }
 
+    /// Get the derived io-core admission-pressure configuration.
+    pub fn core_config(&self) -> &CoreBackpressureConfig {
+        &self.core_config
+    }
+
     /// Get the monitor
     pub fn monitor(&self) -> Arc<CoreBackpressureMonitor> {
         self.monitor.clone()
@@ -94,7 +113,7 @@ impl BackpressureManager {
 
     /// Create a backpressure pipe
     pub fn create_pipe(&self) -> BackpressurePipe {
-        BackpressurePipe::new(self.config.clone(), self.monitor.clone())
+        BackpressurePipe::new(self.config, self.monitor.clone())
     }
 
     /// Get current state
@@ -112,13 +131,24 @@ impl BackpressureManager {
 pub struct BackpressurePipe {
     reader: DuplexStream,
     writer: DuplexStream,
-    config: BackpressureConfig,
+    config: PipeBackpressurePolicy,
     monitor: Arc<CoreBackpressureMonitor>,
     created_at: Instant,
 }
 
+/// Shared pipe metadata snapshot for facade-level backpressure pipes.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct BackpressurePipeMeta {
+    /// Configured duplex buffer capacity in bytes.
+    pub buffer_capacity: usize,
+    /// Current backpressure state reported by the shared core monitor.
+    pub state: BackpressureState,
+    /// Age of the pipe since creation.
+    pub age: std::time::Duration,
+}
+
 impl BackpressurePipe {
-    fn new(config: BackpressureConfig, monitor: Arc<CoreBackpressureMonitor>) -> Self {
+    fn new(config: PipeBackpressurePolicy, monitor: Arc<CoreBackpressureMonitor>) -> Self {
         let (reader, writer) = duplex(config.buffer_size);
 
         Self {
@@ -146,7 +176,7 @@ impl BackpressurePipe {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &BackpressureConfig {
+    pub fn config(&self) -> &PipeBackpressurePolicy {
         &self.config
     }
 
@@ -160,6 +190,15 @@ impl BackpressurePipe {
         self.created_at.elapsed()
     }
 
+    /// Get a compact metadata snapshot for the pipe.
+    pub fn meta(&self) -> BackpressurePipeMeta {
+        BackpressurePipeMeta {
+            buffer_capacity: self.config.buffer_size,
+            state: self.state(),
+            age: self.age(),
+        }
+    }
+
     /// Check if should apply backpressure
     pub fn should_apply_backpressure(&self) -> bool {
         let should = self.monitor.should_apply_backpressure();
@@ -170,45 +209,26 @@ impl BackpressurePipe {
     }
 }
 
-/// Backpressure event
-#[allow(dead_code)]
-#[derive(Debug, Clone)]
-pub struct BackpressureEvent {
-    /// Event timestamp
-    pub timestamp: Instant,
-    /// Event type
-    pub event_type: BackpressureEventType,
-    /// Buffer usage
-    pub buffer_usage: usize,
-    /// Buffer capacity
-    pub buffer_capacity: usize,
-}
-
-/// Backpressure event type
-#[allow(dead_code)]
-#[derive(Debug, Clone, Copy)]
-pub enum BackpressureEventType {
-    /// High watermark reached
-    HighWatermarkReached,
-    /// High watermark exited
-    HighWatermarkExited,
-    /// Backpressure applied
-    BackpressureApplied,
-    /// Backpressure released
-    BackpressureReleased,
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
 
     #[test]
     fn test_backpressure_config() {
-        let config = BackpressureConfig::default();
+        let config = PipeBackpressurePolicy::default();
         assert_eq!(config.buffer_size, 4 * 1024 * 1024);
         assert!(config.high_watermark > config.low_watermark);
     }
 
+    #[test]
+    fn test_backpressure_policy_to_core_config() {
+        let policy = PipeBackpressurePolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.high_water_mark, policy.high_watermark as f64 / 100.0);
+        assert_eq!(core.low_water_mark, policy.low_watermark as f64 / 100.0);
+        assert!(core.enabled);
+    }
+
     #[test]
     fn test_backpressure_manager() {
         let manager = BackpressureManager::new(1024, 80, 50);
@@ -220,5 +240,6 @@ mod tests {
         let manager = BackpressureManager::new(1024, 80, 50);
         let pipe = manager.create_pipe();
         assert_eq!(pipe.state(), BackpressureState::Normal);
+        assert_eq!(pipe.meta().buffer_capacity, 1024);
     }
 }

crates/concurrency/src/config.rs

@@ -14,6 +14,10 @@
 
 //! Configuration for concurrency management
 
+use crate::{
+    backpressure::PipeBackpressurePolicy, deadlock::DeadlockMonitorPolicy, scheduler::SchedulerPolicy,
+    timeout::TimeoutManagerPolicy,
+};
 use std::time::Duration;
 
 /// Feature flags for concurrency modules
@@ -72,84 +76,39 @@ impl ConcurrencyFeatures {
     }
 }
 
+/// Facade policy for lock manager behavior.
+#[derive(Debug, Clone, Copy)]
+pub struct LockManagerPolicy {
+    /// Enable lock optimization.
+    pub enabled: bool,
+    /// Lock acquisition timeout.
+    pub acquire_timeout: Duration,
+}
+
+impl Default for LockManagerPolicy {
+    fn default() -> Self {
+        Self {
+            enabled: true,
+            acquire_timeout: Duration::from_secs(5),
+        }
+    }
+}
+
 /// Main configuration for concurrency management
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Default)]
 pub struct ConcurrencyConfig {
     /// Feature flags
     pub features: ConcurrencyFeatures,
-
-    // Timeout configuration
-    /// Default timeout duration
-    pub default_timeout: Duration,
-    /// Maximum timeout duration
-    pub max_timeout: Duration,
-    /// Enable dynamic timeout
-    pub enable_dynamic_timeout: bool,
-
-    // Lock configuration
-    /// Enable lock optimization
-    pub enable_lock_optimization: bool,
-    /// Lock acquisition timeout
-    pub lock_acquire_timeout: Duration,
-
-    // Deadlock configuration
-    /// Enable deadlock detection
-    pub enable_deadlock_detection: bool,
-    /// Deadlock check interval
-    pub deadlock_check_interval: Duration,
-    /// Hang threshold
-    pub hang_threshold: Duration,
-
-    // Backpressure configuration
-    /// Buffer size for backpressure
-    pub backpressure_buffer_size: usize,
-    /// High watermark percentage
-    pub high_watermark: u32,
-    /// Low watermark percentage
-    pub low_watermark: u32,
-
-    // Scheduler configuration
-    /// Base buffer size for I/O
-    pub io_buffer_size: usize,
-    /// Maximum buffer size
-    pub max_buffer_size: usize,
-    /// High priority size threshold
-    pub high_priority_threshold: usize,
-    /// Low priority size threshold
-    pub low_priority_threshold: usize,
-}
-
-impl Default for ConcurrencyConfig {
-    fn default() -> Self {
-        Self {
-            features: ConcurrencyFeatures::default(),
-
-            // Timeout defaults
-            default_timeout: Duration::from_secs(30),
-            max_timeout: Duration::from_secs(300),
-            enable_dynamic_timeout: true,
-
-            // Lock defaults
-            enable_lock_optimization: true,
-            lock_acquire_timeout: Duration::from_secs(5),
-
-            // Deadlock defaults
-            enable_deadlock_detection: false,
-            deadlock_check_interval: Duration::from_secs(10),
-            hang_threshold: Duration::from_secs(60),
-
-            // Backpressure defaults
-            backpressure_buffer_size: 4 * 1024 * 1024, // 4MB
-            high_watermark: 80,
-            low_watermark: 50,
-
-            // Scheduler defaults
-            io_buffer_size: 64 * 1024,                // 64KB
-            max_buffer_size: 4 * 1024 * 1024,         // 4MB
-            high_priority_threshold: 1024 * 1024,     // 1MB
-            low_priority_threshold: 10 * 1024 * 1024, // 10MB
-        }
-    }
+    /// Timeout facade policy.
+    pub timeout_policy: TimeoutManagerPolicy,
+    /// Lock facade policy.
+    pub lock_policy: LockManagerPolicy,
+    /// Deadlock facade policy.
+    pub deadlock_policy: DeadlockMonitorPolicy,
+    /// Backpressure facade policy.
+    pub backpressure_policy: PipeBackpressurePolicy,
+    /// Scheduler facade policy.
+    pub scheduler_policy: SchedulerPolicy,
 }
 
 impl ConcurrencyConfig {
@@ -161,25 +120,25 @@ impl ConcurrencyConfig {
         if let Ok(val) = std::env::var("RUSTFS_TIMEOUT_DEFAULT")
             && let Ok(secs) = val.parse::<u64>()
         {
-            config.default_timeout = Duration::from_secs(secs);
+            config.timeout_policy.default_timeout = Duration::from_secs(secs);
         }
 
         if let Ok(val) = std::env::var("RUSTFS_TIMEOUT_MAX")
             && let Ok(secs) = val.parse::<u64>()
         {
-            config.max_timeout = Duration::from_secs(secs);
+            config.timeout_policy.max_timeout = Duration::from_secs(secs);
         }
 
         if let Ok(val) = std::env::var("RUSTFS_BACKPRESSURE_BUFFER_SIZE")
             && let Ok(size) = val.parse::<usize>()
         {
-            config.backpressure_buffer_size = size;
+            config.backpressure_policy.buffer_size = size;
         }
 
         if let Ok(val) = std::env::var("RUSTFS_IO_BUFFER_SIZE")
             && let Ok(size) = val.parse::<usize>()
         {
-            config.io_buffer_size = size;
+            config.scheduler_policy.base_buffer_size = size;
         }
 
         config
@@ -187,18 +146,25 @@ impl ConcurrencyConfig {
 
     /// Validate configuration
     pub fn validate(&self) -> Result<(), ConfigError> {
-        if self.default_timeout > self.max_timeout {
+        if self.timeout_policy.default_timeout > self.timeout_policy.max_timeout {
             return Err(ConfigError::InvalidTimeout("default_timeout cannot exceed max_timeout".to_string()));
         }
+        if self.timeout_policy.min_timeout > self.timeout_policy.max_timeout {
+            return Err(ConfigError::InvalidTimeout("min_timeout cannot exceed max_timeout".to_string()));
+        }
 
-        if self.high_watermark <= self.low_watermark || self.high_watermark > 100 {
+        if self.backpressure_policy.high_watermark <= self.backpressure_policy.low_watermark
+            || self.backpressure_policy.high_watermark > 100
+        {
             return Err(ConfigError::InvalidBackpressure(
                 "high_watermark must be > low_watermark and <= 100".to_string(),
             ));
         }
 
-        if self.io_buffer_size > self.max_buffer_size {
-            return Err(ConfigError::InvalidScheduler("io_buffer_size cannot exceed max_buffer_size".to_string()));
+        if self.scheduler_policy.base_buffer_size > self.scheduler_policy.max_buffer_size {
+            return Err(ConfigError::InvalidScheduler(
+                "base_buffer_size cannot exceed max_buffer_size".to_string(),
+            ));
         }
 
         Ok(())
@@ -235,8 +201,12 @@ mod tests {
     #[test]
     fn test_invalid_timeout() {
         let config = ConcurrencyConfig {
-            default_timeout: Duration::from_secs(100),
-            max_timeout: Duration::from_secs(50),
+            timeout_policy: TimeoutManagerPolicy {
+                default_timeout: Duration::from_secs(100),
+                max_timeout: Duration::from_secs(50),
+                enable_dynamic: true,
+                ..Default::default()
+            },
             ..Default::default()
         };
         assert!(
@@ -245,6 +215,22 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_invalid_min_timeout() {
+        let config = ConcurrencyConfig {
+            timeout_policy: TimeoutManagerPolicy {
+                min_timeout: Duration::from_secs(100),
+                max_timeout: Duration::from_secs(50),
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+        assert!(
+            config.validate().is_err(),
+            "validate() should return an error when min_timeout > max_timeout"
+        );
+    }
+
     #[test]
     fn test_features() {
         let features = ConcurrencyFeatures::all();

crates/concurrency/src/deadlock.rs

@@ -14,15 +14,15 @@
 
 //! Deadlock detection management
 
-use rustfs_io_core::{DeadlockDetector as CoreDeadlockDetector, LockType};
+use rustfs_io_core::{DeadlockDetector as CoreDeadlockDetector, DeadlockDetectorConfig as CoreDeadlockConfig, LockType};
 use rustfs_io_metrics::deadlock_metrics;
 use std::collections::HashMap;
 use std::sync::Arc;
 use std::time::{Duration, Instant};
 
-/// Deadlock configuration
-#[derive(Debug, Clone)]
-pub struct DeadlockConfig {
+/// Facade policy for the concurrency-layer deadlock monitor.
+#[derive(Debug, Clone, Copy)]
+pub struct DeadlockMonitorPolicy {
     /// Enable deadlock detection
     pub enabled: bool,
     /// Check interval
@@ -31,7 +31,7 @@ pub struct DeadlockConfig {
     pub hang_threshold: Duration,
 }
 
-impl Default for DeadlockConfig {
+impl Default for DeadlockMonitorPolicy {
     fn default() -> Self {
         Self {
             enabled: false,
@@ -41,9 +41,20 @@ impl Default for DeadlockConfig {
     }
 }
 
+impl DeadlockMonitorPolicy {
+    /// Convert the facade policy into the reusable io-core deadlock config.
+    pub fn to_core_config(&self) -> CoreDeadlockConfig {
+        CoreDeadlockConfig {
+            enabled: self.enabled,
+            detection_interval: self.check_interval,
+            max_hold_time: self.hang_threshold,
+        }
+    }
+}
+
 /// Deadlock manager
 pub struct DeadlockManager {
-    config: DeadlockConfig,
+    config: DeadlockMonitorPolicy,
     detector: Arc<CoreDeadlockDetector>,
     running: Arc<tokio::sync::Mutex<bool>>,
 }
@@ -51,18 +62,16 @@ pub struct DeadlockManager {
 impl DeadlockManager {
     /// Create a new deadlock manager
     pub fn new(enabled: bool, check_interval: Duration, hang_threshold: Duration) -> Self {
-        let config = DeadlockConfig {
+        Self::from_policy(DeadlockMonitorPolicy {
             enabled,
             check_interval,
             hang_threshold,
-        };
-
-        let core_config = rustfs_io_core::DeadlockDetectorConfig {
-            enabled,
-            detection_interval: check_interval,
-            max_hold_time: hang_threshold,
-        };
+        })
+    }
 
+    /// Create a new deadlock manager from the facade policy type.
+    pub fn from_policy(config: DeadlockMonitorPolicy) -> Self {
+        let core_config = config.to_core_config();
         Self {
             config,
             detector: Arc::new(CoreDeadlockDetector::new(core_config)),
@@ -71,7 +80,7 @@ impl DeadlockManager {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &DeadlockConfig {
+    pub fn config(&self) -> &DeadlockMonitorPolicy {
         &self.config
     }
 
@@ -129,7 +138,11 @@ impl DeadlockManager {
     }
 }
 
-/// Request tracker for tracking resources
+/// Lightweight compatibility wrapper for request-scoped deadlock bookkeeping.
+///
+/// This type intentionally stays minimal in the concurrency layer. Rich
+/// request-level lock/resource diagnostics belong to
+/// `rustfs::storage::deadlock_detector::RequestResourceTracker`.
 pub struct RequestTracker {
     request_id: String,
     description: String,
@@ -174,6 +187,11 @@ impl RequestTracker {
         deadlock_metrics::record_lock_acquisition("read");
     }
 
+    /// Return a read-only view of tracked resource names.
+    pub fn resources(&self) -> &HashMap<String, Vec<String>> {
+        &self.resources
+    }
+
     /// Record a lock release
     pub fn record_lock_release(&mut self, lock_id: u64) {
         self.detector.record_release(lock_id);
@@ -196,12 +214,24 @@ mod tests {
         assert!(!manager.config().enabled);
     }
 
+    #[test]
+    fn test_deadlock_policy_to_core_config() {
+        let policy = DeadlockMonitorPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.enabled, policy.enabled);
+        assert_eq!(core.detection_interval, policy.check_interval);
+        assert_eq!(core.max_hold_time, policy.hang_threshold);
+    }
+
     #[tokio::test]
     async fn test_request_tracker() {
         let manager = DeadlockManager::new(true, Duration::from_secs(10), Duration::from_secs(60));
-        let tracker = manager.track_request("req-1".to_string(), "test request".to_string());
+        let mut tracker = manager.track_request("req-1".to_string(), "test request".to_string());
+        let lock_id = manager.register_lock(LockType::Mutex);
+        tracker.record_lock_acquire(lock_id, "bucket/key".to_string());
 
         assert_eq!(tracker.request_id(), "req-1");
         assert_eq!(tracker.description(), "test request");
+        assert_eq!(tracker.resources().get("locks").map(Vec::len), Some(1));
     }
 }

crates/concurrency/src/lib.rs

@@ -124,21 +124,23 @@ mod backpressure;
 #[cfg(feature = "scheduler")]
 mod scheduler;
 
+pub mod workers;
+
 // Public module exports with feature gates
 #[cfg(feature = "timeout")]
-pub use timeout::{TimeoutConfig, TimeoutGuard, TimeoutManager};
+pub use timeout::{TimeoutGuard, TimeoutManager, TimeoutManagerPolicy};
 
 #[cfg(feature = "lock")]
 pub use lock::{LockConfig, LockManager, LockScopeGuard, OptimizedLockGuard};
 
 #[cfg(feature = "deadlock")]
-pub use deadlock::{DeadlockConfig, DeadlockManager, RequestTracker};
+pub use deadlock::{DeadlockManager, DeadlockMonitorPolicy, RequestTracker};
 
 #[cfg(feature = "backpressure")]
-pub use backpressure::{BackpressureConfig, BackpressureManager, BackpressurePipe};
+pub use backpressure::{BackpressureManager, BackpressurePipe, PipeBackpressurePolicy};
 
 #[cfg(feature = "scheduler")]
-pub use scheduler::{IoStrategy, SchedulerConfig, SchedulerManager};
+pub use scheduler::{IoStrategy, SchedulerManager, SchedulerPolicy};
 
 // Configuration
 mod config;
@@ -153,19 +155,19 @@ pub mod prelude {
     //! Prelude module for convenient imports
 
     #[cfg(feature = "timeout")]
-    pub use crate::timeout::{TimeoutConfig, TimeoutGuard, TimeoutManager};
+    pub use crate::timeout::{TimeoutGuard, TimeoutManager, TimeoutManagerPolicy};
 
     #[cfg(feature = "lock")]
     pub use crate::lock::{LockConfig, LockManager, LockScopeGuard, OptimizedLockGuard};
 
     #[cfg(feature = "deadlock")]
-    pub use crate::deadlock::{DeadlockConfig, DeadlockManager, RequestTracker};
+    pub use crate::deadlock::{DeadlockManager, DeadlockMonitorPolicy, RequestTracker};
 
     #[cfg(feature = "backpressure")]
-    pub use crate::backpressure::{BackpressureConfig, BackpressureManager, BackpressurePipe};
+    pub use crate::backpressure::{BackpressureManager, BackpressurePipe, PipeBackpressurePolicy};
 
     #[cfg(feature = "scheduler")]
-    pub use crate::scheduler::{IoStrategy, SchedulerConfig, SchedulerManager};
+    pub use crate::scheduler::{IoStrategy, SchedulerManager, SchedulerPolicy};
 
     pub use crate::{ConcurrencyConfig, ConcurrencyFeatures, ConcurrencyManager};
 }

crates/concurrency/src/manager.rs

@@ -85,39 +85,22 @@ impl ConcurrencyManager {
 
         Self {
             #[cfg(feature = "timeout")]
-            timeout: Arc::new(crate::timeout::TimeoutManager::new(
-                config.default_timeout,
-                config.max_timeout,
-                config.enable_dynamic_timeout,
-            )),
+            timeout: Arc::new(crate::timeout::TimeoutManager::from_policy(config.timeout_policy)),
 
             #[cfg(feature = "lock")]
             lock: Arc::new(crate::lock::LockManager::new(
-                config.enable_lock_optimization,
-                config.lock_acquire_timeout,
+                config.lock_policy.enabled,
+                config.lock_policy.acquire_timeout,
             )),
 
             #[cfg(feature = "deadlock")]
-            deadlock: Arc::new(crate::deadlock::DeadlockManager::new(
-                config.enable_deadlock_detection,
-                config.deadlock_check_interval,
-                config.hang_threshold,
-            )),
+            deadlock: Arc::new(crate::deadlock::DeadlockManager::from_policy(config.deadlock_policy)),
 
             #[cfg(feature = "backpressure")]
-            backpressure: Arc::new(crate::backpressure::BackpressureManager::new(
-                config.backpressure_buffer_size,
-                config.high_watermark,
-                config.low_watermark,
-            )),
+            backpressure: Arc::new(crate::backpressure::BackpressureManager::from_policy(config.backpressure_policy)),
 
             #[cfg(feature = "scheduler")]
-            scheduler: Arc::new(crate::scheduler::SchedulerManager::new(
-                config.io_buffer_size,
-                config.max_buffer_size,
-                config.high_priority_threshold,
-                config.low_priority_threshold,
-            )),
+            scheduler: Arc::new(crate::scheduler::SchedulerManager::from_policy(config.scheduler_policy)),
 
             config,
         }
@@ -244,7 +227,7 @@ impl ConcurrencyManager {
     pub async fn start(&self) {
         #[cfg(feature = "deadlock")]
         {
-            if self.config.enable_deadlock_detection {
+            if self.config.deadlock_policy.enabled {
                 self.deadlock.start().await;
             }
         }

crates/concurrency/src/scheduler.rs

@@ -22,9 +22,9 @@ use rustfs_io_metrics::io_metrics;
 use std::sync::Arc;
 use std::time::Duration;
 
-/// Scheduler configuration
-#[derive(Debug, Clone)]
-pub struct SchedulerConfig {
+/// Facade policy for the concurrency-layer scheduler manager.
+#[derive(Debug, Clone, Copy)]
+pub struct SchedulerPolicy {
     /// Base buffer size
     pub base_buffer_size: usize,
     /// Maximum buffer size
@@ -35,7 +35,7 @@ pub struct SchedulerConfig {
     pub low_priority_threshold: usize,
 }
 
-impl Default for SchedulerConfig {
+impl Default for SchedulerPolicy {
     fn default() -> Self {
         Self {
             base_buffer_size: 64 * 1024,              // 64KB
@@ -46,9 +46,23 @@ impl Default for SchedulerConfig {
     }
 }
 
+impl SchedulerPolicy {
+    /// Convert facade policy to io-core scheduler config.
+    pub fn to_core_config(&self) -> rustfs_io_core::IoSchedulerConfig {
+        rustfs_io_core::IoSchedulerConfig {
+            base_buffer_size: self.base_buffer_size,
+            max_buffer_size: self.max_buffer_size,
+            high_priority_size_threshold: self.high_priority_threshold,
+            low_priority_size_threshold: self.low_priority_threshold,
+            ..rustfs_io_core::IoSchedulerConfig::default()
+        }
+    }
+}
+
 /// Scheduler manager
 pub struct SchedulerManager {
-    config: SchedulerConfig,
+    config: SchedulerPolicy,
+    core_config: rustfs_io_core::IoSchedulerConfig,
     scheduler: Arc<CoreIoScheduler>,
 }
 
@@ -60,26 +74,35 @@ impl SchedulerManager {
         high_priority_threshold: usize,
         low_priority_threshold: usize,
     ) -> Self {
-        let config = SchedulerConfig {
+        Self::from_policy(SchedulerPolicy {
             base_buffer_size,
             max_buffer_size,
             high_priority_threshold,
             low_priority_threshold,
-        };
+        })
+    }
 
-        let core_config = rustfs_io_core::IoSchedulerConfig::default();
+    /// Create a scheduler manager from facade policy.
+    pub fn from_policy(config: SchedulerPolicy) -> Self {
+        let core_config = config.to_core_config();
 
         Self {
             config,
+            core_config: core_config.clone(),
             scheduler: Arc::new(CoreIoScheduler::new(core_config)),
         }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &SchedulerConfig {
+    pub fn config(&self) -> &SchedulerPolicy {
         &self.config
     }
 
+    /// Get the derived io-core scheduler config.
+    pub fn core_config(&self) -> &rustfs_io_core::IoSchedulerConfig {
+        &self.core_config
+    }
+
     /// Get the scheduler
     pub fn scheduler(&self) -> Arc<CoreIoScheduler> {
         self.scheduler.clone()
@@ -87,7 +110,7 @@ impl SchedulerManager {
 
     /// Create an I/O strategy
     pub fn create_strategy(&self) -> IoStrategy {
-        IoStrategy::new(self.config.clone(), self.scheduler.clone())
+        IoStrategy::new(self.config, self.scheduler.clone())
     }
 
     /// Calculate buffer size
@@ -111,12 +134,12 @@ impl SchedulerManager {
 
 /// I/O strategy
 pub struct IoStrategy {
-    config: SchedulerConfig,
+    config: SchedulerPolicy,
     scheduler: Arc<CoreIoScheduler>,
 }
 
 impl IoStrategy {
-    fn new(config: SchedulerConfig, scheduler: Arc<CoreIoScheduler>) -> Self {
+    fn new(config: SchedulerPolicy, scheduler: Arc<CoreIoScheduler>) -> Self {
         Self { config, scheduler }
     }
 
@@ -191,7 +214,7 @@ impl IoStrategy {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &SchedulerConfig {
+    pub fn config(&self) -> &SchedulerPolicy {
         &self.config
     }
 }
@@ -202,10 +225,20 @@ mod tests {
 
     #[test]
     fn test_scheduler_config() {
-        let config = SchedulerConfig::default();
+        let config = SchedulerPolicy::default();
         assert!(config.base_buffer_size < config.max_buffer_size);
     }
 
+    #[test]
+    fn test_scheduler_policy_to_core_config() {
+        let policy = SchedulerPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.base_buffer_size, policy.base_buffer_size);
+        assert_eq!(core.max_buffer_size, policy.max_buffer_size);
+        assert_eq!(core.high_priority_size_threshold, policy.high_priority_threshold);
+        assert_eq!(core.low_priority_size_threshold, policy.low_priority_threshold);
+    }
+
     #[test]
     fn test_scheduler_manager() {
         let manager = SchedulerManager::new(1024, 4096, 512, 2048);

crates/concurrency/src/timeout.rs

@@ -14,60 +14,101 @@
 
 //! Timeout management for operations
 
-use rustfs_io_core::{TimeoutError, calculate_adaptive_timeout};
+use rustfs_io_core::{TimeoutConfig as CoreTimeoutConfig, TimeoutError, calculate_adaptive_timeout};
 use std::time::{Duration, Instant};
 use tokio_util::sync::CancellationToken;
 
-/// Timeout configuration
-#[derive(Debug, Clone)]
-pub struct TimeoutConfig {
+/// Facade policy for the concurrency-layer timeout manager.
+#[derive(Debug, Clone, Copy)]
+pub struct TimeoutManagerPolicy {
     /// Default timeout duration
     pub default_timeout: Duration,
     /// Maximum timeout duration
     pub max_timeout: Duration,
+    /// Minimum timeout floor (prevents dynamic calculation from going too low).
+    pub min_timeout: Duration,
     /// Enable dynamic timeout calculation
     pub enable_dynamic: bool,
 }
 
-impl Default for TimeoutConfig {
+impl Default for TimeoutManagerPolicy {
     fn default() -> Self {
         Self {
             default_timeout: Duration::from_secs(30),
             max_timeout: Duration::from_secs(300),
+            min_timeout: Duration::from_secs(5),
             enable_dynamic: true,
         }
     }
 }
 
+impl TimeoutManagerPolicy {
+    /// Convert the facade policy into the reusable io-core timeout configuration.
+    ///
+    /// This keeps the concurrency layer explicitly wired to the shared core
+    /// timeout primitives without changing the facade's public behavior.
+    pub fn to_core_config(&self) -> CoreTimeoutConfig {
+        CoreTimeoutConfig {
+            base_timeout: self.default_timeout,
+            timeout_per_mb: Duration::ZERO,
+            max_timeout: self.max_timeout,
+            min_timeout: self.min_timeout,
+            get_object_timeout: self.default_timeout,
+            put_object_timeout: self.max_timeout,
+            list_objects_timeout: self.default_timeout,
+            enable_dynamic_timeout: self.enable_dynamic,
+        }
+    }
+}
+
 /// Timeout manager
 pub struct TimeoutManager {
-    config: TimeoutConfig,
+    config: TimeoutManagerPolicy,
+    core_config: CoreTimeoutConfig,
 }
 
 impl TimeoutManager {
     /// Create a new timeout manager
     pub fn new(default_timeout: Duration, max_timeout: Duration, enable_dynamic: bool) -> Self {
-        Self {
-            config: TimeoutConfig {
-                default_timeout,
-                max_timeout,
-                enable_dynamic,
-            },
-        }
+        let min_timeout = default_timeout.min(max_timeout);
+        Self::from_policy(TimeoutManagerPolicy {
+            default_timeout,
+            max_timeout,
+            min_timeout,
+            enable_dynamic,
+        })
+    }
+
+    /// Create a new timeout manager from the facade policy type.
+    pub fn from_policy(config: TimeoutManagerPolicy) -> Self {
+        let config = TimeoutManagerPolicy {
+            // Guard clamp(min, max) from panic when callers provide an
+            // out-of-order policy (or very small max_timeout).
+            min_timeout: config.min_timeout.min(config.max_timeout),
+            ..config
+        };
+        let core_config = config.to_core_config();
+        Self { config, core_config }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &TimeoutConfig {
+    pub fn config(&self) -> &TimeoutManagerPolicy {
         &self.config
     }
 
+    /// Get the derived io-core timeout configuration.
+    pub fn core_config(&self) -> &CoreTimeoutConfig {
+        &self.core_config
+    }
+
     /// Calculate timeout for a given size
     pub fn calculate_timeout(&self, size: u64, _history: &[Duration]) -> Duration {
         if !self.config.enable_dynamic {
             return self.config.default_timeout;
         }
 
-        calculate_adaptive_timeout(self.config.default_timeout, None, 0, size).min(self.config.max_timeout)
+        calculate_adaptive_timeout(self.core_config.base_timeout, None, 0, size)
+            .clamp(self.core_config.min_timeout, self.core_config.max_timeout)
     }
 
     /// Wrap an operation with timeout control
@@ -87,7 +128,7 @@ impl TimeoutManager {
 
     /// Create a timeout guard for manual timeout control
     pub fn create_guard(&self, timeout: Option<Duration>) -> TimeoutGuard {
-        TimeoutGuard::new(timeout.unwrap_or(self.config.default_timeout))
+        TimeoutGuard::new(timeout.unwrap_or(self.core_config.base_timeout))
     }
 }
 
@@ -134,10 +175,41 @@ mod tests {
 
     #[test]
     fn test_timeout_config() {
-        let config = TimeoutConfig::default();
+        let config = TimeoutManagerPolicy::default();
         assert!(config.default_timeout < config.max_timeout);
     }
 
+    #[test]
+    fn test_timeout_policy_to_core_config() {
+        let policy = TimeoutManagerPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.base_timeout, policy.default_timeout);
+        assert_eq!(core.max_timeout, policy.max_timeout);
+        assert_eq!(core.min_timeout, policy.min_timeout);
+        assert_eq!(core.get_object_timeout, policy.default_timeout);
+        assert!(core.enable_dynamic_timeout);
+    }
+
+    #[test]
+    fn test_timeout_manager_new_sanitizes_min_timeout_with_small_max_timeout() {
+        let manager = TimeoutManager::new(Duration::from_secs(1), Duration::from_secs(1), true);
+        let timeout = manager.calculate_timeout(1024, &[]);
+        assert_eq!(timeout, Duration::from_secs(1));
+    }
+
+    #[test]
+    fn test_timeout_manager_from_policy_sanitizes_min_timeout() {
+        let manager = TimeoutManager::from_policy(TimeoutManagerPolicy {
+            default_timeout: Duration::from_secs(30),
+            max_timeout: Duration::from_secs(1),
+            min_timeout: Duration::from_secs(5),
+            enable_dynamic: true,
+        });
+
+        assert_eq!(manager.config().min_timeout, Duration::from_secs(1));
+        assert_eq!(manager.core_config().min_timeout, Duration::from_secs(1));
+    }
+
     #[tokio::test]
     async fn test_wrap_operation_success() {
         let manager = TimeoutManager::new(Duration::from_secs(5), Duration::from_secs(10), true);

crates/concurrency/src/workers.rs

@@ -12,10 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+//! Worker slot limiter used by long-running background workflows.
+
 use std::sync::Arc;
 use tokio::sync::{Mutex, Notify};
 use tracing::info;
 
+/// Cooperative worker-slot controller for async tasks.
 pub struct Workers {
     available: Mutex<usize>, // Available working slots
     notify: Notify,          // Used to notify waiting tasks
@@ -23,20 +26,20 @@ pub struct Workers {
 }
 
 impl Workers {
-    // Create a Workers object that allows up to n jobs to execute concurrently
-    pub fn new(n: usize) -> Result<Arc<Workers>, &'static str> {
+    /// Create a [`Workers`] object that allows up to `n` jobs to execute concurrently.
+    pub fn new(n: usize) -> Result<Arc<Self>, &'static str> {
         if n == 0 {
             return Err("n must be > 0");
         }
 
-        Ok(Arc::new(Workers {
+        Ok(Arc::new(Self {
             available: Mutex::new(n),
             notify: Notify::new(),
             limit: n,
         }))
     }
 
-    // Give a job a chance to be executed
+    /// Acquire a worker slot, waiting until one becomes available.
     pub async fn take(&self) {
         loop {
             let mut available = self.available.lock().await;
@@ -51,7 +54,7 @@ impl Workers {
         }
     }
 
-    // Release a job's slot
+    /// Release a worker slot.
     pub async fn give(&self) {
         let mut available = self.available.lock().await;
         info!("worker give, {}", *available);
@@ -59,7 +62,7 @@ impl Workers {
         self.notify.notify_one(); // Notify a waiting task
     }
 
-    // Wait for all concurrent jobs to complete
+    /// Wait until all worker slots are released.
     pub async fn wait(&self) {
         loop {
             {
@@ -74,6 +77,7 @@ impl Workers {
         info!("worker wait end");
     }
 
+    /// Return the current number of available worker slots.
     pub async fn available(&self) -> usize {
         *self.available.lock().await
     }

crates/config/README.md

@@ -83,6 +83,28 @@ Legacy compatibility fallback:
 - `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION`
   This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.
 
+Drive timeout health-action policy:
+- `RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION`
+  - `mark_failure` (default): timeout marks failure and may transition drive runtime state.
+  - `ignore_scanner`: timeout does not mark failure for scanner-sensitive operations (`walk_dir`, `read_metadata`, `list_dir`, `disk_info`).
+
+Drive timeout profile preset:
+- `RUSTFS_DRIVE_TIMEOUT_PROFILE`
+  - `default` (default): keep current timeout defaults.
+  - `high_latency`: use 60s default timeout for scanner-sensitive operations when no per-operation timeout override is set (`read_metadata`, `disk_info`, `list_dir`, `walk_dir`, `walk_dir_stall`).
+- Precedence:
+  - Explicit per-operation timeout env (`RUSTFS_DRIVE_*_TIMEOUT_SECS`) takes highest precedence.
+  - Then `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION` legacy fallback.
+  - Then the profile-derived default (`default` or `high_latency`).
+
+## Startup filesystem boundary policy
+
+- `RUSTFS_UNSUPPORTED_FS_POLICY` controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
+  - `warn` (default): log warning and continue startup.
+  - `fail`: abort startup with an error.
+
+RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard.
+
 ## 📄 License
 
 This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.

crates/config/src/constants/app.rs

@@ -155,6 +155,22 @@ pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
+/// Environment variable to explicitly allow public default root credentials.
+///
+/// This is intended for local development only. Production startup paths should
+/// provide non-default `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` values.
+pub const ENV_RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS: &str = "RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS";
+
+/// Environment variable controlling startup behavior when unsupported filesystem types are detected.
+///
+/// Accepted values:
+/// - "warn" (default): log a warning and continue startup
+/// - "fail": abort startup with an error
+pub const ENV_RUSTFS_UNSUPPORTED_FS_POLICY: &str = "RUSTFS_UNSUPPORTED_FS_POLICY";
+pub const RUSTFS_UNSUPPORTED_FS_POLICY_WARN: &str = "warn";
+pub const RUSTFS_UNSUPPORTED_FS_POLICY_FAIL: &str = "fail";
+pub const DEFAULT_RUSTFS_UNSUPPORTED_FS_POLICY: &str = RUSTFS_UNSUPPORTED_FS_POLICY_WARN;
+
 /// Environment variable for server OBS endpoint.
 pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 
@@ -293,9 +309,9 @@ pub const DEFAULT_OBS_LOGS_EXPORT_ENABLED: bool = true;
 
 /// Default profiling export enabled
 /// It is used to enable or disable exporting profiles
-/// Default value: false
+/// Default value: true
 /// Environment variable: RUSTFS_OBS_PROFILING_EXPORT_ENABLED
-pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = false;
+pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = true;
 
 /// Default log local logging enabled for rustfs
 /// This is the default log local logging enabled for rustfs.

crates/config/src/constants/drive.rs

@@ -47,6 +47,16 @@ pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;
 pub const ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS";
 pub const DEFAULT_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: u64 = 5;
 
+/// Timeout-to-health transition policy for drive operations.
+///
+/// Accepted values:
+/// - "mark_failure" (default): timeout marks failure and may transition runtime state.
+/// - "ignore_scanner": timeout does not mark failure for scanner-sensitive operations.
+pub const ENV_DRIVE_TIMEOUT_HEALTH_ACTION: &str = "RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION";
+pub const DRIVE_TIMEOUT_HEALTH_ACTION_MARK_FAILURE: &str = "mark_failure";
+pub const DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER: &str = "ignore_scanner";
+pub const DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION: &str = DRIVE_TIMEOUT_HEALTH_ACTION_MARK_FAILURE;
+
 /// Number of consecutive failures before a suspect drive is classified as offline.
 pub const ENV_DRIVE_SUSPECT_FAILURE_THRESHOLD: &str = "RUSTFS_DRIVE_SUSPECT_FAILURE_THRESHOLD";
 pub const DEFAULT_DRIVE_SUSPECT_FAILURE_THRESHOLD: u64 = 2;
@@ -66,3 +76,19 @@ pub const DEFAULT_DRIVE_OFFLINE_GRACE_PERIOD_SECS: u64 = 30;
 /// Duration in seconds after which a recovered drive is classified as long offline.
 pub const ENV_DRIVE_LONG_OFFLINE_THRESHOLD_SECS: &str = "RUSTFS_DRIVE_LONG_OFFLINE_THRESHOLD_SECS";
 pub const DEFAULT_DRIVE_LONG_OFFLINE_THRESHOLD_SECS: u64 = 172_800;
+
+/// Drive timeout profile preset.
+///
+/// Accepted values:
+/// - "default": keep current timeout defaults.
+/// - "high_latency": use a higher default timeout preset for scanner-sensitive and metadata operations.
+///
+/// Explicit per-operation overrides (`RUSTFS_DRIVE_*_TIMEOUT_SECS`) still take precedence.
+pub const ENV_DRIVE_TIMEOUT_PROFILE: &str = "RUSTFS_DRIVE_TIMEOUT_PROFILE";
+pub const DRIVE_TIMEOUT_PROFILE_DEFAULT: &str = "default";
+pub const DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY: &str = "high_latency";
+pub const DEFAULT_DRIVE_TIMEOUT_PROFILE: &str = DRIVE_TIMEOUT_PROFILE_DEFAULT;
+
+/// Timeout preset (seconds) used when `RUSTFS_DRIVE_TIMEOUT_PROFILE=high_latency`
+/// and no per-operation timeout override is provided.
+pub const DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS: u64 = 60;

crates/config/src/constants/internode.rs

@@ -32,6 +32,16 @@ pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 3;
 pub const ENV_INTERNODE_RPC_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS";
 pub const DEFAULT_INTERNODE_RPC_TIMEOUT_SECS: u64 = 10;
 
+/// Environment variable for selecting the internode data-plane transport backend.
+pub const ENV_RUSTFS_INTERNODE_DATA_TRANSPORT: &str = "RUSTFS_INTERNODE_DATA_TRANSPORT";
+pub const DEFAULT_INTERNODE_DATA_TRANSPORT: &str = "tcp-http";
+
+/// Legacy alias for "tcp-http". Both values select the TCP/HTTP transport backend.
+pub const INTERNODE_DATA_TRANSPORT_TCP: &str = "tcp";
+
+/// Known internode transport backend names accepted by the config parser.
+pub const KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS: &[&str] = &[DEFAULT_INTERNODE_DATA_TRANSPORT, INTERNODE_DATA_TRANSPORT_TCP];
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -58,5 +68,9 @@ mod tests {
             "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS"
         );
         assert_eq!(ENV_INTERNODE_RPC_TIMEOUT_SECS, "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS");
+        assert_eq!(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT, "RUSTFS_INTERNODE_DATA_TRANSPORT");
+        assert_eq!(DEFAULT_INTERNODE_DATA_TRANSPORT, "tcp-http");
+        assert_eq!(INTERNODE_DATA_TRANSPORT_TCP, "tcp");
+        assert_eq!(KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS, &["tcp-http", "tcp"]);
     }
 }

crates/config/src/constants/runtime.rs

@@ -59,6 +59,25 @@ pub const DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT: usize = 10;
 pub const DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE: f64 = 1.0; // 100% sampling
 // Note: S3 bucket/prefix have no default; absence means upload is disabled (modeled as Option<String>)
 
+/// Maximum transition workers used as a local fallback when runtime env is unset.
+pub const DEFAULT_TRANSITION_WORKERS_CAP: i64 = 16;
+/// Absolute upper bound for transition workers accepted from runtime env.
+pub const DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX: i64 = 32;
+/// Default capacity for the transition queue.
+pub const DEFAULT_TRANSITION_QUEUE_CAPACITY: usize = 1000;
+/// Default send timeout for transition queue enqueue attempts, in milliseconds.
+pub const DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS: usize = 100;
+/// Test-only fault injection env var that forces the immediate transition enqueue timeout path.
+pub const ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT: &str = "RUSTFS_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT";
+/// Runtime env var controlling the transition worker count.
+pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
+/// Runtime env var controlling the absolute maximum transition workers.
+pub const ENV_TRANSITION_WORKERS_ABSOLUTE_MAX: &str = "RUSTFS_ABSOLUTE_MAX_WORKERS";
+/// Runtime env var controlling the transition queue capacity.
+pub const ENV_TRANSITION_QUEUE_CAPACITY: &str = "RUSTFS_TRANSITION_QUEUE_CAPACITY";
+/// Runtime env var controlling the transition queue send timeout in milliseconds.
+pub const ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS: &str = "RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS";
+
 // Allocator reclaim configuration
 pub const ENV_ALLOCATOR_RECLAIM_ENABLED: &str = "RUSTFS_ALLOCATOR_RECLAIM_ENABLED";
 pub const ENV_ALLOCATOR_RECLAIM_INTERVAL_SECS: &str = "RUSTFS_ALLOCATOR_RECLAIM_INTERVAL_SECS";

crates/config/src/constants/scanner.rs

@@ -40,6 +40,37 @@ pub const ENV_SCANNER_SPEED: &str = "RUSTFS_SCANNER_SPEED";
 /// Default scanner speed preset.
 pub const DEFAULT_SCANNER_SPEED: &str = "default";
 
+/// Environment variable that specifies the periodic bitrot scan cycle in seconds.
+/// When set to `0`, `true`, `on`, or `yes`, every scanner cycle runs in deep mode.
+/// When set to `false`, `off`, `no`, or `disabled`, periodic deep scans are disabled.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_BITROT_CYCLE_SECS=2592000` (30 days)
+pub const ENV_SCANNER_BITROT_CYCLE_SECS: &str = "RUSTFS_SCANNER_BITROT_CYCLE_SECS";
+
+/// Default bitrot scan cycle used by the scanner.
+pub const DEFAULT_SCANNER_BITROT_CYCLE_SECS: u64 = 30 * 24 * 60 * 60;
+
+/// Environment variable that controls how many object versions trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS=100`
+pub const ENV_SCANNER_ALERT_EXCESS_VERSIONS: &str = "RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS";
+
+/// Default object version count that triggers scanner alerts.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS: u64 = 100;
+
+/// Environment variable that controls how many cumulative bytes across object versions trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE=1099511627776`
+pub const ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE";
+
+/// Default cumulative object version bytes that trigger scanner alerts.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE: u64 = 1024 * 1024 * 1024 * 1024;
+
+/// Environment variable that controls how many subfolders trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS=50000`
+pub const ENV_SCANNER_ALERT_EXCESS_FOLDERS: &str = "RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS";
+
+/// Default subfolder count that triggers scanner alerts.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS: u64 = 50_000;
+
 /// Environment variable that controls whether the scanner sleeps between operations.
 /// When `true` (default), the scanner throttles itself. When `false`, it runs at full speed.
 /// - Example: `export RUSTFS_SCANNER_IDLE_MODE=false`

crates/credentials/Cargo.toml

@@ -1,3 +1,17 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 [package]
 name = "rustfs-credentials"
 edition.workspace = true
@@ -12,9 +26,11 @@ categories = ["web-programming", "development-tools", "data-structures", "securi
 
 [dependencies]
 base64-simd = { workspace = true }
+hmac = { workspace = true }
 rand = { workspace = true }
 serde = { workspace = true }
 serde_json.workspace = true
+sha2 = { workspace = true }
 time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }
 
 [lints]

crates/credentials/src/credentials.rs

@@ -12,10 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::{DEFAULT_SECRET_KEY, ENV_RPC_SECRET, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+use crate::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ENV_RPC_SECRET, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+use base64_simd::URL_SAFE_NO_PAD;
+use hmac::{Hmac, KeyInit, Mac};
 use rand::{Rng, RngExt};
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
+use sha2::Sha256;
 use std::collections::HashMap;
 use std::env;
 use std::fmt;
@@ -33,7 +36,12 @@ pub static GLOBAL_RUSTFS_RPC_SECRET: OnceLock<String> = OnceLock::new();
 pub const RPC_SECRET_REQUIRED_MESSAGE: &str = "RPC authentication secret is not configured";
 
 /// Operator-facing guidance for configuring RPC authentication safely.
-pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str = "RUSTFS_RPC_SECRET must be set to a non-default value or RUSTFS_SECRET_KEY must be changed from the default for RPC authentication";
+pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str =
+    "RUSTFS_RPC_SECRET can be set explicitly; otherwise the RPC secret is derived from the active access/secret key pair";
+
+type HmacSha256 = Hmac<Sha256>;
+
+const RPC_SECRET_DERIVATION_CONTEXT: &[u8] = b"rustfs-rpc-secret:v1";
 
 /// Error type for credentials operations
 #[derive(Debug)]
@@ -217,37 +225,59 @@ pub fn gen_secret_key(length: usize) -> std::io::Result<String> {
     Ok(encoded)
 }
 
-/// Get the RPC authentication token from environment variable
+/// Get the RPC authentication token from the environment or derive it from the active credentials.
 ///
 /// # Returns
 /// * `String` - The RPC authentication token
 ///
-fn resolve_rpc_secret(env_secret: Option<&str>, global_secret: Option<&str>) -> Option<String> {
-    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
-        return (secret != DEFAULT_SECRET_KEY).then(|| secret.to_string());
+fn normalize_rpc_secret(secret: &str) -> Option<String> {
+    let secret = secret.trim();
+    (!secret.is_empty() && secret != DEFAULT_SECRET_KEY && secret != DEFAULT_ACCESS_KEY).then(|| secret.to_string())
+}
+
+fn derive_rpc_secret(access_key: &str, secret_key: &str) -> Option<String> {
+    let access_key = access_key.trim();
+    let secret_key = secret_key.trim();
+
+    if access_key.is_empty() || secret_key.is_empty() {
+        return None;
     }
 
-    global_secret
-        .map(str::trim)
-        .filter(|secret| !secret.is_empty() && *secret != DEFAULT_SECRET_KEY)
-        .map(ToOwned::to_owned)
+    let mut mac = <HmacSha256 as KeyInit>::new_from_slice(secret_key.as_bytes()).expect("HMAC can take key of any size");
+    mac.update(RPC_SECRET_DERIVATION_CONTEXT);
+    mac.update(&[0]);
+    mac.update(access_key.as_bytes());
+
+    Some(URL_SAFE_NO_PAD.encode_to_string(mac.finalize().into_bytes()))
+}
+
+fn resolve_rpc_secret(env_secret: Option<&str>, global_access: Option<&str>, global_secret: Option<&str>) -> Option<String> {
+    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
+        return normalize_rpc_secret(secret);
+    }
+
+    match (global_access, global_secret) {
+        (Some(access_key), Some(secret_key)) => derive_rpc_secret(access_key, secret_key),
+        _ => None,
+    }
 }
 
 pub fn try_get_rpc_token() -> std::io::Result<String> {
     if let Some(secret) = GLOBAL_RUSTFS_RPC_SECRET.get() {
-        return resolve_rpc_secret(None, Some(secret)).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
+        return normalize_rpc_secret(secret).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
     }
 
     let env_secret = env::var(ENV_RPC_SECRET).ok();
+    let global_access = get_global_access_key_opt();
     let global_secret = get_global_secret_key_opt();
-    let secret = resolve_rpc_secret(env_secret.as_deref(), global_secret.as_deref())
+    let secret = resolve_rpc_secret(env_secret.as_deref(), global_access.as_deref(), global_secret.as_deref())
         .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE))?;
 
     match GLOBAL_RUSTFS_RPC_SECRET.set(secret.clone()) {
         Ok(()) => Ok(secret),
         Err(_) => GLOBAL_RUSTFS_RPC_SECRET
             .get()
-            .and_then(|stored| resolve_rpc_secret(None, Some(stored)))
+            .and_then(|stored| normalize_rpc_secret(stored))
             .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE)),
     }
 }
@@ -407,7 +437,7 @@ impl Credentials {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::{IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+    use crate::{DEFAULT_ACCESS_KEY, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
     use time::Duration;
 
     #[test]
@@ -510,11 +540,12 @@ mod tests {
         // If it hasn't already been initialized, the test automatically generates logic
         if get_global_action_cred().is_none() {
             init_global_action_credentials(None, None).ok();
-            let ak = get_global_access_key();
-            let sk = get_global_secret_key();
-            assert_eq!(ak.len(), 20);
-            assert_eq!(sk.len(), 32);
         }
+
+        let ak = get_global_access_key();
+        let sk = get_global_secret_key();
+        assert!(ak.len() >= 3);
+        assert!(sk.len() >= 8);
     }
 
     #[test]
@@ -528,10 +559,26 @@ mod tests {
     }
 
     #[test]
-    fn test_resolve_rpc_secret_rejects_default_fallback() {
-        assert!(resolve_rpc_secret(None, None).is_none());
-        assert!(resolve_rpc_secret(None, Some(DEFAULT_SECRET_KEY)).is_none());
-        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-global-secret")).is_none());
+    fn test_gen_access_key_length_and_charset() {
+        let err = gen_access_key(2).expect_err("length below 3 should fail");
+        assert_eq!(err.to_string(), "access key length is too short");
+
+        let key = gen_access_key(20).expect("access key should generate");
+        assert_eq!(key.len(), 20);
+        assert!(key.chars().all(|ch| ch.is_ascii_uppercase() || ch.is_ascii_digit()));
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_allows_default_credentials_for_derivation() {
+        assert!(resolve_rpc_secret(None, None, None).is_none());
+
+        let expected = derive_rpc_secret(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY).expect("secret should derive");
+        assert_eq!(
+            resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).as_deref(),
+            Some(expected.as_str())
+        );
+
+        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-access"), Some("custom-global-secret")).is_none());
     }
 
     #[test]
@@ -551,37 +598,64 @@ mod tests {
 
     #[test]
     fn test_resolve_rpc_secret_accepts_non_default_secret() {
-        assert_eq!(resolve_rpc_secret(Some("custom-rpc-secret"), None).as_deref(), Some("custom-rpc-secret"));
         assert_eq!(
-            resolve_rpc_secret(None, Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("custom-rpc-secret"), None, None).as_deref(),
+            Some("custom-rpc-secret")
+        );
+        let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
+        assert_eq!(
+            resolve_rpc_secret(None, Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
     }
 
     #[test]
     fn test_resolve_rpc_secret_trims_and_falls_back_from_blank_env() {
+        let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
         assert_eq!(
-            resolve_rpc_secret(Some("  custom-rpc-secret  "), None).as_deref(),
+            resolve_rpc_secret(Some("  custom-rpc-secret  "), None, None).as_deref(),
             Some("custom-rpc-secret")
         );
         assert_eq!(
-            resolve_rpc_secret(Some(""), Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some(""), Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
         assert_eq!(
-            resolve_rpc_secret(Some("  "), Some("  custom-global-secret  ")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("  "), Some("  custom-access  "), Some("  custom-global-secret  ")).as_deref(),
+            Some(expected.as_str())
         );
         assert_eq!(
-            resolve_rpc_secret(Some("  "), Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("  "), Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
     }
 
     #[test]
     fn test_resolve_rpc_secret_returns_none_for_trimmed_default_secret() {
         let padded_default_secret = format!("  {}  ", DEFAULT_SECRET_KEY);
-        assert!(resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-global-secret")).is_none());
+        assert!(
+            resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-access"), Some("custom-global-secret"))
+                .is_none()
+        );
+
+        let padded_default_access = format!("  {}  ", DEFAULT_ACCESS_KEY);
+        assert!(
+            resolve_rpc_secret(Some(padded_default_access.as_str()), Some("custom-access"), Some("custom-global-secret"))
+                .is_none()
+        );
+    }
+
+    #[test]
+    fn test_derive_rpc_secret_is_stable_and_not_plaintext() {
+        let first = derive_rpc_secret("custom-access", "custom-secret").expect("secret should derive");
+        let second = derive_rpc_secret("custom-access", "custom-secret").expect("secret should derive");
+
+        assert_eq!(first, second);
+        assert_ne!(first, "custom-access");
+        assert_ne!(first, "custom-secret");
+        assert_ne!(first, format!("{}{}", "custom-access", "custom-secret"));
+        assert!(!first.contains("custom-access"));
+        assert!(!first.contains("custom-secret"));
     }
 
     #[test]

crates/crypto/Cargo.toml

@@ -33,8 +33,11 @@ aes-gcm = { workspace = true, optional = true }
 argon2 = { workspace = true, optional = true }
 chacha20poly1305 = { workspace = true, optional = true }
 jsonwebtoken = { workspace = true }
+base64-simd = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }
 rand = { workspace = true, optional = true }
+rsa = { workspace = true, features = ["sha2"] }
+serde = { workspace = true, features = ["derive"] }
 sha2 = { workspace = true, optional = true }
 thiserror.workspace = true
 serde_json.workspace = true

crates/crypto/src/lib.rs

@@ -16,6 +16,7 @@
 mod encdec;
 mod error;
 mod jwt;
+pub mod license_token;
 
 pub use encdec::decrypt::decrypt_data;
 pub use encdec::encrypt::encrypt_data;
@@ -25,3 +26,4 @@ pub use encdec::stream_io::{decrypt_stream_io, encrypt_stream_io};
 pub use error::Error;
 pub use jwt::decode::decode as jwt_decode;
 pub use jwt::encode::encode as jwt_encode;
+pub use license_token::{Token, parse_license_with_public_key, parse_signed_license_token, sign_license_token};

crates/crypto/src/license_token.rs

@@ -115,12 +115,20 @@ mod tests {
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
 
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600, // 1 hour from now
         };
 
         let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
@@ -139,12 +147,20 @@ mod tests {
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
 
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600,
         };
 
         let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
@@ -159,11 +175,19 @@ mod tests {
         let mut rng = rand::rng();
         let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600,
         };
 
         let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
@@ -181,7 +205,7 @@ mod tests {
 
     #[test]
     fn test_source_does_not_embed_private_key() {
-        let source = include_str!("token.rs");
+        let source = include_str!("license_token.rs");
         let forbidden = ["BEGIN", "PRIVATE KEY"].join(" ");
 
         assert!(!source.contains(&forbidden));
@@ -192,7 +216,9 @@ mod tests {
         let mut rng = rand::rng();
         let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let invalid_token = "invalid_base64_token";
         let result = parse_signed_license_token(invalid_token, &public_key_pem);
@@ -204,7 +230,11 @@ mod tests {
     fn test_sign_license_token_with_invalid_signing_key() {
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600, // 1 hour from now
         };
 
         let invalid_key = "invalid_private_key";

crates/data-usage/Cargo.toml

@@ -13,24 +13,26 @@
 # limitations under the License.
 
 [package]
-name = "rustfs-workers"
+name = "rustfs-data-usage"
+version.workspace = true
 edition.workspace = true
 license.workspace = true
 repository.workspace = true
 rust-version.workspace = true
-version.workspace = true
 homepage.workspace = true
-description = "Workers for RustFS, providing background processing capabilities for tasks such as data synchronization, indexing, and more."
-keywords = ["workers", "tasks", "rustfs", "Minio"]
-categories = ["web-programming", "development-tools"]
-documentation = "https://docs.rs/rustfs-workers/latest/rustfs_workers/"
+description = "Shared data usage models and algorithms for RustFS"
+keywords = ["rustfs", "data-usage", "cache", "histogram"]
+categories = ["data-structures", "filesystem"]
 
 [lints]
 workspace = true
 
 [dependencies]
-tokio = { workspace = true, features = ["sync", "time", "macros"] }
-tracing.workspace = true
+serde = { workspace = true }
+path-clean = { workspace = true }
+rmp-serde = { workspace = true }
+async-trait = { workspace = true }
+rustfs-filemeta = { workspace = true }
 
 [lib]
 doctest = false

crates/data-usage/src/data_usage.rs

@@ -312,6 +312,12 @@ impl SizeHistogram {
         }
         res
     }
+
+    pub fn merge_from(&mut self, other: &Self) {
+        for (dst, src) in self.0.iter_mut().zip(other.0.iter()) {
+            *dst += src;
+        }
+    }
 }
 
 /// Versions histogram for version count distribution
@@ -361,6 +367,12 @@ impl VersionsHistogram {
         }
         res
     }
+
+    pub fn merge_from(&mut self, other: &Self) {
+        for (dst, src) in self.0.iter_mut().zip(other.0.iter()) {
+            *dst += src;
+        }
+    }
 }
 
 /// Replication statistics for a single target
@@ -419,6 +431,9 @@ pub struct DataUsageEntry {
     pub obj_versions: VersionsHistogram,
     pub replication_stats: Option<ReplicationAllStats>,
     pub compacted: bool,
+    /// Number of objects that failed to scan (e.g., IO errors)
+    #[serde(default)]
+    pub failed_objects: usize,
 }
 
 impl DataUsageEntry {
@@ -456,6 +471,7 @@ impl DataUsageEntry {
         self.versions += other.versions;
         self.delete_markers += other.delete_markers;
         self.size += other.size;
+        self.failed_objects += other.failed_objects;
 
         if let Some(o_rep) = &other.replication_stats {
             let s_rep = self.replication_stats.get_or_insert_with(ReplicationAllStats::default);
@@ -490,9 +506,11 @@ impl DataUsageEntry {
 #[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageCacheInfo {
     pub name: String,
-    pub next_cycle: u32,
+    pub next_cycle: u64,
     pub last_update: Option<SystemTime>,
     pub skip_healing: bool,
+    #[serde(default)]
+    pub failed_objects: HashMap<String, u64>,
 }
 
 /// Data usage cache

crates/data-usage/src/lib.rs

@@ -12,4 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-pub mod workers;
+pub mod data_usage;
+
+pub use data_usage::*;

crates/e2e_test/Cargo.toml

@@ -31,7 +31,7 @@ sftp = []
 [dependencies]
 rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-ecstore.workspace = true
-rustfs-common.workspace = true
+rustfs-data-usage.workspace = true
 rustfs-rio.workspace = true
 flatbuffers.workspace = true
 futures.workspace = true

crates/e2e_test/src/data_usage_test.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use aws_sdk_s3::primitives::ByteStream;
-use rustfs_common::data_usage::DataUsageInfo;
+use rustfs_data_usage::DataUsageInfo;
 use serial_test::serial;
 
 use crate::common::{RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -0,0 +1,212 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Erasure-set healing regression tests.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, execute_awscurl, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use serial_test::serial;
+    use std::collections::HashSet;
+    use std::path::{Path, PathBuf};
+    use tokio::time::{Duration, sleep};
+    use tracing::info;
+
+    fn has_file_under(path: &Path) -> bool {
+        let Ok(entries) = std::fs::read_dir(path) else {
+            return false;
+        };
+
+        for entry in entries.filter_map(Result::ok) {
+            let path = entry.path();
+            if path.is_dir() {
+                if has_file_under(&path) {
+                    return true;
+                }
+            } else {
+                return true;
+            }
+        }
+
+        false
+    }
+
+    fn object_metadata_exists_on_disk(disk: &Path, bucket: &str, key: &str) -> bool {
+        disk.join(bucket).join(key).join("xl.meta").is_file()
+    }
+
+    async fn assert_object_body(env: &RustFSTestEnvironment, bucket: &str, key: &str, expected: &[u8]) {
+        let client = env.create_s3_client();
+        let response = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("GET should succeed during/after heal");
+        let body = response.body.collect().await.expect("GET body should collect").into_bytes();
+        assert_eq!(body.as_ref(), expected, "object body changed for {key}");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_admin_deep_heal_rebuilds_cleared_disk_in_single_node_erasure_set() {
+        init_logging();
+        info!("Discussion #2964: admin deep heal should rebuild a wiped disk in a 4-disk single-node erasure set");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        let root = PathBuf::from(env.temp_dir.clone());
+        let disk0 = root.join("disk0");
+        let disk1 = root.join("disk1");
+        let disk2 = root.join("disk2");
+        let disk3 = root.join("disk3");
+        for disk in [&disk0, &disk1, &disk2, &disk3] {
+            std::fs::create_dir_all(disk).expect("disk directory should be created");
+        }
+
+        // The test helper always appends env.temp_dir as the final storage path.
+        // Point it at disk3 and pass the other three disks explicitly.
+        env.temp_dir = disk3.to_string_lossy().to_string();
+        let disk0_arg = disk0.to_string_lossy().to_string();
+        let disk1_arg = disk1.to_string_lossy().to_string();
+        let disk2_arg = disk2.to_string_lossy().to_string();
+        env.start_rustfs_server_with_env(
+            vec![disk0_arg.as_str(), disk1_arg.as_str(), disk2_arg.as_str()],
+            &[("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true")],
+        )
+        .await
+        .expect("Failed to start 4-disk RustFS");
+
+        let client = env.create_s3_client();
+        let bucket = "heal-cleared-disk-regression";
+        let target_object_count = std::env::var("RUSTFS_HEAL_REBUILD_OBJECT_COUNT")
+            .ok()
+            .and_then(|value| value.parse::<usize>().ok())
+            .unwrap_or(4)
+            .max(4);
+        let heal_timeout_secs = std::env::var("RUSTFS_HEAL_REBUILD_TIMEOUT_SECS")
+            .ok()
+            .and_then(|value| value.parse::<u64>().ok())
+            .unwrap_or(60);
+
+        let mut objects: Vec<(String, Vec<u8>, &'static str)> = vec![
+            (
+                "中文/报告-0001.json".to_string(),
+                "{\"message\":\"hello 中文\"}".as_bytes().to_vec(),
+                "application/json",
+            ),
+            (
+                "english/images/photo-0002.jpg".to_string(),
+                vec![0xff, 0xd8, 0xff, 0x00, 0x42, 0x24],
+                "image/jpeg",
+            ),
+            (
+                "mixed/空 格 + symbols @#%.txt".to_string(),
+                b"text object with spaces and symbols".to_vec(),
+                "text/plain; charset=utf-8",
+            ),
+            (
+                "bin/archive-0004.bin".to_string(),
+                (0..=255).collect::<Vec<u8>>(),
+                "application/octet-stream",
+            ),
+        ];
+        for index in objects.len()..target_object_count {
+            objects.push((
+                format!("bulk/prefix-{}/object-{index:04}.txt", index % 17),
+                format!("bulk object {index}: heal regression payload").into_bytes(),
+                "text/plain; charset=utf-8",
+            ));
+        }
+
+        let object_keys = objects.iter().map(|(key, _, _)| key.clone()).collect::<Vec<_>>();
+        let mut remaining_rebuild_keys: HashSet<String> = object_keys.iter().cloned().collect();
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("bucket create should succeed");
+        for (key, body, content_type) in &objects {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .content_type(*content_type)
+                .body(ByteStream::from(body.clone()))
+                .send()
+                .await
+                .expect("PUT should succeed");
+        }
+
+        assert!(has_file_under(&disk0), "disk0 should contain object shards before wipe");
+        env.stop_server();
+
+        std::fs::remove_dir_all(&disk0).expect("disk0 wipe should succeed");
+        std::fs::create_dir_all(&disk0).expect("disk0 should be recreated empty");
+        assert!(!has_file_under(&disk0), "disk0 must be empty before restart");
+
+        env.start_rustfs_server_with_env(
+            vec![disk0_arg.as_str(), disk1_arg.as_str(), disk2_arg.as_str()],
+            &[("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true")],
+        )
+        .await
+        .expect("Failed to restart 4-disk RustFS after disk wipe");
+        // The helper's Drop cleanup removes env.temp_dir. Reset it to the parent
+        // directory after server startup so all four disk directories are cleaned
+        // without manually deleting a path Drop will also try to remove.
+        env.temp_dir = root.to_string_lossy().to_string();
+
+        let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
+        let heal_url = format!("{}/rustfs/admin/v3/heal/{}?forceStart=true", env.url, bucket);
+        execute_awscurl(&heal_url, "POST", Some(heal_body), &env.access_key, &env.secret_key)
+            .await
+            .expect("admin deep heal should be accepted");
+
+        for _ in 0..heal_timeout_secs {
+            if !remaining_rebuild_keys.is_empty() {
+                let mut rebuilt = Vec::new();
+                for key in &remaining_rebuild_keys {
+                    if object_metadata_exists_on_disk(&disk0, bucket, key) {
+                        rebuilt.push(key.clone());
+                    }
+                }
+                for key in rebuilt {
+                    let _ = remaining_rebuild_keys.remove(&key);
+                }
+            }
+
+            if remaining_rebuild_keys.is_empty() {
+                for (key, body, _) in &objects {
+                    assert_object_body(&env, bucket, key, body).await;
+                }
+
+                env.stop_server();
+                for key in &object_keys {
+                    assert!(
+                        object_metadata_exists_on_disk(&disk0, bucket, key),
+                        "wiped disk should contain rebuilt xl.meta for {key}"
+                    );
+                }
+                return;
+            }
+
+            sleep(Duration::from_secs(1)).await;
+        }
+
+        panic!("admin deep heal did not rebuild all files on the wiped disk within timeout");
+    }
+}

crates/e2e_test/src/kms/README.md

@@ -244,9 +244,9 @@ Designed to run inside CI/CD pipelines:
 
 ## 📚 References
 
-- [KMS configuration guide](../../../../docs/kms/README.md)
-- [Dynamic configuration API](../../../../docs/kms/http-api.md)
-- [Troubleshooting](../../../../docs/kms/troubleshooting.md)
+- [KMS configuration types](../../../kms/src/config.rs)
+- [Dynamic configuration API handlers](../../../../rustfs/src/admin/handlers/kms_dynamic.rs)
+- [KMS management API handlers](../../../../rustfs/src/admin/handlers/kms_management.rs)
 
 ---
 

crates/e2e_test/src/lib.rs

@@ -114,6 +114,9 @@ mod head_object_range_test;
 #[cfg(test)]
 mod head_object_consistency_test;
 
+#[cfg(test)]
+mod heal_erasure_disk_rebuild_test;
+
 #[cfg(test)]
 mod copy_object_metadata_test;
 

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -612,4 +612,421 @@ mod tests {
 
         env.stop_server();
     }
+
+    /// Regression test: delimiter listing must not produce false-positive truncation
+    /// when many raw keys collapse into a small number of CommonPrefixes.
+    ///
+    /// Scenario: 30 objects across 3 directories + 2 direct files under prefix.
+    /// With max_keys=1000, all 5 visible results (3 prefixes + 2 objects) fit in one
+    /// page, so IsTruncated must be false even though raw entry count is much larger.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_delimiter_collapsed_prefix_no_false_truncation() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 delimiter collapsed-prefix no false truncation");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-collapsed-prefix";
+        let prefix = "data/";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create objects under 3 subdirectories (10 each = 30 raw keys)
+        let dirs = ["alpha/", "beta/", "gamma/"];
+        let mut expected_keys = Vec::new();
+        for dir in &dirs {
+            for i in 0..10 {
+                let key = format!("{prefix}{dir}file{i:02}.txt");
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                expected_keys.push(key);
+            }
+        }
+        // Add 2 direct objects under prefix
+        for name in ["readme.txt", "config.json"] {
+            let key = format!("{prefix}{name}");
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(&key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put object");
+            expected_keys.push(key);
+        }
+
+        // List with delimiter="/", max_keys large enough to cover all visible results
+        // Visible: 3 common prefixes + 2 direct objects = 5
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .prefix(prefix)
+            .delimiter("/")
+            .max_keys(1000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        let listed_keys: Vec<String> = output
+            .contents()
+            .iter()
+            .filter_map(|obj| obj.key())
+            .map(|k| k.to_string())
+            .collect();
+        let listed_prefixes: Vec<String> = output
+            .common_prefixes()
+            .iter()
+            .filter_map(|cp| cp.prefix())
+            .map(|p| p.to_string())
+            .collect();
+
+        // KEY ASSERTION: IsTruncated must be false because all visible results fit within max_keys
+        let is_truncated = output.is_truncated().unwrap_or(false);
+        assert!(
+            !is_truncated,
+            "BUG: IsTruncated should be false when all visible results ({} objects + {} prefixes = {}) fit within max_keys (1000)",
+            listed_keys.len(),
+            listed_prefixes.len(),
+            listed_keys.len() + listed_prefixes.len()
+        );
+
+        assert!(
+            output.next_continuation_token().is_none(),
+            "NextContinuationToken should be None when IsTruncated is false"
+        );
+
+        // All 32 objects must be covered (listed_keys + keys under listed_prefixes)
+        let total_raw_count = expected_keys.len();
+        let keys_under_prefixes: usize = listed_prefixes
+            .iter()
+            .map(|p| {
+                let dir = p.trim_end_matches('/');
+                expected_keys.iter().filter(|k| k.starts_with(&format!("{dir}/"))).count()
+            })
+            .sum();
+        let covered = listed_keys.len() + keys_under_prefixes;
+
+        assert_eq!(
+            covered,
+            total_raw_count,
+            "Collapsed-prefix listing must cover all {} objects, got {} (keys={}, under_prefixes={})",
+            total_raw_count,
+            covered,
+            listed_keys.len(),
+            keys_under_prefixes
+        );
+
+        info!(
+            "Collapsed-prefix test passed: {} objects covered ({} keys + {} prefixes), IsTruncated=false",
+            covered,
+            listed_keys.len(),
+            listed_prefixes.len()
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: delimiter listing pagination must traverse all keys
+    /// even when the visible result count per page is much smaller than the
+    /// raw entry count (due to CommonPrefix collapse).
+    ///
+    /// Scenario: 1000 objects across 100 directories, paginated with max_keys=50.
+    /// Each page returns up to 50 CommonPrefixes. The server must correctly set
+    /// IsTruncated and provide a valid continuation token across all pages.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_delimiter_small_page_traverses_all() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 delimiter small page traverses all keys");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-delimiter-small-page";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create 1000 objects: 100 directories with 10 files each
+        let mut all_keys = Vec::new();
+        let dirs: Vec<String> = (0..100).map(|i| format!("dir-{:03}/", i)).collect();
+        for dir in &dirs {
+            for i in 0..10 {
+                let key = format!("{dir}file{:02}.txt", i);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                all_keys.push(key);
+            }
+        }
+
+        // Paginate with delimiter="/" and max_keys=50
+        // Visible per page: up to 50 CommonPrefixes
+        let mut listed_keys = Vec::new();
+        let mut listed_prefixes = Vec::new();
+        let mut continuation_token: Option<String> = None;
+        let mut page_count = 0;
+        let mut last_page_is_truncated: bool;
+
+        loop {
+            let mut request = client.list_objects_v2().bucket(bucket).delimiter("/").max_keys(50);
+
+            if let Some(token) = continuation_token.take() {
+                request = request.continuation_token(token);
+            }
+
+            let output = request.send().await.expect("Failed to list objects");
+            last_page_is_truncated = output.is_truncated().unwrap_or(false);
+
+            for obj in output.contents() {
+                if let Some(key) = obj.key() {
+                    listed_keys.push(key.to_string());
+                }
+            }
+            for cp in output.common_prefixes() {
+                if let Some(p) = cp.prefix() {
+                    listed_prefixes.push(p.to_string());
+                }
+            }
+
+            page_count += 1;
+
+            if last_page_is_truncated {
+                continuation_token = output.next_continuation_token().map(|s| s.to_string());
+                assert!(
+                    continuation_token.is_some(),
+                    "BUG: NextContinuationToken must be present when IsTruncated is true"
+                );
+            } else {
+                break;
+            }
+
+            if page_count > 20 {
+                panic!("Too many pages, possible infinite loop in delimiter pagination");
+            }
+        }
+
+        // Last page must have IsTruncated=false
+        assert!(
+            !last_page_is_truncated,
+            "BUG: Last page must have IsTruncated=false after all results returned"
+        );
+
+        // Verify all objects are covered via listed prefixes
+        let keys_under_prefixes: HashSet<String> = all_keys
+            .iter()
+            .filter(|k| {
+                listed_prefixes
+                    .iter()
+                    .any(|p| k.starts_with(&format!("{}/", p.trim_end_matches('/'))))
+            })
+            .cloned()
+            .collect();
+        let listed_set: HashSet<String> = listed_keys.iter().cloned().collect();
+        let covered: HashSet<String> = listed_set.union(&keys_under_prefixes).cloned().collect();
+        let expected_set: HashSet<String> = all_keys.iter().cloned().collect();
+
+        assert_eq!(
+            covered,
+            expected_set,
+            "Delimiter pagination must cover all {} objects, missing: {:?}",
+            expected_set.difference(&covered).count(),
+            expected_set.difference(&covered)
+        );
+
+        info!(
+            "Delimiter small-page test passed: {} objects covered in {} pages",
+            covered.len(),
+            page_count
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: raw entries exceed MaxKeys but all collapse into fewer
+    /// visible CommonPrefixes — IsTruncated must be false because there are no
+    /// more visible results beyond the collapsed prefixes.
+    ///
+    /// Scenario: 10 directories with 200 files each = 2000 raw keys.
+    /// With MaxKeys=1000, the raw entry count exceeds MaxKeys (disk_has_more=true),
+    /// but after delimiter collapse only 10 CommonPrefixes are visible (10 < 1000).
+    /// IsTruncated must be false since there are no additional visible results.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_raw_exceeds_maxkeys_but_visible_below() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 raw > MaxKeys but visible < MaxKeys after collapse");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-raw-exceeds-visible-below";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create 10 directories × 200 files = 2000 raw keys
+        let dir_count = 10;
+        let files_per_dir = 200;
+        let mut all_keys = Vec::new();
+        for d in 0..dir_count {
+            let dir = format!("dir-{:02}/", d);
+            for f in 0..files_per_dir {
+                let key = format!("{}file{:03}.txt", dir, f);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                all_keys.push(key);
+            }
+        }
+
+        // List with delimiter="/", max_keys=1000
+        // Visible: 10 CommonPrefixes (dir-00/ .. dir-09/) = 10 visible << 1000 MaxKeys
+        // But raw entry count (2000+1001 requested) exceeds MaxKeys
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .delimiter("/")
+            .max_keys(1000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        let listed_keys: Vec<String> = output
+            .contents()
+            .iter()
+            .filter_map(|obj| obj.key())
+            .map(|k| k.to_string())
+            .collect();
+        let listed_prefixes: Vec<String> = output
+            .common_prefixes()
+            .iter()
+            .filter_map(|cp| cp.prefix())
+            .map(|p| p.to_string())
+            .collect();
+
+        // KEY ASSERTION: IsTruncated must be false because visible results (10)
+        // are far below MaxKeys (1000), even though raw entries exceeded MaxKeys.
+        let is_truncated = output.is_truncated().unwrap_or(false);
+        assert!(
+            !is_truncated,
+            "BUG: IsTruncated should be false when visible results ({} objects + {} prefixes = {}) < MaxKeys (1000), even if raw entries exceeded MaxKeys",
+            listed_keys.len(),
+            listed_prefixes.len(),
+            listed_keys.len() + listed_prefixes.len()
+        );
+
+        assert!(
+            output.next_continuation_token().is_none(),
+            "NextContinuationToken should be None when IsTruncated is false"
+        );
+
+        // All objects must be covered by listed prefixes
+        assert_eq!(
+            listed_prefixes.len(),
+            dir_count,
+            "Expected {} prefixes, got {}",
+            dir_count,
+            listed_prefixes.len()
+        );
+        for d in 0..dir_count {
+            let prefix = format!("dir-{:02}/", d);
+            assert!(listed_prefixes.contains(&prefix), "Missing prefix: {}", prefix);
+        }
+
+        info!(
+            "Raw-exceeds-visible test passed: {} raw keys → {} prefixes, IsTruncated=false",
+            all_keys.len(),
+            listed_prefixes.len()
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: pagination with MaxKeys exceeding S3 limit (1000) and delimiter.
+    /// The server caps MaxKeys to 1000. With delimiter="/", many raw keys collapse into
+    /// few CommonPrefixes. Visible results (prefixes) < capped MaxKeys → IsTruncated=false.
+    ///
+    /// This complements test_list_objects_v2_max_keys_above_limit_returns_token which
+    /// tests the non-delimiter case.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_maxkeys_above_limit_with_delimiter() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 MaxKeys above limit with delimiter");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-maxkeys-limit-delimiter";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // 12 dirs × 100 files = 1200 raw keys
+        let dir_count = 12;
+        let files_per_dir = 100;
+        for d in 0..dir_count {
+            for f in 0..files_per_dir {
+                let key = format!("dir-{:02}/file{:03}.txt", d, f);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+            }
+        }
+
+        // With delimiter: 12 CommonPrefixes visible, all fit within capped 1000
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .delimiter("/")
+            .max_keys(2000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        assert_eq!(
+            output.common_prefixes().len(),
+            dir_count,
+            "Expected {} prefixes, got {}",
+            dir_count,
+            output.common_prefixes().len()
+        );
+        assert_eq!(output.max_keys(), Some(1000));
+        // 12 visible < 1000 capped MaxKeys → not truncated
+        assert!(
+            !output.is_truncated().unwrap_or(false),
+            "BUG: IsTruncated should be false when visible ({}) < capped MaxKeys (1000)",
+            output.common_prefixes().len()
+        );
+
+        info!("MaxKeys above limit with delimiter test passed");
+
+        env.stop_server();
+    }
 }

crates/e2e_test/src/protocols/sftp_compliance.rs

@@ -39,7 +39,7 @@
 use crate::protocols::sftp_compliance_tests::{
     cmptst_01, cmptst_02, cmptst_03, cmptst_04, cmptst_05, cmptst_06, cmptst_07, cmptst_08, cmptst_09, cmptst_10, cmptst_11,
     cmptst_12, cmptst_13, cmptst_14, cmptst_15, cmptst_16, cmptst_17, cmptst_18, cmptst_19, cmptst_20, cmptst_21, cmptst_22,
-    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, spawn_compliance_rustfs,
+    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, cmptst_34, spawn_compliance_rustfs,
 };
 #[cfg(target_os = "linux")]
 use crate::protocols::sftp_compliance_tests::{cmptst_24, cmptst_25, cmptst_26};
@@ -96,6 +96,15 @@ pub async fn test_sftp_compliance_suite() -> Result<()> {
         cmptst_13::run_implicit_dir_round_trip(&sftp).await?;
         cmptst_14::run_winscp_setstat_shape_on_handle(&sftp).await?;
 
+        // CMPTST-34 cross-checks the SFTP streaming-multipart write
+        // path against the S3 layer. The OPEN-time FileAttributes must
+        // reach the finalised object as x-amz-meta-* user metadata
+        // through the CreateMultipartUpload input field. The S3 client
+        // connects to the same rustfs process this suite already drives.
+        let s3 = build_test_s3_client(&format!("http://{COMPLIANCE_RW_S3_ADDRESS}"));
+        wait_for_s3_ready(&s3, 30).await?;
+        cmptst_34::run_open_attrs_round_trip_multipart(&sftp, &s3).await?;
+
         drop(sftp);
         session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
         info!("SFTP compliance suite passed");

crates/e2e_test/src/protocols/sftp_compliance_tests.rs

@@ -104,6 +104,11 @@
 //!   byte-exact with the production cache window.
 //! - CMPTST-33: read-cache disabled regression, 8 MiB download
 //!   byte-exact with RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES=0.
+//! - CMPTST-34: OPEN with non-default FileAttributes followed by a
+//!   payload that crosses the 5 MiB multipart boundary preserves the
+//!   client-supplied mtime and permissions through the streaming
+//!   CreateMultipartUpload path. HeadObject through aws-sdk-s3
+//!   confirms the metadata reached the finalised S3 object.
 
 use crate::common::rustfs_binary_path_with_features;
 use crate::protocols::sftp_helpers::{
@@ -3276,6 +3281,80 @@ pub(crate) mod cmptst_33 {
     }
 }
 
+// CMPTST-34: OPEN-time client attrs preservation across the streaming
+// multipart write path. The payload crosses the 5 MiB part-size
+// boundary so the driver transitions Buffering -> Streaming and
+// finalises via CompleteMultipartUpload. The OPEN-supplied mtime and
+// permissions must reach the resulting object as x-amz-meta-mtime and
+// x-amz-meta-mode. The S3 client connects to the same rustfs process
+// the shared-server suite already drives.
+pub(crate) mod cmptst_34 {
+    use super::*;
+
+    const COMPLIANCE_TEST_OUTPUT_ID: &str = "CMPTST-34";
+    const REQUESTED_MTIME: u32 = 1_715_000_010;
+    const REQUESTED_MODE: u32 = 0o600;
+
+    pub(crate) async fn run_open_attrs_round_trip_multipart(sftp: &SftpSession, s3: &S3Client) -> Result<()> {
+        info!("{COMPLIANCE_TEST_OUTPUT_ID}: OPEN with mtime + mode, multi-part payload, streaming path");
+        let bucket = "complopenattrsmpbucket";
+        let bucket_path = format!("/{bucket}");
+        sftp.create_dir(&bucket_path).await?;
+
+        let path = format!("/{bucket}/attr-mp.bin");
+        // 6 MiB exceeds the 5 MiB part-size boundary so the streaming
+        // path runs at least one full UploadPart before the CLOSE-time
+        // CompleteMultipartUpload finalises the object.
+        let payload = vec![0xA5u8; 6 * 1024 * 1024];
+
+        let client_attrs = FileAttributes {
+            mtime: Some(REQUESTED_MTIME),
+            atime: Some(REQUESTED_MTIME),
+            permissions: Some(REQUESTED_MODE),
+            ..FileAttributes::default()
+        };
+
+        let mut writer = sftp
+            .open_with_flags_and_attributes(&path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE, client_attrs)
+            .await?;
+        writer.write_all(&payload).await?;
+        writer.flush().await?;
+        writer.shutdown().await?;
+
+        let head = s3
+            .head_object()
+            .bucket(bucket)
+            .key("attr-mp.bin")
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 HeadObject failed: {e:?}"))?;
+        let content_length = head.content_length().unwrap_or(0);
+        if content_length != payload.len() as i64 {
+            return Err(anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} unexpected size: got {content_length} bytes"));
+        }
+        let metadata = head
+            .metadata()
+            .ok_or_else(|| anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} HeadObject returned no metadata map"))?;
+        let mtime_value = metadata
+            .get("mtime")
+            .ok_or_else(|| anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} mtime key missing on the object"))?;
+        if mtime_value != &REQUESTED_MTIME.to_string() {
+            return Err(anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} mtime mismatch: got {mtime_value}"));
+        }
+        let mode_value = metadata
+            .get("mode")
+            .ok_or_else(|| anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} mode key missing on the object"))?;
+        if mode_value != &REQUESTED_MODE.to_string() {
+            return Err(anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} mode mismatch: got {mode_value}"));
+        }
+
+        sftp.remove_file(&path).await?;
+        sftp.remove_dir(&bucket_path).await?;
+        info!("PASS {COMPLIANCE_TEST_OUTPUT_ID}: multipart upload preserved mtime + mode end to end");
+        Ok(())
+    }
+}
+
 // Shared parameters for CMPTST-32 (cache enabled) and CMPTST-33 (cache
 // disabled). Both cases seed the same fixture and download it
 // end-to-end, then assert byte-count and SHA256 against the

crates/e2e_test/src/replication_extension_test.rs

@@ -651,6 +651,22 @@ async fn list_service_accounts(
     Ok(response.json().await?)
 }
 
+async fn get_account_info(
+    env: &RustFSTestEnvironment,
+    signer_access_key: &str,
+    signer_secret_key: &str,
+) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
+    let url = format!("{}/rustfs/admin/v3/accountinfo", env.url);
+    let response = signed_request(http::Method::GET, &url, signer_access_key, signer_secret_key, None, None).await?;
+    if response.status() != StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        return Err(format!("account info failed: {status} {body}").into());
+    }
+
+    Ok(response.json().await?)
+}
+
 async fn wait_for_service_accounts(
     env: &RustFSTestEnvironment,
     signer_access_key: &str,
@@ -2379,6 +2395,57 @@ async fn test_site_replication_replicates_group_policy_backed_access_real_dual_n
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn test_service_account_policy_from_accountinfo_round_trips_real_single_node() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let account_info = get_account_info(&env, &env.access_key, &env.secret_key).await?;
+    let policy_str = account_info
+        .get("policy")
+        .and_then(|value| value.as_str())
+        .ok_or("account info policy should be a JSON string")?;
+
+    let policy: serde_json::Value = serde_json::from_str(policy_str)?;
+    let statements = policy
+        .get("Statement")
+        .and_then(|value| value.as_array())
+        .ok_or("account info policy should include Statement array")?;
+
+    assert!(!statements.is_empty(), "account info policy Statement should not be empty: {policy}");
+
+    let req = AddServiceAccountReq {
+        policy: Some(policy),
+        target_user: None,
+        access_key: "svcacct-info-sample".to_string(),
+        secret_key: "svcacct-info-sample-secret-key-123456".to_string(),
+        name: Some("svcacct-info-sample".to_string()),
+        description: Some("service account created from accountinfo sample policy".to_string()),
+        expiration: None,
+        comment: None,
+    };
+
+    let created = add_service_account(&env, &env.access_key, &env.secret_key, &req).await?;
+    assert_eq!(created.0, "svcacct-info-sample");
+
+    let listed =
+        wait_for_service_accounts(&env, &env.access_key, &env.secret_key, Some(&env.access_key), &["svcacct-info-sample"])
+            .await?;
+    assert!(
+        listed
+            .accounts
+            .iter()
+            .any(|account| account.access_key == "svcacct-info-sample"),
+        "created service account should be listed for parent user: {:?}",
+        listed.accounts
+    );
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_site_replication_replicates_multiple_service_accounts_real_dual_node() -> Result<(), Box<dyn Error + Send + Sync>> {

crates/ecstore/Cargo.toml

@@ -38,6 +38,7 @@ rustfs-filemeta.workspace = true
 rustfs-utils = { workspace = true, features = ["full"] }
 rustfs-rio.workspace = true
 rustfs-signer.workspace = true
+rustfs-tls-runtime.workspace = true
 rustfs-checksums.workspace = true
 rustfs-config = { workspace = true, features = ["constants", "notify", "audit"] }
 rustfs-credentials = { workspace = true }
@@ -45,7 +46,9 @@ rustfs-common.workspace = true
 rustfs-policy.workspace = true
 rustfs-protos.workspace = true
 rustfs-kms.workspace = true
-rustfs-s3-common = { workspace = true }
+rustfs-s3-types = { workspace = true }
+rustfs-data-usage.workspace = true
+rustfs-object-capacity.workspace = true
 async-trait.workspace = true
 bytes.workspace = true
 byteorder = { workspace = true }
@@ -89,6 +92,7 @@ hyper.workspace = true
 hyper-util.workspace = true
 hyper-rustls.workspace = true
 rustls.workspace = true
+rustls-pki-types.workspace = true
 tokio = { workspace = true, features = ["io-util", "sync", "signal","io-uring"] }
 tonic.workspace = true
 xxhash-rust = { workspace = true, features = ["xxh64", "xxh3"] }
@@ -103,7 +107,7 @@ memmap2 = { workspace = true }
 libc.workspace = true
 rustix = { workspace = true }
 rustfs-madmin.workspace = true
-rustfs-workers.workspace = true
+rustfs-concurrency.workspace = true
 reqwest = { workspace = true }
 aes-gcm.workspace = true
 aws-sdk-s3 = { workspace = true }
@@ -145,5 +149,9 @@ harness = false
 name = "comparison_benchmark"
 harness = false
 
+[[bench]]
+name = "rename_data_meta_benchmark"
+harness = false
+
 [lib]
 doctest = false

crates/ecstore/benches/rename_data_meta_benchmark.rs

@@ -0,0 +1,118 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main};
+use rustfs_filemeta::{ErasureAlgo, FileInfo, FileMeta};
+use std::hint::black_box;
+use std::time::Duration;
+use time::OffsetDateTime;
+use uuid::Uuid;
+
+const VERSION_COUNT_CASES: &[usize] = &[1, 8, 32, 64];
+const BENCH_BASE_TIME_UNIX_SECS: i64 = 1_700_000_000;
+
+fn make_file_info(version_id: Uuid, data_dir: Uuid, size: i64, mod_time: OffsetDateTime) -> FileInfo {
+    FileInfo {
+        version_id: Some(version_id),
+        data_dir: Some(data_dir),
+        size,
+        mod_time: Some(mod_time),
+        metadata: [("etag".to_string(), format!("etag-{version_id}"))].into_iter().collect(),
+        erasure: rustfs_filemeta::ErasureInfo {
+            algorithm: ErasureAlgo::ReedSolomon.to_string(),
+            data_blocks: 4,
+            parity_blocks: 2,
+            block_size: 1024 * 1024,
+            index: 1,
+            distribution: vec![1, 2, 3, 4, 5, 6],
+            ..Default::default()
+        },
+        ..Default::default()
+    }
+}
+
+fn build_meta_with_versions(version_count: usize) -> FileMeta {
+    let mut meta = FileMeta::new();
+    let base_time = OffsetDateTime::from_unix_timestamp(BENCH_BASE_TIME_UNIX_SECS).expect("valid bench base timestamp");
+    for i in 0..version_count {
+        let fi = make_file_info(Uuid::new_v4(), Uuid::new_v4(), 64 * 1024, base_time - Duration::from_secs(i as u64));
+        meta.add_version(fi).expect("seed add_version should succeed");
+    }
+    meta
+}
+
+fn bench_rename_data_meta_path(c: &mut Criterion) {
+    let mut group = c.benchmark_group("rename_data_meta");
+    group.sample_size(20);
+    group.measurement_time(Duration::from_secs(10));
+
+    for &version_count in VERSION_COUNT_CASES {
+        let seeded = build_meta_with_versions(version_count);
+        let dst_buf = seeded.marshal_msg().expect("marshal seeded meta");
+        let base_time = OffsetDateTime::from_unix_timestamp(BENCH_BASE_TIME_UNIX_SECS).expect("valid bench base timestamp");
+        let replace_version_id = seeded
+            .versions
+            .first()
+            .and_then(|v| v.header.version_id)
+            .unwrap_or(Uuid::nil());
+
+        group.bench_with_input(BenchmarkId::new("read_modify_write", version_count), &version_count, |b, _| {
+            b.iter(|| {
+                let mut xlmeta = FileMeta::load(black_box(&dst_buf)).expect("load dst meta");
+                let search_version_id = Some(replace_version_id);
+                let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(search_version_id);
+                if let Some(old_data_dir) = has_old_data_dir {
+                    let _ = xlmeta.data.remove_two(replace_version_id, old_data_dir);
+                }
+                let fi = make_file_info(replace_version_id, Uuid::new_v4(), 64 * 1024, base_time + Duration::from_millis(1));
+                xlmeta.add_version(fi).expect("add new version");
+                let out = xlmeta.marshal_msg().expect("marshal updated meta");
+                black_box(out);
+            });
+        });
+
+        let mut prepared = FileMeta::load(&dst_buf).expect("load prepared meta");
+        if let Some(old_data_dir) = prepared.find_unshared_data_dir_for_version(Some(replace_version_id)) {
+            let _ = prepared.data.remove_two(replace_version_id, old_data_dir);
+        }
+        group.bench_with_input(BenchmarkId::new("add_version_marshal_only", version_count), &version_count, |b, _| {
+            b.iter_batched(
+                || prepared.clone(),
+                |mut xlmeta| {
+                    let fi = make_file_info(replace_version_id, Uuid::new_v4(), 64 * 1024, base_time + Duration::from_millis(1));
+                    xlmeta.add_version(fi).expect("add new version");
+                    let out = xlmeta.marshal_msg().expect("marshal updated meta");
+                    black_box(out);
+                },
+                BatchSize::SmallInput,
+            );
+        });
+
+        group.bench_with_input(BenchmarkId::new("remove_two_only", version_count), &version_count, |b, _| {
+            b.iter(|| {
+                let mut xlmeta = FileMeta::load(black_box(&dst_buf)).expect("load dst meta");
+                let removed = if let Some(old_data_dir) = xlmeta.find_unshared_data_dir_for_version(Some(replace_version_id)) {
+                    xlmeta.data.remove_two(replace_version_id, old_data_dir).expect("remove two")
+                } else {
+                    false
+                };
+                black_box(removed);
+                black_box(xlmeta);
+            });
+        });
+    }
+}
+
+criterion_group!(benches, bench_rename_data_meta_path);
+criterion_main!(benches);

crates/ecstore/docs/internode-transport/internode-transport-adapter-rfc.md

@@ -0,0 +1,484 @@
+# RFC: Internode Transport Adapter Boundary
+
+> Status: draft
+> Last updated: 2026-05-22
+> Scope: OSS internode data-plane adapter analysis, benchmark baseline, and
+> transport boundary
+
+## Summary
+
+The current distributed internode paths use TCP-based HTTP/gRPC transports:
+
+- `tonic` gRPC `NodeService` for most control, metadata, lock, health, and
+  peer operations.
+- HTTP streaming routes under `/rustfs/rpc/` for remote disk file streams.
+
+This document frames the existing work as an OSS `InternodeDataTransport`
+adapter boundary. The adapter keeps RustFS data-plane logic separate from the
+concrete transport backend while preserving the current TCP/HTTP behavior as
+the default implementation.
+
+Current implementation status:
+
+- `InternodeDataTransport` exists in
+  `crates/ecstore/src/rpc/internode_data_transport.rs`.
+- The default and only production backend is `tcp-http`; `tcp` is accepted as
+  an alias.
+- `RUSTFS_INTERNODE_DATA_TRANSPORT` selects the backend. Blank or unset values
+  use `tcp-http`; invalid values fail closed.
+- `RemoteDisk::read_file_stream`, `RemoteDisk::create_file`,
+  `RemoteDisk::append_file`, and `RemoteDisk::walk_dir` delegate to the
+  transport.
+- `NodeService` gRPC remains the internode control plane and continues to carry
+  metadata/control operations.
+
+Related design notes in this directory:
+
+- `transport-capabilities.md`
+- `transport-buffer-lifecycle.md`
+- `transport-buffer-contract.md`
+- `transport-fallback-and-selection.md`
+- `transport-metrics-and-baseline.md`
+
+## Open-source Scope
+
+The OSS scope is:
+
+- define a clear `InternodeDataTransport` adapter boundary;
+- keep `tcp-http` as the default backend;
+- keep existing TCP/HTTP behavior unchanged;
+- keep internode data-plane behavior observable through metrics and baseline
+  tooling;
+- document buffer ownership, fallback, and capability expectations for
+  maintainable transport code;
+- avoid adding dependencies or backend implementations.
+
+The OSS scope is not:
+
+- adding another transport backend;
+- replacing the gRPC control plane;
+- adding benchmark plans for another transport;
+- adding runtime plugin loading;
+- changing object correctness semantics.
+
+## Goals
+
+- Document the current internode control plane and data plane.
+- Identify the existing transfer paths covered by the
+  `InternodeDataTransport` adapter and the paths that remain on gRPC.
+- Define the minimum benchmark baseline required before transport changes.
+- Sketch a pluggable transport boundary that preserves the current TCP/HTTP
+  behavior as the default backend.
+- Document backend-neutral capability, fallback, buffer ownership, and
+  observability expectations.
+
+## Non-Goals
+
+- Implement another transport backend.
+- Replace `tonic` gRPC for control-plane RPCs.
+- Redesign erasure coding, quorum handling, disk health tracking, or object
+  correctness semantics.
+- Change default development, CI, or ordinary RustFS deployment requirements.
+
+## Current Internode Architecture
+
+### Server-side entry points
+
+The main HTTP server builds a hybrid service per connection:
+
+- `rustfs/src/server/http.rs` wires a `NodeServiceServer` for gRPC.
+- `rustfs/src/storage/rpc/InternodeRpcService` intercepts HTTP paths under
+  `/rustfs/rpc/`.
+- Other HTTP/S3 traffic continues through the normal S3 service.
+
+Compression logic already treats `/rustfs/rpc/` and `/rustfs/peer/` as internode
+RPC paths and skips normal response compression for them.
+
+### gRPC channel management
+
+`crates/protos/src/lib.rs` creates internode gRPC channels with `tonic`
+`Endpoint`:
+
+- connect timeout
+- TCP keepalive
+- HTTP/2 keepalive interval and timeout
+- request timeout
+- optional TLS configuration
+- global channel caching and failed-connection eviction
+
+This confirms the current gRPC transport is TCP/HTTP2-based.
+
+### NodeService layout
+
+`crates/protos/src/node.proto` defines one `NodeService` that mixes several
+classes of RPCs:
+
+- meta service: bucket and metadata operations
+- disk service: local/remote disk operations
+- lock service: distributed lock operations
+- peer rest service: node health, metrics, IAM/policy reload, rebalance,
+  profiling, events, and admin-style operations
+
+The service layout is practical today, but it is too broad to become the
+transport adapter surface. A pluggable data transport should target only disk
+data streams and keep this gRPC service as the control plane.
+
+## Control Plane vs Data Plane
+
+### Control plane
+
+These paths carry coordination, metadata, health, and administrative state.
+They should remain on gRPC/TCP:
+
+| Area | Client/server code | Examples | Notes |
+| --- | --- | --- | --- |
+| Bucket peer ops | `crates/ecstore/src/rpc/peer_s3_client.rs`, `rustfs/src/storage/rpc/bucket.rs` | `MakeBucket`, `ListBucket`, `DeleteBucket`, `GetBucketInfo`, `HealBucket` | Small metadata/control payloads. |
+| Locking | `crates/ecstore/src/rpc/remote_locker.rs`, `rustfs/src/storage/rpc/lock.rs` | `Lock`, `UnLock`, `Refresh`, batch lock/unlock | Latency-sensitive but not bulk data; correctness and timeout semantics matter more than transport bandwidth. |
+| Peer/admin state | `crates/ecstore/src/rpc/peer_rest_client.rs`, `rustfs/src/storage/rpc/health.rs`, `metrics.rs`, `event.rs` | `LocalStorageInfo`, `ServerInfo`, `GetMetrics`, `GetLiveEvents`, reload APIs, rebalance APIs | Operational control plane. |
+| Disk metadata/control | `crates/ecstore/src/rpc/remote_disk.rs`, `rustfs/src/storage/rpc/disk.rs` | `DiskInfo`, `ReadXL`, `ReadVersion`, `ReadMetadata`, `WriteMetadata`, `RenameFile`, `RenamePart`, `Delete*`, `VerifyFile`, `CheckParts` | Usually metadata, integrity checks, or namespace mutations. |
+| Connection health | `RemoteDisk`, `RemotePeerS3Client`, `PeerRestClient` | TCP connectivity probes and fault/recovery state | Must remain available even if an optional data backend is unavailable. |
+
+### Data plane candidates
+
+These paths move object shard bytes or stream potentially large disk data and
+are the only reasonable first candidates for a pluggable transport.
+
+| Priority | Path | Current client | Current server | Current transport | Why it matters |
+| --- | --- | --- | --- | --- | --- |
+| P0 | `read_file_stream` | `RemoteDisk::read_file_stream` | `handle_read_file` in `http_service.rs` | HTTP `GET /rustfs/rpc/read_file_stream` with a streaming response body | Main remote disk read stream used by bitrot readers and erasure reads. |
+| P0 | `put_file_stream` | `RemoteDisk::create_file` and `RemoteDisk::append_file` | `handle_put_file` in `http_service.rs` | HTTP `PUT /rustfs/rpc/put_file_stream` with a streaming request body | Main remote disk write stream used by bitrot writers and erasure writes. |
+| P1 | `walk_dir` | `RemoteDisk::walk_dir` | `handle_walk_dir` in `http_service.rs` | HTTP `GET /rustfs/rpc/walk_dir` with a streamed metadata listing | Can be high-volume during scans/healing, but it is metadata-oriented rather than object byte data. |
+| P1 | `ReadAll` / `WriteAll` | `RemoteDisk::read_all` / `write_all` | gRPC unary disk handlers | gRPC unary `bytes` payload | Moves bytes today, but should be measured before treating it as a high-throughput data path. |
+| P2 | proto `WriteStream` / `ReadAt` | currently not used | currently returns unimplemented | gRPC streaming definitions exist but are not implemented | Declared proto shape, not a current production path. |
+
+## P1 Data Path Inventory
+
+Classification:
+
+- Covered by `InternodeDataTransport`: `RemoteDisk` opens the transfer through
+  the transport abstraction.
+- Still direct TCP/HTTP/gRPC: bytes move over a fixed internode protocol
+  outside the transport abstraction.
+- Metadata/control-plane only: payloads are expected to be small metadata,
+  namespace, lock, health, or admin messages.
+- Not relevant: declared or test-only paths that are not current production
+  data paths.
+
+### Covered by `InternodeDataTransport`
+
+| Path | Owner references | Server references | Classification | Notes |
+| --- | --- | --- | --- | --- |
+| Remote shard read stream | `crates/ecstore/src/rpc/remote_disk.rs::RemoteDisk::read_file_stream`; `crates/ecstore/src/rpc/internode_data_transport.rs::InternodeDataTransport::open_read`; `crates/ecstore/src/bitrot.rs::create_bitrot_reader` | `rustfs/src/storage/rpc/http_service.rs::handle_read_file` | Covered by `InternodeDataTransport` | Object GET, repair reads, and erasure decode use this path for remote shard bytes. |
+| Remote shard write stream | `RemoteDisk::create_file`; `RemoteDisk::append_file`; `InternodeDataTransport::open_write`; `crates/ecstore/src/bitrot.rs::create_bitrot_writer` | `rustfs/src/storage/rpc/http_service.rs::handle_put_file` | Covered by `InternodeDataTransport` | Object PUT and multipart part upload use this path for remote shard bytes. |
+| Remote namespace walk stream | `RemoteDisk::walk_dir`; `InternodeDataTransport::open_walk_dir`; `crates/ecstore/src/cache_value/metacache_set.rs` walk producers | `rustfs/src/storage/rpc/http_service.rs::handle_walk_dir` | Covered by `InternodeDataTransport` | High-volume listing/scanner/heal metadata stream. It is not object byte data, but it is a large internode stream. |
+| Remote zero-copy read fallback | `RemoteDisk::read_file_zero_copy` | same as remote shard read stream | Covered by `InternodeDataTransport` through `read_file_stream` | The remote path buffers the stream into `Bytes`; true zero-copy is not guaranteed for remote disks. |
+
+### Still Direct TCP/HTTP/gRPC
+
+| Path | Owner references | Server references | Classification | Notes |
+| --- | --- | --- | --- | --- |
+| `ReadAll` | `RemoteDisk::read_all`; `crates/ecstore/src/store_init.rs`; heal resume metadata readers | `rustfs/src/storage/rpc/disk.rs::handle_read_all` | Still direct gRPC | Unary `bytes` response. Currently used mostly for metadata/config files; measure before moving. |
+| `WriteAll` | `RemoteDisk::write_all`; `crates/ecstore/src/store_init.rs`; heal resume metadata writers | `rustfs/src/storage/rpc/disk.rs::handle_write_all` | Still direct gRPC | Unary `bytes` request. Currently used mostly for metadata/config/checkpoint writes. |
+| `ReadMultiple` | `RemoteDisk::read_multiple`; `crates/ecstore/src/set_disk/read.rs::read_multiple_files` | `rustfs/src/storage/rpc/disk.rs::handle_read_multiple` | Still direct gRPC | Returns multiple small file payloads, usually metadata/listing support. Could become large with many entries. |
+| `ReadParts` | `RemoteDisk::read_parts`; `crates/ecstore/src/set_disk/read.rs::read_parts`; multipart list/complete paths | `rustfs/src/storage/rpc/disk.rs::handle_read_parts` | Still direct gRPC | Encoded `ObjectPartInfo` metadata, not object data. |
+| `RenamePart` | `RemoteDisk::rename_part`; `crates/ecstore/src/set_disk/write.rs::rename_part` | `rustfs/src/storage/rpc/disk.rs::handle_rename_part` | Still direct gRPC | Carries part metadata while committing multipart data already written through stream writers. |
+| `ListDir` | `RemoteDisk::list_dir`; multipart/lifecycle metadata listing callers | `rustfs/src/storage/rpc/disk.rs::handle_list_dir` | Still direct gRPC | Directory name listing, metadata/control-plane unless measured otherwise. |
+| Legacy gRPC `WalkDir` | `rustfs/src/storage/rpc/node_service.rs::NodeService::walk_dir` | same file | Still direct gRPC | Server implementation remains, but current `RemoteDisk::walk_dir` uses HTTP through the transport. Keep until callers are audited or compatibility policy is set. |
+
+### Metadata/control-plane only
+
+| Area | Owner references | Classification | Notes |
+| --- | --- | --- | --- |
+| Disk metadata and namespace mutations | `RemoteDisk::{read_metadata,write_metadata,update_metadata,read_version,read_xl,rename_data,rename_file,delete*,verify_file,check_parts,disk_info}` | Metadata/control-plane only | These remain on gRPC by design. |
+| Peer/bucket/admin operations | `crates/ecstore/src/rpc/{peer_s3_client.rs,peer_rest_client.rs,remote_locker.rs}` and matching `rustfs/src/storage/rpc/*` handlers | Metadata/control-plane only | Not candidates for a data-plane backend without separate measurements. |
+| Store init and format operations | `crates/ecstore/src/store_init.rs` | Metadata/control-plane only | Uses `ReadAll`/`WriteAll` for small format/config objects. |
+| Heal orchestration | `crates/heal/src/heal/storage.rs` and `crates/ecstore/src/set_disk.rs::heal_object` | Metadata/control-plane plus covered data reads | Heal object data reads go through `get_object_reader` and then covered shard streams; resume/checkpoint metadata uses direct gRPC disk metadata calls. |
+
+### Not Relevant Current Paths
+
+| Path | Owner references | Classification | Notes |
+| --- | --- | --- | --- |
+| Proto `Write` | `crates/protos/src/node.proto`; `rustfs/src/storage/rpc/disk.rs::handle_write` | Not relevant | Handler is unimplemented. |
+| Proto `WriteStream` | `crates/protos/src/node.proto`; `rustfs/src/storage/rpc/node_service.rs::write_stream` | Not relevant | Returns `unimplemented`. |
+| Proto `ReadAt` | `crates/protos/src/node.proto`; `rustfs/src/storage/rpc/node_service.rs::read_at` | Not relevant | Returns `unimplemented`. |
+| E2E reliant gRPC helpers | `crates/e2e_test/src/reliant/*` | Not relevant | Test harnesses, not production internode data-path callers. |
+
+### Current Limitations
+
+| Risk | Limitation |
+| --- | --- |
+| Medium | `ReadAll` and `WriteAll` still carry unary `bytes` over gRPC. They appear metadata-oriented today, but there is no size threshold or routing policy. |
+| Medium | `ReadMultiple` can aggregate many metadata files into one gRPC response. |
+| Low | Legacy gRPC `WalkDir` remains implemented while `RemoteDisk::walk_dir` uses HTTP through the transport. |
+| Medium | Remote `read_file_zero_copy` is a buffered read over the transport, not a remote zero-copy contract. |
+| Medium | Server-side TCP HTTP route handling is outside the client-side trait. |
+
+## Current Object Write Path
+
+For object PUTs in distributed erasure mode, the relevant flow is:
+
+1. Upper storage layers prepare object data and erasure metadata.
+2. `SetDisks` selects local and remote disks.
+3. `create_bitrot_writer` calls `disk.create_file(...)` for each shard writer.
+4. For a remote disk, `RemoteDisk::create_file` delegates to
+   `InternodeDataTransport::open_write`.
+5. `HttpWriter` sends an HTTP `PUT` to `/rustfs/rpc/put_file_stream`.
+6. The remote node's `handle_put_file` opens the local file writer and copies
+   incoming body chunks into it.
+7. `Erasure::encode` writes shards through `MultiWriter` to all selected
+   writers while enforcing write quorum.
+
+This is the primary write data-plane candidate.
+
+## Current Object Read Path
+
+For object GETs and repair reads in distributed erasure mode, the relevant flow is:
+
+1. `SetDisks` prepares shard readers for the selected disks.
+2. `create_bitrot_reader` uses local zero-copy only when `disk.is_local()`.
+3. For a remote disk, it calls `disk.read_file_stream(...)`.
+4. `RemoteDisk::read_file_stream` delegates to
+   `InternodeDataTransport::open_read`.
+5. `HttpReader` sends an HTTP `GET` to `/rustfs/rpc/read_file_stream`.
+6. The remote node's `handle_read_file` opens the local disk stream and returns
+   it as an HTTP streaming body.
+7. The erasure decoder reads from the shard streams and reconstructs the object.
+
+This is the primary read data-plane candidate.
+
+## Existing Metrics and Benchmark Surface
+
+RustFS already has coarse internode metrics in `crates/io-metrics/src/internode_metrics.rs`:
+
+- sent bytes
+- received bytes
+- outgoing requests
+- incoming requests
+- errors
+- dial errors
+- average dial time
+
+These metrics are useful as a starting point. For backend comparisons, the
+relevant route-level and operation-level dimensions are:
+
+- `read_file_stream`
+- `put_file_stream`
+- `walk_dir`
+- gRPC `ReadAll` / `WriteAll`
+- gRPC control-plane request volume
+
+Existing benchmark assets:
+
+- `scripts/run_object_batch_bench.sh`
+- `scripts/run_object_batch_bench_enhanced.sh`
+- `scripts/run_object_batch_bench_abc.sh`
+- `scripts/run_four_node_cluster_failover_bench.sh`
+- `scripts/run_internode_transport_baseline.sh` (scenario matrix wrapper for local vs distributed TCP baseline artifacts)
+- Criterion benches under `crates/ecstore/benches/`
+
+These mostly cover S3/object workload or erasure coding performance. They do
+not yet isolate internode transport cost.
+
+## Required TCP Baseline
+
+Before changing internode data transport behavior or comparing a non-default
+backend, collect a baseline for the current TCP/HTTP/gRPC implementation.
+
+### Topology
+
+Minimum:
+
+- 1-node local erasure deployment, to measure local disk and erasure overhead.
+- 4-node distributed erasure deployment, to measure internode overhead.
+
+Preferred:
+
+- Same host count and disk layout for every run.
+- Dedicated network interface or isolated VLAN.
+- Fixed CPU governor and no unrelated background load.
+- Recorded kernel version, NIC model, MTU, RustFS commit, Rust toolchain, and
+  benchmark tool versions.
+
+### Workloads
+
+| Workload | Sizes | Concurrency | Main signal |
+| --- | --- | --- | --- |
+| S3 PUT | 4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB | 1, 16, 64, 128 | End-to-end write throughput and tail latency. |
+| S3 GET | 4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB | 1, 16, 64, 128 | End-to-end read throughput and tail latency. |
+| Remote disk stream read | shard-sized ranges from `read_file_stream` | 1, 16, 64 | Isolated internode read path. |
+| Remote disk stream write | shard-sized writes through `put_file_stream` | 1, 16, 64 | Isolated internode write path. |
+| Healing / repair | missing disk or missing shard scenario | controlled | Rebuild throughput and read/write amplification. |
+| Scanner walk | large bucket/object namespace | controlled | Metadata streaming pressure, not the primary object-byte transport path. |
+
+### Measurements
+
+Collect:
+
+- throughput in bytes/s and objects/s
+- p50, p95, p99, and max latency
+- CPU utilization per process and per core
+- memory RSS and allocation pressure where available
+- `rustfs_system_network_internode_*` metrics
+- TCP retransmits, socket errors, and NIC throughput
+- disk throughput and utilization
+- failure/retry/fallback counts
+
+The baseline should produce a machine-readable artifact, for example
+`target/bench/internode-transport/<timestamp>/summary.csv`, plus the exact
+commands and configuration used.
+
+### Baseline runner entry point
+
+Use `scripts/run_internode_transport_baseline.sh` to execute a reproducible
+S3 PUT/GET matrix against `local` and `distributed` scenarios and export:
+
+- `summary.csv` (throughput/latency summary per workload and object size)
+- `internode_metric_deltas.csv` (operation-level internode metric deltas when
+  `--metrics-url` is provided)
+
+See `transport-metrics-and-baseline.md` for current metric names, labels,
+operation values, baseline inputs, and baseline artifact fields.
+
+## Transport Abstraction Proposal
+
+### Design principle
+
+Keep `NodeService` as the control plane. Introduce a separate data transport
+only below `RemoteDisk`, where remote disk byte streams are opened today.
+
+The first implementation should be a no-behavior-change TCP/HTTP backend that
+wraps the current `HttpReader`, `HttpWriter`, and `/rustfs/rpc/*` handlers.
+Non-default backend work should not proceed until the default wrapper is
+measured and adapter gaps are documented.
+
+### Candidate boundary
+
+The current boundary is remote disk stream transfer:
+
+```rust
+#[async_trait::async_trait]
+pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
+    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
+    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
+    async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
+    fn name(&self) -> &'static str;
+    fn capabilities(&self) -> InternodeDataTransportCapabilities;
+}
+```
+
+Initial request fields should mirror the current HTTP query parameters:
+
+- peer endpoint
+- disk reference
+- volume
+- path
+- offset
+- length
+- append/create mode
+- expected size
+- optional stall timeout for long-running listing streams
+
+The initial TCP backend can keep the current signed HTTP URLs internally.
+
+### Integration point
+
+`RemoteDisk` delegates only these methods to the data transport:
+
+- `read_file_stream`
+- `read_file_zero_copy` as the current wrapper over `read_file_stream`
+- `append_file`
+- `create_file`
+- `walk_dir`
+
+All other `RemoteDisk` methods continue using the current gRPC client
+in this adapter scope.
+
+### Capability model
+
+Avoid hard-coding transport-specific assumptions into the generic interface.
+The current conservative capability fields are:
+
+- streaming read
+- streaming write
+- streaming walk-dir
+- ordered delivery
+- maximum transfer size
+- fallback support
+
+The TCP/HTTP backend should report only capabilities that it actually provides.
+
+## TCP Fallback Requirements
+
+TCP/HTTP/gRPC must remain the default and required backend.
+
+Fallback rules:
+
+- If no explicit data transport is configured, use the current TCP/HTTP
+  implementation.
+- The current accepted values for `RUSTFS_INTERNODE_DATA_TRANSPORT` are
+  `tcp-http` and the `tcp` alias. Empty and unset values use `tcp-http`.
+- Invalid configured values fail closed with an error that includes the env var
+  name and invalid value.
+- Unsupported configured backends fail closed during transport construction.
+- Runtime fallback must preserve object correctness and quorum semantics.
+- Fallback events must be logged and counted in metrics.
+
+Do not add fallback settings until there is an implementation PR that uses them.
+
+## Baseline Validation Commands
+
+Dry-run command:
+
+```bash
+scripts/run_internode_transport_baseline.sh \
+  --access-key minioadmin \
+  --secret-key minioadmin \
+  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
+  --sizes 4KiB,1MiB \
+  --concurrencies 1 \
+  --duration 10s \
+  --dry-run
+```
+
+Real TCP baseline command with metrics:
+
+```bash
+RUSTFS_INTERNODE_DATA_TRANSPORT=tcp-http \
+scripts/run_internode_transport_baseline.sh \
+  --access-key "$RUSTFS_ACCESS_KEY" \
+  --secret-key "$RUSTFS_SECRET_KEY" \
+  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
+  --metrics-url http://127.0.0.1:9000/metrics \
+  --out-dir target/bench/internode-transport/manual-run
+```
+
+Expected artifacts:
+
+- `run_manifest.txt`
+- `summary.csv`
+- `internode_metric_deltas.csv` when `--metrics-url` is provided
+
+The baseline validates the default TCP/HTTP path only. It must not be used to
+claim support or performance for any other transport.
+
+## Adapter Constraints
+
+The current adapter boundary has these constraints:
+
+- `tcp-http` is the default and only OSS backend.
+- Backend selection is explicit and fail-closed.
+- The gRPC control plane remains responsible for metadata, health, locks, and
+  coordination.
+- Transport errors must preserve existing disk health, quorum, timeout, and
+  integrity semantics.
+- Metrics must identify the selected backend and operation without high-cardinality
+  labels.
+
+The current `RemoteDisk::walk_dir` stream is routed through the adapter.
+Metadata RPCs, locks, admin RPCs, bucket coordination, and the legacy gRPC
+`WalkDir` handler remain outside the current data-plane boundary.
+
+## Out of Scope
+
+This RFC does not add a plugin system, split the adapter into a separate crate,
+add accepted backend values, or implement a new transport backend.

crates/ecstore/docs/internode-transport/transport-buffer-contract.md

@@ -0,0 +1,91 @@
+# Internode Transport Buffer Contract
+
+Status: design note only. This document defines a backend-neutral buffer
+ownership and lifecycle contract for the `InternodeDataTransport` adapter. It
+does not implement a new backend and does not change production behavior.
+
+## Open-source Scope
+
+The open-source RustFS path keeps `tcp-http` as the default internode data
+transport. This document defines adapter contracts only:
+
+- no additional production backend is introduced;
+- no dependency is added;
+- no new accepted production backend value is added;
+- RustFS core data-plane logic remains independent of the concrete transport
+  implementation.
+
+## Current Adapter Surface
+
+The current data-plane surface is byte-stream based:
+
+| Current path | Current API shape | Current ownership |
+| --- | --- | --- |
+| Remote read stream | `InternodeDataTransport::open_read(...) -> FileReader` | Backend returns boxed `AsyncRead`; callers provide temporary `ReadBuf` storage per poll. |
+| Remote write stream | `InternodeDataTransport::open_write(...) -> FileWriter` | Callers pass borrowed `&[u8]` slices into boxed `AsyncWrite`; the backend owns any async body staging. |
+| Walk-dir stream | `InternodeDataTransport::open_walk_dir(...) -> FileReader` | Same boxed stream model as read, with a small serialized request body. |
+
+This API is correct for the current TCP/HTTP backend. The adapter contract
+describes current ownership boundaries without assuming implementation details
+outside `TcpHttpInternodeDataTransport`.
+
+## Buffer Ownership Model
+
+| Buffer role | Allocator | Lifetime owner | Transport state | TCP/HTTP behavior |
+| --- | --- | --- | --- | --- |
+| Send buffer | Caller or RustFS-owned pool | Caller until the writer copies or accepts bytes for the HTTP body | Existing HTTP body staging | Copy into the existing `AsyncWrite` path when the writer cannot use the borrowed slice directly. |
+| Receive buffer | Caller-provided storage | Reader while filling; caller after `poll_read` returns | Existing HTTP response body chunks | Copy from `AsyncRead` into caller storage as today. |
+| Control metadata | RustFS caller | Caller/request object | Not buffer-managed by the data-plane backend | Serialize into HTTP/gRPC/control-plane messages. |
+| Fallback staging | TCP/HTTP backend | TCP/HTTP backend | Existing `HttpReader`/`HttpWriter` buffers | Existing buffering semantics. |
+
+The current writer must not retain borrowed caller slices beyond the write call.
+When bytes must outlive the call, they are copied into owned HTTP body chunks.
+
+This contract does not claim zero-copy behavior. The current TCP/HTTP path
+documents where copies occur.
+
+## Compatibility Contract
+
+The current stream API remains the OSS compatibility contract:
+
+```rust
+#[async_trait::async_trait]
+pub trait InternodeDataTransport {
+    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
+    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
+    async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
+}
+```
+
+## Current Adapter Contract
+
+| Area | Required contract |
+| --- | --- |
+| Ownership | Define when caller-owned bytes are copied or accepted by the transport. |
+| Completion | Return from stream operations only when bytes are accepted or an error is reported. |
+| Staging | Keep staging behavior inside the TCP/HTTP implementation. |
+| Size limits | Report any RustFS-visible `max_transfer_size`; TCP/HTTP currently reports none. |
+| Ordering | Preserve ordered byte-stream semantics. |
+| Copy accounting | Document known copy boundaries and avoid unmeasured zero-copy claims. |
+
+## Current API Limitations
+
+| Current API | Current limitation |
+| --- | --- |
+| `FileReader = Box<dyn AsyncRead + Send + Sync + Unpin>` | `AsyncRead` exposes temporary caller `ReadBuf` storage. |
+| `FileWriter = Box<dyn AsyncWrite + Send + Sync + Unpin>` | `AsyncWrite::poll_write` receives borrowed `&[u8]` that cannot outlive the poll. |
+| `HttpWriter` | The async HTTP body must own `Bytes`, so borrowed write buffers are copied into `BytesMut` or `Bytes`. |
+| `write_body_chunks_to_writer` | Server-side HTTP body chunks are copied into `BytesMut` before local disk write. |
+| Erasure encode output | Encoded shards are represented as `Vec<Bytes>` and written through `AsyncWrite`. |
+| Erasure decode input | Shard reads allocate `Vec<u8>` buffers before decode. |
+
+These limitations do not block the current `tcp-http` backend.
+
+## Adapter Stability
+
+`InternodeDataTransport` should keep RustFS core data-plane logic separate from
+the concrete transport implementation. The trait and `tcp-http` backend remain
+inside `ecstore`.
+
+This PR does not perform a crate split, add runtime loading, introduce a plugin
+system, add a backend value, or implement a new transport backend.

crates/ecstore/docs/internode-transport/transport-buffer-lifecycle.md

@@ -0,0 +1,122 @@
+# Internode Buffer Lifecycle and Copy Count
+
+Status: P1-D analysis only. This document records the current TCP/HTTP
+internode data path and the ownership boundaries that matter for the
+backend-neutral `InternodeDataTransport` adapter. It does not implement a new
+backend or change production behavior.
+
+## Open-source Scope
+
+The OSS scope is:
+
+- define buffer ownership and copy-count behavior for the current
+  `InternodeDataTransport` adapter;
+- keep `tcp-http` as the default backend;
+- keep existing TCP/HTTP behavior unchanged;
+- document copy hotspots and ownership gaps for maintainable transport code;
+- avoid adding dependencies or backend implementations.
+
+The OSS scope is not:
+
+- adding another transport backend;
+- replacing the current TCP/HTTP path;
+- adding benchmark plans for another transport;
+- changing object correctness semantics.
+
+## Scope
+
+The covered paths are the large internode data-plane calls currently routed
+through `InternodeDataTransport`:
+
+| Path | Entry | Transport owner | Server owner |
+| --- | --- | --- | --- |
+| Read stream | `RemoteDisk::read_file_stream` in `crates/ecstore/src/rpc/remote_disk.rs` | `TcpHttpInternodeDataTransport::open_read` in `crates/ecstore/src/rpc/internode_data_transport.rs`, `HttpReader` in `crates/rio/src/http_reader.rs` | `handle_read_file` in `rustfs/src/storage/rpc/http_service.rs` |
+| Write stream | `RemoteDisk::create_file`, `RemoteDisk::append_file` in `crates/ecstore/src/rpc/remote_disk.rs` | `TcpHttpInternodeDataTransport::open_write` in `crates/ecstore/src/rpc/internode_data_transport.rs`, `HttpWriter` in `crates/rio/src/http_reader.rs` | `handle_put_file` in `rustfs/src/storage/rpc/http_service.rs` |
+| Walk dir stream | `RemoteDisk::walk_dir` in `crates/ecstore/src/rpc/remote_disk.rs` | `TcpHttpInternodeDataTransport::open_walk_dir`, `HttpReader` | `handle_walk_dir` in `rustfs/src/storage/rpc/http_service.rs` |
+
+Object read/write/heal callers enter these streams through
+`create_bitrot_reader` and `create_bitrot_writer` in
+`crates/ecstore/src/bitrot.rs`. Erasure decode and encode then move data
+through `ParallelReader` in `crates/ecstore/src/erasure_coding/decode.rs` and
+`MultiWriter` in `crates/ecstore/src/erasure_coding/encode.rs`.
+
+## Read Stream
+
+| Step | Owner | Buffer type | Copy? | Reason |
+| --- | --- | --- | --- | --- |
+| Build request | `RemoteDisk::read_file_stream` | `String` fields in `ReadStreamRequest` | Yes | Volume, path, endpoint, and disk references are copied into an owned request before async transport dispatch. This is metadata, not payload. |
+| Select transport | `TcpHttpInternodeDataTransport::open_read` | URL `String`, `HeaderMap` | Yes | URL and auth headers are HTTP control data. No object bytes are copied here. |
+| Open local file on server | `handle_read_file`, `LocalDisk::read_file_stream` | `FileCacheReclaimReader` boxed as `FileReader` | No payload copy | The server owns an async file reader positioned at the requested offset. |
+| File to HTTP body | `read_file_body_stream` | `ReaderStream<AsyncRead>` yielding `Bytes` | Yes | `ReaderStream::with_capacity` reads from the file into chunk buffers. This is the file-to-network buffer materialization point. |
+| Length limiting | `rustfs_utils::net::bytes_stream` | `Bytes` | Usually no | `Bytes::truncate` adjusts the chunk view when the last chunk exceeds the requested length. It does not copy the retained prefix. |
+| HTTP receive | `HttpReader::with_capacity_and_stall_timeout` | `reqwest::Response::bytes_stream()` yielding `Bytes` | Network stack dependent | The user-level object is `Bytes`; any kernel/TLS/hyper copy is below the current RustFS abstraction. |
+| Stream to caller buffer | `HttpReader::poll_read` | `StreamReader<Stream<Item = Bytes>>`, caller `ReadBuf` | Yes | `StreamReader` exposes `AsyncRead`, so it copies bytes from each `Bytes` chunk into the caller-provided `ReadBuf`. |
+| Bitrot verification | `BitrotReader::read` | caller `&mut [u8]`, `hash_buf: Vec<u8>` | No additional payload copy | The bitrot reader reads hash bytes into `hash_buf` and payload bytes directly into the supplied output slice. Hash calculation reads the slice. |
+| Erasure shard read | `ParallelReader::read` | `Vec<u8>` per shard | Yes | Each shard read allocates `vec![0u8; shard_size]`; data is filled there before decode/reconstruction. |
+| Object response write | `write_data_blocks` | slices of shard `Vec<u8>` | No extra staging copy | Decoded data block slices are written to the target writer with `write_all`; the target may copy internally. |
+| Remote zero-copy helper | `RemoteDisk::read_file_zero_copy` | `Vec<u8>` then `Bytes` | Yes | The remote implementation reads the full stream into a `Vec` and converts it into `Bytes`. It is a convenience fallback, not network zero-copy. |
+
+## Write Stream
+
+| Step | Owner | Buffer type | Copy? | Reason |
+| --- | --- | --- | --- | --- |
+| Build writer request | `RemoteDisk::create_file`, `RemoteDisk::append_file` | `String` fields in `WriteStreamRequest` | Yes | Volume, path, endpoint, and disk references are copied into an owned request. This is metadata. |
+| Select transport | `TcpHttpInternodeDataTransport::open_write` | URL `String`, `HeaderMap` | Yes | URL and auth headers are HTTP control data. No object bytes are copied here. |
+| Erasure encode input | `Erasure::encode` in `encode.rs` | reusable `Vec<u8>` sized to `block_size` | Yes | `rustfs_utils::read_full` fills a block buffer from the source reader before encoding. |
+| Erasure encode output | `Erasure::encode_data` caller in `encode.rs` | `Vec<Bytes>` per encoded block | Yes | Encoding creates shard `Bytes` for data and parity blocks before queueing them to writers. |
+| Multi-writer fanout | `MultiWriter::write` | borrowed `Bytes` shards | No additional fanout copy | The writer fanout passes borrowed `Bytes` references to each `BitrotWriterWrapper`. |
+| Bitrot write | `BitrotWriter::write` | shard `&[u8]`, checksum bytes | Yes for checksum bytes | The payload slice is passed to the inner writer, while checksum bytes are generated and written before payload when enabled. |
+| Client HTTP writer buffer | `HttpWriter::poll_write` and `poll_write_vectored` | `BytesMut` pending chunk or `Bytes::copy_from_slice` | Yes | Small writes are coalesced with `BytesMut::extend_from_slice`; large single writes still copy into an owned `Bytes` because the async body must outlive the caller's borrowed buffer. |
+| Client channel to reqwest | `HttpWriter::poll_send_pending_chunk`, `ReceiverStream` | `Bytes` | No | `BytesMut::split().freeze()` transfers owned chunk storage to `Bytes`; the mpsc channel and stream move the `Bytes` handle. |
+| HTTP receive body on server | `handle_put_file` | `Incoming::into_data_stream()` yielding `Bytes` | Network stack dependent | The server receives owned `Bytes` chunks from hyper. |
+| Server body coalescing | `write_body_chunks_to_writer` | `BytesMut` sized to `DEFAULT_READ_BUFFER_SIZE` | Yes | Each incoming `Bytes` chunk is copied into `pending` before writing to the local file writer. This normalizes chunk size but adds a full payload copy. |
+| Local file write | `LocalDisk::create_file`, `LocalDisk::append_file`, `FileCacheReclaimWriter` | `&[u8]` into `tokio::fs::File` | Kernel dependent | RustFS passes slices to Tokio file writes. Kernel page-cache copies are below the RustFS abstraction. |
+
+## Request and Serialization Boundaries
+
+| Boundary | Owner | Buffer type | Copy? | Notes |
+| --- | --- | --- | --- | --- |
+| Read/write query parameters | `build_read_file_stream_url`, `build_put_file_stream_url` | URL-encoded `String` | Yes | Metadata only. It includes disk, volume, path, offset, length, append, and size. |
+| Auth headers | `build_auth_headers` callers | `HeaderMap` | Yes | Metadata only. This is currently tied to HTTP request construction. |
+| Walk dir request | `RemoteDisk::walk_dir`, `open_walk_dir`, `handle_walk_dir` | JSON `Vec<u8>` body, collected `Bytes` on server | Yes | Walk dir is a streamed response but its request body is serialized JSON control data. |
+| gRPC read/write-all | `RemoteDisk::read_all`, `RemoteDisk::write_all`, `NodeService::{handle_read_all,handle_write_all}` | Prost `Bytes`/message bodies | Yes | These paths are still gRPC byte paths, not `InternodeDataTransport`; they matter for metrics and inventory but are outside this P1-D stream-copy count. |
+
+## Hotspots
+
+| Rank | Hotspot | Impact | Reason |
+| --- | --- | --- | --- |
+| 1 | `HttpWriter::poll_write` and `poll_write_vectored` | High on write path | Every borrowed caller buffer is copied into owned `BytesMut` or `Bytes` before it can be sent by an async HTTP body. |
+| 2 | `write_body_chunks_to_writer` | High on write path | The server copies every received `Bytes` chunk into a coalescing `BytesMut` before local disk write. |
+| 3 | `ParallelReader::read` shard buffers | High on read path | Each shard read allocates and fills a `Vec<u8>` before decode can proceed. This is also where degraded reads wait on quorum. |
+| 4 | `ReaderStream::with_capacity` plus `StreamReader` | Medium on read path | Server file reads create `Bytes` chunks, then client `AsyncRead` copies those chunks into the caller's `ReadBuf`. |
+| 5 | `Erasure::encode` block and shard materialization | Medium on write path | Source data is first read into a block `Vec<u8>`, then encoded into per-shard `Bytes`. This is necessary for the current erasure API. |
+| 6 | `RemoteDisk::read_file_zero_copy` | Medium when used | Remote zero-copy reads buffer the whole stream into memory. The name does not mean zero-copy over the network. |
+| 7 | URL/query/header/JSON serialization | Low | Metadata copies are small and not on the large payload hot path. |
+
+## Adapter Ownership Gaps
+
+1. `FileReader` and `FileWriter` are boxed `AsyncRead`/`AsyncWrite` trait
+   objects. They expose borrowed buffers per poll, not stable backend-owned
+   regions, transfer handles, or explicit completion ownership.
+2. `InternodeDataTransport` currently returns stream traits only. Its
+   capabilities advertise that TCP/HTTP does not require backend-specific
+   buffer registration and is not a zero-copy candidate, but there is no
+   backend API to pass stable backend-managed buffers.
+3. `HttpWriter` must own outgoing chunks because the async request body outlives
+   the caller's borrowed `&[u8]`. A lower-copy backend would need a different
+   lifetime contract or an owned buffer pool.
+4. Server write handling normalizes all incoming body chunks into a new
+   `BytesMut`. Avoiding that copy would require passing incoming `Bytes` or
+   backend-owned receive buffers directly into the disk/bitrot writer contract.
+5. Erasure decode owns shard `Vec<u8>` buffers and write-back happens through
+   `AsyncWrite`. A lower-copy backend would need explicit ownership of shard
+   buffers across decode, reconstruction, and network completion.
+6. Erasure encode materializes `Vec<Bytes>` blocks before fanout. A
+   backend that can send multiple stable slices would need an encode output
+   representation that can be transferred without repacking.
+7. The HTTP auth and URL construction boundary is part of the current TCP/HTTP
+   backend. A non-HTTP backend would need equivalent peer authentication and
+   disk addressing without assuming URL query parameters.
+8. Local disk zero-copy exists only for local reads via `read_file_zero_copy`.
+   Remote disks deliberately fall back to network streaming and full-buffer
+   collection for the zero-copy helper.

crates/ecstore/docs/internode-transport/transport-capabilities.md

@@ -0,0 +1,67 @@
+# Internode Transport Capabilities
+
+Status: design note for backend-neutral capability reporting. This document
+does not add a backend or transport crate.
+
+## Open-source Scope
+
+The OSS scope is:
+
+- define honest capability reporting for the `InternodeDataTransport` adapter;
+- keep `tcp-http` as the default backend;
+- keep existing TCP/HTTP behavior unchanged;
+- document the capability fields needed for maintainable transport code;
+- avoid adding dependencies or backend implementations.
+
+The OSS scope is not:
+
+- adding another transport backend;
+- replacing the current TCP/HTTP path;
+- adding benchmark plans for another transport;
+- changing object correctness semantics.
+
+## Purpose
+
+`InternodeDataTransportCapabilities` describes what a backend can honestly do
+for RustFS internode data-plane transfers. The fields describe observable
+adapter behavior without naming a specific transport implementation.
+
+The capability report is descriptive. It does not select a backend, negotiate
+with peers, or weaken object correctness semantics.
+
+## Capability Fields
+
+| Field | Meaning |
+| --- | --- |
+| `streaming_read` | The backend can open a remote disk reader for `read_file_stream`. |
+| `streaming_write` | The backend can open a remote disk writer for `create_file` or `append_file`. |
+| `streaming_walk_dir` | The backend can stream `walk_dir` responses. |
+| `ordered_delivery` | Bytes for each opened transfer are delivered in order. |
+| `max_transfer_size` | Optional RustFS-level cap for a single transfer. `None` means no additional cap beyond the backend/protocol/runtime limits. |
+| `fallback_supported` | The backend can participate in the behavior-preserving TCP fallback path. |
+
+## TCP/HTTP Backend
+
+The default TCP/HTTP backend reports only capabilities it actually provides:
+
+| Field | TCP/HTTP value | Reason |
+| --- | --- | --- |
+| `streaming_read` | `true` | `HttpReader` streams `/rustfs/rpc/read_file_stream` responses. |
+| `streaming_write` | `true` | `HttpWriter` streams `/rustfs/rpc/put_file_stream` request bodies. |
+| `streaming_walk_dir` | `true` | `HttpReader` streams `/rustfs/rpc/walk_dir` responses. |
+| `ordered_delivery` | `true` | Each HTTP request body or response body is consumed as an ordered byte stream. |
+| `max_transfer_size` | `None` | RustFS does not impose an extra per-transfer cap at the capability layer. |
+| `fallback_supported` | `true` | TCP/HTTP is the behavior-preserving default and fallback path. |
+
+## Capability Compatibility
+
+Any new capability field should describe observable RustFS behavior without
+assuming a specific transport implementation:
+
+| Capability shape | Interpretation |
+| --- | --- |
+| `max_transfer_size=Some(n)` | The backend has a RustFS-visible transfer size ceiling and callers must split larger transfers or use fallback behavior. |
+| `ordered_delivery=false` | The backend cannot be used behind the current stream API without an ordering or reassembly layer. |
+
+Unsupported or mismatched capabilities must not silently change quorum,
+integrity verification, retry, or timeout semantics.

crates/ecstore/docs/internode-transport/transport-fallback-and-selection.md

@@ -0,0 +1,135 @@
+# Internode Transport Fallback and Backend Selection Model
+
+Status: design note only. This document defines backend-neutral selection,
+fallback, failure handling, negotiation, security, and observability rules for
+the `InternodeDataTransport` adapter. It does not implement a new backend and
+does not change production behavior.
+
+## Open-source Scope
+
+The open-source RustFS path keeps `tcp-http` as the default internode data
+transport. This document defines adapter contracts only:
+
+- no additional production backend is introduced;
+- no dependency is added;
+- no new accepted production backend value is added;
+- RustFS core data-plane logic remains independent of the concrete transport
+  implementation.
+
+## Static Backend Selection
+
+Static config is the first selection model. Existing accepted values remain:
+
+| Config value | Meaning |
+| --- | --- |
+| unset | Use default TCP/HTTP backend. |
+| `tcp-http` | Use default TCP/HTTP backend. |
+| `tcp` | Alias for `tcp-http`. |
+| any unsupported value | Fail closed with a diagnostic naming `RUSTFS_INTERNODE_DATA_TRANSPORT` and the invalid value. |
+
+Unknown backend values must fail closed. Unsupported backend values must fail
+closed. Any additional backend value must be explicitly added and must not
+silently replace `tcp-http`.
+
+Backend selection must expose an observable backend identity for metrics, logs,
+and benchmark interpretation. The default and fallback path remains `tcp-http`.
+
+## Fallback Contract
+
+Fallback must be explicit and observable. Silent fallback is not allowed for
+benchmark or production interpretation because it hides which backend moved the
+payload.
+
+| Condition | Default behavior | Explicit fallback behavior | Observability |
+| --- | --- | --- | --- |
+| Unsupported configured backend | Fail closed during transport construction. | Fall back only when a separately configured policy explicitly allows unsupported-backend fallback. | Error includes config key and invalid value; fallback event is counted when fallback is enabled. |
+| Peer does not support selected backend | Fail before payload transfer. | Use TCP/HTTP only when both local policy and peer policy allow it. | Count peer mismatch and selected fallback backend. |
+| Capability mismatch | Fail before payload transfer. | Use TCP/HTTP only if it satisfies the operation and policy allows fallback. | Record missing capability names or a low-cardinality reason. |
+| Connection setup failure | Fail the operation. | Retry on TCP/HTTP only when fallback is allowed and no payload bytes have transferred. | Count setup failure, retry, fallback backend, and fallback result. |
+| Partial transfer failure | Fail the operation and let existing object/quorum logic decide retry behavior. | Do not silently resume on another backend unless the transfer protocol can prove byte range, checksum, and idempotency boundaries. | Count partial failure with bytes completed. |
+| Max transfer size exceeded | Fail before payload transfer or split at a higher layer. | Use TCP/HTTP if policy allows and TCP has no RustFS-level cap. | Record rejected size and selected backend. |
+| Auth or encryption mismatch | Fail closed. | No fallback unless the fallback path satisfies the same or stronger security requirements. | Security failure metric and audit log entry. |
+
+Fallback settings should not be added until there is an implementation that
+uses them. A backend must define failure behavior before production use.
+
+## Dynamic Negotiation Boundary
+
+Dynamic negotiation is not implemented by this PR. If it is added later, it
+belongs on the existing gRPC control plane. Data transfer must start only after
+both peers agree on:
+
+| Negotiated item | Required property |
+| --- | --- |
+| Backend name | Both peers know the backend and have it enabled. |
+| Capability set | Required capabilities match the operation. |
+| Max transfer size | The selected operation fits or is split before transfer starts. |
+| Buffer rules | Both peers agree on staging and ownership rules. |
+| Completion semantics | Both peers agree when a transfer is considered complete and when buffers may be reused. |
+| Security mode | Authentication and encryption requirements are satisfied before any out-of-band transfer. |
+| Fallback policy | Both peers agree whether TCP/HTTP fallback is allowed for this operation. |
+
+Negotiation must not silently downgrade security or bypass existing disk
+health, quorum, timeout, and integrity semantics.
+
+## Failure Handling Requirements
+
+| Failure mode | Requirement |
+| --- | --- |
+| Invalid config | Fail closed with `RUSTFS_INTERNODE_DATA_TRANSPORT` and the invalid value. |
+| Backend disabled | Fail closed with the selected backend name and the missing enablement condition. |
+| Backend unavailable | Fail closed with an actionable diagnostic; do not silently use TCP/HTTP. |
+| Peer mismatch | Fail before payload transfer unless explicit fallback is configured. |
+| Connection failure | Fail the operation and record setup failure; fallback only if policy allows and no payload bytes moved. |
+| Completion failure | Return an operation error and release backend-owned resources. |
+| Timeout | Return an operation error and preserve existing disk health and quorum semantics. |
+| Partial transfer | Do not silently resume on another backend without a safe byte-range/checksum/idempotency proof. |
+| Unsupported operation | Return a clear unsupported-operation error. |
+
+## Security Requirements
+
+- Backend selection must preserve peer authentication.
+- Fallback must not weaken encryption or authorization.
+- Partial transfers must not bypass bitrot verification or erasure quorum
+  handling.
+- Any adapter implementation must preserve the same request authority, disk,
+  volume, path, and operation binding as the current TCP/HTTP path.
+
+## Metrics and Observability Requirements
+
+Metrics and logs must use low-cardinality labels or metadata:
+
+- selected backend;
+- requested backend;
+- fallback backend, when used;
+- operation name;
+- success/failure;
+- transferred bytes;
+- setup failure count;
+- partial transfer failure count;
+- capability mismatch count;
+- fallback decision count.
+
+Adapter implementations must not add high-cardinality labels such as object
+names, full paths, full URLs, peer-specific dynamic strings, memory addresses,
+or buffer identifiers.
+
+## TCP/HTTP Compatibility
+
+The `tcp-http` backend remains the default and behavior-preserving path. It
+uses ordinary byte streams, does not require backend-specific buffer
+registration, and remains suitable as the fallback path when an explicit
+fallback policy exists.
+
+Any adapter implementation must not change the correctness semantics of
+object writes, object reads, healing, bitrot verification, erasure quorum,
+timeouts, or disk health handling.
+
+## Adapter Stability
+
+`InternodeDataTransport` should keep RustFS core data-plane logic separate from
+the concrete transport implementation. The trait and `tcp-http` backend remain
+inside `ecstore`.
+
+This PR does not perform a crate split, add runtime loading, introduce a plugin
+system, add a backend value, or implement a new transport backend.

crates/ecstore/docs/internode-transport/transport-metrics-and-baseline.md

@@ -0,0 +1,143 @@
+# Internode Transport Metrics and Baseline
+
+Status: current OSS behavior. This document describes the metrics and baseline
+runner used to observe the existing `tcp-http` internode transport adapter. It
+does not add a backend, change runtime behavior, or make performance claims.
+
+## Scope
+
+The current OSS adapter scope is:
+
+- `tcp-http` is the default and only supported internode data transport backend;
+- `tcp` is a legacy alias for `tcp-http`;
+- unsupported backend values fail closed during transport construction;
+- metrics and baseline artifacts are used to understand the current data-plane
+  behavior.
+
+## Operation Metrics
+
+Operation-level internode metrics are defined in
+`crates/io-metrics/src/internode_metrics.rs`.
+
+| Metric | Labels | Meaning |
+| --- | --- | --- |
+| `rustfs_system_network_internode_operation_sent_bytes_total` | `operation`, `backend` | Bytes sent for a known internode operation. |
+| `rustfs_system_network_internode_operation_recv_bytes_total` | `operation`, `backend` | Bytes received for a known internode operation. |
+| `rustfs_system_network_internode_operation_requests_outgoing_total` | `operation`, `backend` | Outgoing requests opened for a known internode operation. |
+| `rustfs_system_network_internode_operation_requests_incoming_total` | `operation`, `backend` | Incoming requests handled for a known internode operation. |
+| `rustfs_system_network_internode_operation_errors_total` | `operation`, `backend` | Errors observed for a known internode operation. |
+
+Current operation label values are:
+
+| Operation | Transport path | Notes |
+| --- | --- | --- |
+| `read_file_stream` | HTTP `/rustfs/rpc/read_file_stream` | Remote disk read stream opened by `InternodeDataTransport::open_read`. |
+| `put_file_stream` | HTTP `/rustfs/rpc/put_file_stream` | Remote disk write stream opened by `InternodeDataTransport::open_write`. |
+| `walk_dir` | HTTP `/rustfs/rpc/walk_dir` | Remote walk-dir stream opened by `InternodeDataTransport::open_walk_dir`. |
+| `grpc_read_all` | gRPC `ReadAll` | Unary bytes response outside the adapter. |
+| `grpc_write_all` | gRPC `WriteAll` | Unary bytes request outside the adapter. |
+
+Current backend label values are:
+
+| Backend | Meaning |
+| --- | --- |
+| `tcp-http` | Current HTTP stream backend for adapter-routed operations. |
+| `grpc` | Current gRPC path for `ReadAll` and `WriteAll`. |
+| `unknown` | Aggregate internode metric path when an operation-specific backend is not available. |
+
+The `backend` label reflects the current instrumentation source, for example
+the `tcp-http` stream path, gRPC path, or aggregate unknown path. It does not
+imply additional supported transport backends.
+
+Metric labels must stay low cardinality. Do not add object names, full paths,
+full URLs, peer-specific dynamic strings, request IDs, or other per-request
+values as labels.
+
+## Current Instrumentation Points
+
+| Area | File | Current signal |
+| --- | --- | --- |
+| Outgoing HTTP stream requests | `crates/rio/src/http_reader.rs` | Outgoing request count, sent bytes for writer bodies, received bytes for reader bodies, and errors for known `/rustfs/rpc/` operations. |
+| Incoming HTTP stream requests | `rustfs/src/storage/rpc/http_service.rs` | Incoming request count, sent bytes for read and walk streams, received bytes for put streams, and route errors. |
+| gRPC `ReadAll` / `WriteAll` | `rustfs/src/storage/rpc/disk.rs` | Incoming request count, sent/received bytes, and errors with the `grpc` backend label. |
+
+Known gaps:
+
+- Operation-level metrics do not currently expose latency histograms.
+- The baseline runner reads Prometheus text output only when `--metrics-url` is
+  provided.
+- Metrics are useful for attribution, but they do not replace end-to-end
+  correctness tests or object benchmark output.
+
+## Baseline Runner
+
+Use `scripts/run_internode_transport_baseline.sh` to run the current S3 PUT/GET
+matrix against one or more configured endpoints.
+
+Required inputs:
+
+- `--access-key`
+- `--secret-key`
+
+Common optional inputs:
+
+- `--tool warp|s3bench`
+- `--scenarios name=url,...`
+- `--sizes 4KiB,1MiB,...`
+- `--concurrencies 1,16,...`
+- `--duration 90s`
+- `--metrics-url http://host:port/metrics`
+- `--out-dir target/bench/internode-transport/manual-run`
+- `--dry-run`
+
+Dry-run example:
+
+```bash
+scripts/run_internode_transport_baseline.sh \
+  --access-key minioadmin \
+  --secret-key minioadmin \
+  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
+  --sizes 4KiB,1MiB \
+  --concurrencies 1 \
+  --duration 10s \
+  --dry-run
+```
+
+Real baseline example with metrics:
+
+```bash
+RUSTFS_INTERNODE_DATA_TRANSPORT=tcp-http \
+scripts/run_internode_transport_baseline.sh \
+  --access-key "$RUSTFS_ACCESS_KEY" \
+  --secret-key "$RUSTFS_SECRET_KEY" \
+  --scenarios local=http://127.0.0.1:9000,distributed=http://127.0.0.1:9001 \
+  --metrics-url http://127.0.0.1:9000/metrics \
+  --out-dir target/bench/internode-transport/manual-run
+```
+
+The real baseline requires a running RustFS deployment and working benchmark
+tool. Do not invent or copy synthetic benchmark numbers into PR descriptions.
+
+## Output Artifacts
+
+The baseline runner writes:
+
+| File | Created when | Contents |
+| --- | --- | --- |
+| `run_manifest.txt` | Always | Timestamp, git commit, dirty flag, Rust compiler version, kernel string, scenario matrix, tool, workload settings, metrics URL, and redacted credentials. |
+| `summary.csv` | Always | `scenario`, `endpoint`, `workload`, `concurrency`, `size`, `status`, `throughput`, `requests_per_sec`, `avg_latency`, `error_count`, `log_file`, and `run_dir`. |
+| `internode_metric_deltas.csv` | When `--metrics-url` is set | `scenario`, `workload`, `concurrency`, `size`, metric name, operation, backend, before value, after value, and delta. |
+
+In dry-run mode, the runner still creates the manifest and CSV headers, and the
+underlying object benchmark command is printed with credentials redacted.
+
+## Interpretation
+
+Use baseline artifacts to record and inspect the current `tcp-http` adapter
+behavior across commits, environments, and scenario matrices. A baseline run is
+valid only for the exact environment and command recorded in
+`run_manifest.txt`.
+
+Baseline artifacts should state whether metrics were collected. When
+`internode_metric_deltas.csv` is absent, benchmark output cannot attribute
+internode operation deltas from Prometheus metrics.

crates/ecstore/src/admin_server_info.rs

@@ -17,7 +17,7 @@ use crate::error::{Error, Result};
 use crate::rpc::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client};
 use crate::{
     disk::endpoint::Endpoint,
-    global::{GLOBAL_BOOT_TIME, GLOBAL_Endpoints},
+    global::{GLOBAL_BOOT_TIME, GLOBAL_Endpoints, get_global_deployment_id},
     new_object_layer_fn,
     notification_sys::get_global_notification_sys,
     store_api::StorageAPI,
@@ -291,7 +291,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
         domain: None,
         region: None,
         sqs_arn: None,
-        deployment_id: None,
+        deployment_id: get_global_deployment_id(),
         buckets: Some(buckets),
         objects: Some(objects),
         versions: Some(versions),
@@ -393,3 +393,21 @@ pub fn get_commit_id() -> String {
 
     format!("{}@{}", build::COMMIT_DATE_3339, ver)
 }
+
+#[cfg(test)]
+mod tests {
+    use serial_test::serial;
+
+    use crate::global::get_global_deployment_id;
+
+    use super::get_server_info;
+
+    #[serial]
+    #[tokio::test]
+    async fn server_info_includes_global_deployment_id() {
+        let expected_deployment_id = get_global_deployment_id();
+        let info = get_server_info(false).await;
+
+        assert_eq!(info.deployment_id, expected_deployment_id);
+    }
+}

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -44,14 +44,19 @@ use async_channel::{Receiver as A_Receiver, Sender as A_Sender, bounded};
 use futures::Future;
 use http::HeaderMap;
 use lazy_static::lazy_static;
-use rustfs_common::data_usage::TierStats;
 use rustfs_common::heal_channel::rep_has_active_rules;
 use rustfs_common::metrics::{IlmAction, Metrics};
+use rustfs_config::{
+    DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
+    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, ENV_TRANSITION_QUEUE_CAPACITY,
+    ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS, ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
+};
+use rustfs_data_usage::TierStats;
 use rustfs_filemeta::{
     FileInfo, FileInfoOpts, NULL_VERSION_ID, REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicationState, RestoreStatusOps,
     VersionPurgeStatusType, get_file_info, is_restored_object_on_disk,
 };
-use rustfs_s3_common::EventName;
+use rustfs_s3_types::EventName;
 use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::strings_has_prefix_fold};
 use s3s::dto::{
     BucketLifecycleConfiguration, DefaultRetention, ExpirationStatus, ReplicationConfiguration, RestoreRequest,
@@ -60,7 +65,7 @@ use s3s::dto::{
 use s3s::header::{X_AMZ_RESTORE, X_AMZ_SERVER_SIDE_ENCRYPTION};
 use sha2::{Digest, Sha256};
 use std::any::Any;
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
 use std::env;
 use std::pin::Pin;
 use std::sync::atomic::{AtomicI64, Ordering};
@@ -99,6 +104,57 @@ lazy_static! {
     pub static ref GLOBAL_TransitionState: Arc<TransitionState> = TransitionState::new();
 }
 
+fn resolve_transition_worker_count() -> (i64, i64, i64) {
+    let fallback = std::cmp::min(num_cpus::get() as i64, DEFAULT_TRANSITION_WORKERS_CAP);
+    let configured = env::var(ENV_TRANSITION_WORKERS)
+        .ok()
+        .and_then(|value| value.parse::<i64>().ok())
+        .filter(|value| *value > 0)
+        .unwrap_or(fallback);
+    let mut effective = configured;
+    let absolute_max = resolve_transition_workers_absolute_max();
+    effective = std::cmp::min(effective, absolute_max);
+    (configured, absolute_max, effective)
+}
+
+fn resolve_transition_workers_absolute_max() -> i64 {
+    let absolute_max = get_env_i64(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX);
+    if absolute_max > 0 {
+        absolute_max
+    } else {
+        DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX
+    }
+}
+
+fn resolve_transition_queue_capacity() -> usize {
+    get_env_usize(ENV_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_QUEUE_CAPACITY).max(1)
+}
+
+fn resolve_transition_queue_send_timeout() -> StdDuration {
+    StdDuration::from_millis(
+        get_env_usize(ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS).max(1) as u64,
+    )
+}
+
+fn is_immediate_transition_source(src: &LcEventSrc) -> bool {
+    matches!(
+        src,
+        LcEventSrc::S3PutObject | LcEventSrc::S3CopyObject | LcEventSrc::S3CompleteMultipartUpload
+    )
+}
+
+#[cfg(any(test, debug_assertions))]
+fn should_force_immediate_transition_enqueue_timeout() -> bool {
+    env::var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT)
+        .ok()
+        .is_some_and(|value| value == "1")
+}
+
+#[cfg(not(any(test, debug_assertions)))]
+fn should_force_immediate_transition_enqueue_timeout() -> bool {
+    false
+}
+
 pub struct LifecycleSys;
 
 impl LifecycleSys {
@@ -537,15 +593,34 @@ pub struct TransitionState {
     pub num_workers: AtomicI64,
     kill_tx: A_Sender<()>,
     kill_rx: A_Receiver<()>,
+    transition_queue_capacity: usize,
+    transition_queue_send_timeout: StdDuration,
     active_tasks: AtomicI64,
     missed_immediate_tasks: AtomicI64,
+    queue_full_tasks: AtomicI64,
+    queue_send_timeout_tasks: AtomicI64,
+    compensation_scheduled_tasks: AtomicI64,
+    compensation_running_tasks: AtomicI64,
+    compensation_buckets: Arc<Mutex<HashSet<String>>>,
     last_day_stats: Arc<Mutex<HashMap<String, LastDayTierStats>>>,
 }
 
+enum ImmediateEnqueueFailure {
+    ForcedTimeout,
+    QueueClosed { timeout_ms: Option<u64> },
+    QueueSendTimedOut { timeout_ms: u64 },
+}
+
 impl TransitionState {
     #[allow(clippy::new_ret_no_self)]
     pub fn new() -> Arc<Self> {
-        let (tx1, rx1) = bounded(1000);
+        Self::new_with_capacity(resolve_transition_queue_capacity())
+    }
+
+    fn new_with_capacity(capacity: usize) -> Arc<Self> {
+        let capacity = capacity.max(1);
+        let queue_send_timeout = resolve_transition_queue_send_timeout();
+        let (tx1, rx1) = bounded(capacity);
         let (tx2, rx2) = bounded(1);
         Arc::new(Self {
             transition_tx: tx1,
@@ -553,39 +628,191 @@ impl TransitionState {
             num_workers: AtomicI64::new(0),
             kill_tx: tx2,
             kill_rx: rx2,
+            transition_queue_capacity: capacity,
+            transition_queue_send_timeout: queue_send_timeout,
             active_tasks: AtomicI64::new(0),
             missed_immediate_tasks: AtomicI64::new(0),
+            queue_full_tasks: AtomicI64::new(0),
+            queue_send_timeout_tasks: AtomicI64::new(0),
+            compensation_scheduled_tasks: AtomicI64::new(0),
+            compensation_running_tasks: AtomicI64::new(0),
+            compensation_buckets: Arc::new(Mutex::new(HashSet::new())),
             last_day_stats: Arc::new(Mutex::new(HashMap::new())),
         })
     }
 
-    pub async fn queue_transition_task(&self, oi: &ObjectInfo, event: &lifecycle::Event, src: &LcEventSrc) {
+    fn schedule_bucket_compensation(self: &Arc<Self>, bucket: &str) -> bool {
+        let mut scheduled = self.compensation_buckets.lock().unwrap();
+        if !scheduled.insert(bucket.to_string()) {
+            return false;
+        }
+        Self::inc_counter(&self.compensation_scheduled_tasks);
+        let bucket = bucket.to_string();
+        let scheduled = Arc::clone(&self.compensation_buckets);
+        let state = Arc::clone(self);
+        tokio::spawn(async move {
+            Self::inc_counter(&state.compensation_running_tasks);
+            let Some(api) = crate::new_object_layer_fn() else {
+                scheduled.lock().unwrap().remove(&bucket);
+                Self::add_counter(&state.compensation_running_tasks, -1);
+                warn!(bucket = %bucket, "transition compensation skipped because object layer is unavailable");
+                return;
+            };
+
+            if let Err(err) = enqueue_transition_for_existing_objects(api, &bucket).await {
+                warn!(bucket = %bucket, error = ?err, "transition compensation backfill failed");
+            } else {
+                info!(bucket = %bucket, "transition compensation backfill completed");
+            }
+
+            scheduled.lock().unwrap().remove(&bucket);
+            Self::add_counter(&state.compensation_running_tasks, -1);
+        });
+        true
+    }
+
+    #[inline]
+    fn inc_counter(counter: &AtomicI64) {
+        Self::add_counter(counter, 1);
+    }
+
+    #[inline]
+    fn add_counter(counter: &AtomicI64, delta: i64) {
+        counter.fetch_add(delta, Ordering::Relaxed);
+    }
+
+    #[inline]
+    fn counter_value(counter: &AtomicI64) -> i64 {
+        counter.load(Ordering::Relaxed)
+    }
+
+    fn handle_immediate_enqueue_failure(self: &Arc<Self>, oi: &ObjectInfo, src: &LcEventSrc, failure: ImmediateEnqueueFailure) {
+        Self::inc_counter(&self.missed_immediate_tasks);
+        let scheduled = self.schedule_bucket_compensation(&oi.bucket);
+        match failure {
+            ImmediateEnqueueFailure::ForcedTimeout => {
+                Self::inc_counter(&self.queue_send_timeout_tasks);
+                warn!(
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    source = ?src,
+                    compensation_scheduled = scheduled,
+                    "transition enqueue forced into timeout path for test fault injection"
+                );
+            }
+            ImmediateEnqueueFailure::QueueClosed { timeout_ms } => match timeout_ms {
+                Some(timeout_ms) => {
+                    warn!(
+                        bucket = %oi.bucket,
+                        object = %oi.name,
+                        source = ?src,
+                        timeout_ms,
+                        compensation_scheduled = scheduled,
+                        "transition enqueue failed because the queue is closed"
+                    );
+                }
+                None => {
+                    warn!(
+                        bucket = %oi.bucket,
+                        object = %oi.name,
+                        source = ?src,
+                        compensation_scheduled = scheduled,
+                        "transition enqueue failed because the queue is closed"
+                    );
+                }
+            },
+            ImmediateEnqueueFailure::QueueSendTimedOut { timeout_ms } => {
+                Self::inc_counter(&self.queue_send_timeout_tasks);
+                warn!(
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    source = ?src,
+                    timeout_ms,
+                    compensation_scheduled = scheduled,
+                    "transition enqueue timed out under backpressure"
+                );
+            }
+        }
+    }
+
+    pub async fn queue_transition_task(self: &Arc<Self>, oi: &ObjectInfo, event: &lifecycle::Event, src: &LcEventSrc) {
+        if is_immediate_transition_source(src) && should_force_immediate_transition_enqueue_timeout() {
+            self.handle_immediate_enqueue_failure(oi, src, ImmediateEnqueueFailure::ForcedTimeout);
+            return;
+        }
+
         let task = TransitionTask {
             obj_info: oi.clone(),
             src: src.clone(),
             event: event.clone(),
         };
-        select! {
-            //_ -> t.ctx.Done() => (),
-            _ = self.transition_tx.send(Some(task)) => (),
-            else => {
-                match src {
-                    LcEventSrc::S3PutObject | LcEventSrc::S3CopyObject | LcEventSrc::S3CompleteMultipartUpload => {
-                        self.missed_immediate_tasks.fetch_add(1, Ordering::SeqCst);
+        if is_immediate_transition_source(src) {
+            match self.transition_tx.try_send(Some(task)) {
+                Ok(()) => {}
+                Err(async_channel::TrySendError::Full(task)) => {
+                    Self::inc_counter(&self.queue_full_tasks);
+                    let send_timeout = self.transition_queue_send_timeout;
+                    match tokio::time::timeout(send_timeout, self.transition_tx.send(task)).await {
+                        Ok(Ok(())) => {}
+                        Ok(Err(_)) => {
+                            self.handle_immediate_enqueue_failure(
+                                oi,
+                                src,
+                                ImmediateEnqueueFailure::QueueClosed {
+                                    timeout_ms: Some(send_timeout.as_millis() as u64),
+                                },
+                            );
+                        }
+                        Err(_) => {
+                            self.handle_immediate_enqueue_failure(
+                                oi,
+                                src,
+                                ImmediateEnqueueFailure::QueueSendTimedOut {
+                                    timeout_ms: send_timeout.as_millis() as u64,
+                                },
+                            );
+                        }
                     }
-                    _ => ()
                 }
-            },
+                Err(async_channel::TrySendError::Closed(_task)) => {
+                    self.handle_immediate_enqueue_failure(oi, src, ImmediateEnqueueFailure::QueueClosed { timeout_ms: None });
+                }
+            }
+            return;
+        }
+
+        if let Err(err) = self.transition_tx.try_send(Some(task)) {
+            match err {
+                async_channel::TrySendError::Full(_) => {
+                    debug!(
+                        bucket = %oi.bucket,
+                        object = %oi.name,
+                        source = ?src,
+                        "transition queue is full; deferring to scanner/backfill"
+                    );
+                }
+                async_channel::TrySendError::Closed(_) => {
+                    warn!(
+                        bucket = %oi.bucket,
+                        object = %oi.name,
+                        source = ?src,
+                        "transition enqueue failed because the queue is closed"
+                    );
+                }
+            }
         }
     }
 
     pub async fn init(api: Arc<ECStore>) {
-        let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
-        let mut n = max_workers;
-        let tw = 8; //globalILMConfig.getTransitionWorkers();
-        if tw > 0 {
-            n = tw;
-        }
+        let (configured, absolute_max, n) = resolve_transition_worker_count();
+        info!(
+            configured_transition_workers = configured,
+            absolute_max_workers = absolute_max,
+            effective_transition_workers = n,
+            transition_queue_capacity = GLOBAL_TransitionState.transition_queue_capacity,
+            transition_queue_send_timeout_ms = GLOBAL_TransitionState.transition_queue_send_timeout.as_millis() as u64,
+            "transition worker count resolved"
+        );
 
         //let mut transition_state = GLOBAL_TransitionState.write().await;
         //self.objAPI = objAPI
@@ -599,11 +826,27 @@ impl TransitionState {
     }
 
     pub fn active_tasks(&self) -> i64 {
-        self.active_tasks.load(Ordering::SeqCst)
+        Self::counter_value(&self.active_tasks)
     }
 
     pub fn missed_immediate_tasks(&self) -> i64 {
-        self.missed_immediate_tasks.load(Ordering::SeqCst)
+        Self::counter_value(&self.missed_immediate_tasks)
+    }
+
+    pub fn queue_full_tasks(&self) -> i64 {
+        Self::counter_value(&self.queue_full_tasks)
+    }
+
+    pub fn queue_send_timeout_tasks(&self) -> i64 {
+        Self::counter_value(&self.queue_send_timeout_tasks)
+    }
+
+    pub fn compensation_scheduled_tasks(&self) -> i64 {
+        Self::counter_value(&self.compensation_scheduled_tasks)
+    }
+
+    pub fn compensation_running_tasks(&self) -> i64 {
+        Self::counter_value(&self.compensation_running_tasks)
     }
 
     pub async fn worker(api: Arc<ECStore>) {
@@ -626,7 +869,7 @@ impl TransitionState {
                     if task.as_any().is::<TransitionTask>() {
                         let task = task.as_any().downcast_ref::<TransitionTask>().expect("TransitionTask downcast failed");
 
-                        GLOBAL_TransitionState.active_tasks.fetch_add(1, Ordering::SeqCst);
+                        TransitionState::inc_counter(&GLOBAL_TransitionState.active_tasks);
 
                         let obj_info_for_event = ObjectInfo {
                             bucket: task.obj_info.bucket.clone(),
@@ -671,7 +914,7 @@ impl TransitionState {
                                 ..Default::default()
                             });
                         }
-                        GLOBAL_TransitionState.active_tasks.fetch_add(-1, Ordering::SeqCst);
+                        TransitionState::add_counter(&GLOBAL_TransitionState.active_tasks, -1);
                     }
                 }
                 else => ()
@@ -702,15 +945,17 @@ impl TransitionState {
 
     pub async fn update_workers_inner(api: Arc<ECStore>, n: i64) {
         let mut n = n;
+        let requested = n;
         if n == 0 {
-            let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
-            n = max_workers;
+            let (_, _, effective) = resolve_transition_worker_count();
+            n = effective;
         }
         // Allow environment override of maximum workers
-        let absolute_max = get_env_i64("RUSTFS_ABSOLUTE_MAX_WORKERS", 32);
+        let absolute_max = resolve_transition_workers_absolute_max();
         n = std::cmp::min(n, absolute_max);
 
-        let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        let previous_num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        let mut num_workers = previous_num_workers;
         while num_workers < n {
             let clone_api = api.clone();
             tokio::spawn(async move {
@@ -727,6 +972,15 @@ impl TransitionState {
             num_workers -= 1;
             GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
         }
+
+        info!(
+            requested_transition_workers = requested,
+            effective_transition_workers = n,
+            absolute_max_workers = absolute_max,
+            previous_transition_workers = previous_num_workers,
+            current_transition_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst),
+            "transition workers updated"
+        );
     }
 }
 
@@ -2004,11 +2258,13 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc,
 #[cfg(test)]
 mod tests {
     use super::{
-        DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, StaleMultipartUploadCandidate, cleanup_empty_multipart_sha_dirs_on_local_disks,
-        cleanup_stale_multipart_uploads_once_at, lifecycle_deleted_object, lifecycle_rule_has_date_expiration,
-        lifecycle_version_purge_state_from_completed_targets, mark_delete_opts_skip_decommissioned_on_remote_success,
-        merge_stale_multipart_candidate, replication_state_for_delete, should_defer_date_expiry_for_recent_config_update,
-        should_reuse_lifecycle_delete_replication_state,
+        DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
+        DEFAULT_TRANSITION_WORKERS_CAP, GLOBAL_TransitionState, StaleMultipartUploadCandidate, TransitionState,
+        cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at, lifecycle_deleted_object,
+        lifecycle_rule_has_date_expiration, lifecycle_version_purge_state_from_completed_targets,
+        mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate, replication_state_for_delete,
+        resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
+        should_defer_date_expiry_for_recent_config_update, should_reuse_lifecycle_delete_replication_state,
     };
     use crate::bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
     use crate::bucket::metadata_sys;
@@ -2021,12 +2277,16 @@ mod tests {
     use crate::store_api::{
         BucketOperations, BucketOptions, MakeBucketOptions, MultipartOperations, ObjectInfo, ObjectOptions, PutObjReader,
     };
+    use futures::FutureExt;
+    use rustfs_config::ENV_TRANSITION_WORKERS_ABSOLUTE_MAX;
     use rustfs_filemeta::{ReplicateDecision, VersionPurgeStatusType};
     use s3s::dto::{BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, Timestamp};
     use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::collections::HashMap;
+    use std::env;
     use std::path::PathBuf;
+    use std::sync::atomic::Ordering;
     use std::sync::{Arc, OnceLock};
     use std::time::Duration as StdDuration;
     use time::OffsetDateTime;
@@ -2043,6 +2303,216 @@ mod tests {
         assert!(opts.skip_decommissioned);
     }
 
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    fn with_transition_worker_env<F>(transition: Option<&str>, absolute: Option<&str>, test_fn: F)
+    where
+        F: FnOnce(),
+    {
+        let original_transition = env::var_os("RUSTFS_MAX_TRANSITION_WORKERS");
+        let original_absolute = env::var_os(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+
+        match transition {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_MAX_TRANSITION_WORKERS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_MAX_TRANSITION_WORKERS");
+            },
+        }
+        match absolute {
+            Some(value) => unsafe {
+                env::set_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            },
+        }
+
+        let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(test_fn));
+
+        match original_transition {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_MAX_TRANSITION_WORKERS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_MAX_TRANSITION_WORKERS");
+            },
+        }
+        match original_absolute {
+            Some(value) => unsafe {
+                env::set_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    async fn with_transition_worker_env_async<F, Fut>(transition: Option<&str>, absolute: Option<&str>, test_fn: F)
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = ()>,
+    {
+        let original_transition = env::var_os("RUSTFS_MAX_TRANSITION_WORKERS");
+        let original_absolute = env::var_os(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+
+        match transition {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_MAX_TRANSITION_WORKERS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_MAX_TRANSITION_WORKERS");
+            },
+        }
+        match absolute {
+            Some(value) => unsafe {
+                env::set_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            },
+        }
+
+        let result = std::panic::AssertUnwindSafe(test_fn()).catch_unwind().await;
+
+        match original_transition {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_MAX_TRANSITION_WORKERS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_MAX_TRANSITION_WORKERS");
+            },
+        }
+        match original_absolute {
+            Some(value) => unsafe {
+                env::set_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    fn with_transition_queue_env<F>(capacity: Option<&str>, timeout_ms: Option<&str>, test_fn: F)
+    where
+        F: FnOnce(),
+    {
+        let original_capacity = env::var_os("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+        let original_timeout = env::var_os("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+
+        match capacity {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_CAPACITY", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+            },
+        }
+        match timeout_ms {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+            },
+        }
+
+        let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(test_fn));
+
+        match original_capacity {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_CAPACITY", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+            },
+        }
+        match original_timeout {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    async fn with_transition_queue_env_async<F, Fut>(capacity: Option<&str>, timeout_ms: Option<&str>, test_fn: F)
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = ()>,
+    {
+        let original_capacity = env::var_os("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+        let original_timeout = env::var_os("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+
+        match capacity {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_CAPACITY", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+            },
+        }
+        match timeout_ms {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+            },
+        }
+
+        let result = std::panic::AssertUnwindSafe(test_fn()).catch_unwind().await;
+
+        match original_capacity {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_CAPACITY", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+            },
+        }
+        match original_timeout {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
     #[test]
     fn lifecycle_rule_has_date_expiration_detects_enabled_date_rule() {
         let lc = BucketLifecycleConfiguration {
@@ -2068,6 +2538,134 @@ mod tests {
         assert!(!lifecycle_rule_has_date_expiration(&lc, "missing-rule"));
     }
 
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_uses_fallback_when_env_missing() {
+        with_transition_worker_env(None, None, || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            let fallback = std::cmp::min(num_cpus::get() as i64, DEFAULT_TRANSITION_WORKERS_CAP);
+            assert_eq!(configured, fallback);
+            assert_eq!(absolute_max, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            assert_eq!(effective, fallback);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_honors_positive_env_value() {
+        with_transition_worker_env(Some("4"), Some("32"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            assert_eq!(configured, 4);
+            assert_eq!(absolute_max, 32);
+            assert_eq!(effective, 4);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_clamps_to_absolute_max() {
+        with_transition_worker_env(Some("64"), Some("16"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            assert_eq!(configured, 64);
+            assert_eq!(absolute_max, 16);
+            assert_eq!(effective, 16);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_ignores_non_positive_absolute_max() {
+        with_transition_worker_env(Some("4"), Some("0"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            assert_eq!(configured, 4);
+            assert_eq!(absolute_max, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            assert_eq!(effective, 4);
+        });
+
+        with_transition_worker_env(Some("4"), Some("-1"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            assert_eq!(configured, 4);
+            assert_eq!(absolute_max, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            assert_eq!(effective, 4);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_falls_back_for_zero_value() {
+        with_transition_worker_env(Some("0"), Some("32"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            let fallback = std::cmp::min(num_cpus::get() as i64, DEFAULT_TRANSITION_WORKERS_CAP);
+            assert_eq!(configured, fallback);
+            assert_eq!(absolute_max, 32);
+            assert_eq!(effective, fallback);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_queue_capacity_uses_default_when_env_missing() {
+        with_transition_queue_env(None, None, || {
+            assert_eq!(resolve_transition_queue_capacity(), DEFAULT_TRANSITION_QUEUE_CAPACITY);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_queue_capacity_honors_positive_env_value() {
+        with_transition_queue_env(Some("128"), None, || {
+            assert_eq!(resolve_transition_queue_capacity(), 128);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_queue_send_timeout_honors_positive_env_value() {
+        with_transition_queue_env(None, Some("250"), || {
+            assert_eq!(resolve_transition_queue_send_timeout(), StdDuration::from_millis(250));
+        });
+    }
+
+    #[tokio::test(flavor = "current_thread")]
+    async fn schedule_bucket_compensation_deduplicates_same_bucket() {
+        let state = TransitionState::new_with_capacity(1);
+
+        let first = state.schedule_bucket_compensation("bucket-a");
+        let second = state.schedule_bucket_compensation("bucket-a");
+
+        assert!(first);
+        assert!(!second);
+        assert_eq!(state.compensation_scheduled_tasks(), 1);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn transition_state_init_honors_runtime_configured_worker_count() {
+        let (_paths, ecstore) = setup_test_env().await;
+        let original_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        with_transition_worker_env_async(Some("3"), Some("8"), || async {
+            TransitionState::update_workers(ecstore.clone(), 0).await;
+            assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 3);
+        })
+        .await;
+
+        let current_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        if original_workers > 0 {
+            TransitionState::update_workers(ecstore, original_workers).await;
+        } else {
+            for _ in 0..current_workers {
+                let _ = GLOBAL_TransitionState.kill_tx.send(()).await;
+                GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
+            }
+        }
+    }
+
     #[test]
     fn should_defer_date_expiry_for_recent_config_update_respects_grace_window() {
         let now = OffsetDateTime::now_utc();

crates/ecstore/src/bucket/lifecycle/core.rs

@@ -34,7 +34,8 @@ const ERR_LIFECYCLE_NO_RULE: &str = "Lifecycle configuration should have at leas
 const ERR_LIFECYCLE_DUPLICATE_ID: &str = "Rule ID must be unique. Found same ID for more than one rule";
 const _ERR_XML_NOT_WELL_FORMED: &str =
     "The XML you provided was not well-formed or did not validate against our published schema";
-const ERR_LIFECYCLE_BUCKET_LOCKED: &str = "ExpiredObjectDeleteMarker is not allowed on a bucket with Object Lock enabled";
+const ERR_LIFECYCLE_BUCKET_LOCKED: &str =
+    "ExpiredObjectAllVersions element and DelMarkerExpiration action cannot be used on an object locked bucket";
 const ERR_LIFECYCLE_TOO_MANY_RULES: &str = "Lifecycle configuration should have at most 1000 rules";
 const ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS: &str = "Lifecycle expiration days must not be negative";
 const ERR_LIFECYCLE_INVALID_NONCURRENT_EXPIRATION_DAYS: &str = "Lifecycle noncurrent expiration days must not be negative";
@@ -320,14 +321,15 @@ impl Lifecycle for BucketLifecycleConfiguration {
             r.validate()?;
             if let Some(object_lock_enabled) = lr.object_lock_enabled.as_ref()
                 && object_lock_enabled.as_str() == ObjectLockEnabled::ENABLED
-                && let Some(expiration) = r.expiration.as_ref()
             {
-                // Object Lock + ExpiredObjectDeleteMarker conflict
-                if expiration.expired_object_delete_marker.is_some_and(|v| v) {
-                    return Err(std::io::Error::other(ERR_LIFECYCLE_BUCKET_LOCKED));
+                if let Some(expiration) = r.expiration.as_ref() {
+                    // Object Lock + ExpiredObjectAllVersions conflict (MinIO extension)
+                    if expiration.expired_object_all_versions.is_some_and(|v| v) {
+                        return Err(std::io::Error::other(ERR_LIFECYCLE_BUCKET_LOCKED));
+                    }
                 }
-                // Object Lock + ExpiredObjectAllVersions conflict (MinIO extension)
-                if expiration.expired_object_all_versions.is_some_and(|v| v) {
+                // Object Lock + DelMarkerExpiration conflict
+                if r.del_marker_expiration.is_some() {
                     return Err(std::io::Error::other(ERR_LIFECYCLE_BUCKET_LOCKED));
                 }
             }
@@ -2060,11 +2062,11 @@ mod tests {
         assert_eq!(event_after.due, Some(future_date));
     }
 
-    // --- TASK-002 tests: Object Lock + ExpiredObjectDeleteMarker conflict ---
+    // --- TASK-002 tests: Object Lock + ExpiredObjectDeleteMarker compatibility ---
 
     #[tokio::test]
     #[serial]
-    async fn validate_rejects_expired_object_delete_marker_on_locked_bucket() {
+    async fn validate_allows_expired_object_delete_marker_on_locked_bucket() {
         let lc = BucketLifecycleConfiguration {
             expiry_updated_at: None,
             rules: vec![LifecycleRule {
@@ -2089,8 +2091,9 @@ mod tests {
             ..Default::default()
         };
 
-        let err = lc.validate(&locked_config).await.unwrap_err();
-        assert_eq!(err.to_string(), ERR_LIFECYCLE_BUCKET_LOCKED);
+        lc.validate(&locked_config)
+            .await
+            .expect("expected validation to pass for ExpiredObjectDeleteMarker on locked bucket");
     }
 
     #[tokio::test]
@@ -2154,6 +2157,68 @@ mod tests {
             .expect("expected days-based expiration to pass on locked bucket");
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn validate_rejects_del_marker_expiration_on_locked_bucket() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: Some(s3s::dto::DelMarkerExpiration { days: Some(1) }),
+                filter: None,
+                id: Some("test-rule".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let locked_config = ObjectLockConfiguration {
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
+            ..Default::default()
+        };
+
+        let err = lc.validate(&locked_config).await.unwrap_err();
+        assert_eq!(err.to_string(), ERR_LIFECYCLE_BUCKET_LOCKED);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn validate_rejects_zero_day_del_marker_expiration_on_locked_bucket() {
+        let lc = BucketLifecycleConfiguration {
+            expiry_updated_at: None,
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
+                    days: Some(30),
+                    ..Default::default()
+                }),
+                abort_incomplete_multipart_upload: None,
+                del_marker_expiration: Some(s3s::dto::DelMarkerExpiration { days: Some(0) }),
+                filter: None,
+                id: Some("test-rule".to_string()),
+                noncurrent_version_expiration: None,
+                noncurrent_version_transitions: None,
+                prefix: None,
+                transitions: None,
+            }],
+        };
+
+        let locked_config = ObjectLockConfiguration {
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
+            ..Default::default()
+        };
+
+        let err = lc.validate(&locked_config).await.unwrap_err();
+        assert_eq!(err.to_string(), ERR_LIFECYCLE_BUCKET_LOCKED);
+    }
+
     // --- TASK-003 tests: Round up to next UTC processing boundary ---
 
     #[test]

crates/ecstore/src/bucket/lifecycle/tier_last_day_stats.rs

@@ -18,7 +18,7 @@
 #![allow(unused_must_use)]
 #![allow(clippy::all)]
 
-use rustfs_common::data_usage::TierStats;
+use rustfs_data_usage::TierStats;
 use sha2::Sha256;
 use std::collections::HashMap;
 use std::ops::Sub;

crates/ecstore/src/bucket/metadata.rs

@@ -724,7 +724,7 @@ impl BucketMetadata {
             return Err(Error::other("errServerNotInitialized"));
         };
 
-        self.parse_all_configs(store.clone())?;
+        self.parse_all_configs()?;
 
         let mut buf: Vec<u8> = vec![0; 4];
 
@@ -752,7 +752,7 @@ impl BucketMetadata {
         Ok(())
     }
 
-    fn parse_all_configs(&mut self, _api: Arc<ECStore>) -> Result<()> {
+    fn parse_all_configs(&mut self) -> Result<()> {
         if let Err(e) = self.parse_policy_config() {
             tracing::warn!(bucket = %self.name, config = "policy", error = %e, "parse_all_configs: failed to parse");
         }
@@ -881,11 +881,9 @@ pub async fn load_bucket_metadata_parse(api: Arc<ECStore>, bucket: &str, parse:
     bm.default_timestamps();
 
     if parse {
-        bm.parse_all_configs(api)?;
+        bm.parse_all_configs()?;
     }
 
-    // TODO: parse_all_configs
-
     Ok(bm)
 }
 
@@ -939,6 +937,23 @@ mod test {
         assert_eq!(bm.name, new.name);
     }
 
+    #[test]
+    fn parse_all_configs_parses_stored_configs_without_store_dependency() {
+        let mut bm = BucketMetadata::new("test-bucket");
+        bm.policy_config_json = br#"{"Version":"2012-10-17","Statement":[]}"#.to_vec();
+        bm.bucket_targets_config_json =
+            br#"{"targets":[{"endpoint":"s3.amazonaws.com","targetbucket":"target-bucket","arn":"arn:aws:s3:::target-bucket"}]}"#
+                .to_vec();
+
+        bm.parse_all_configs().unwrap();
+
+        assert!(bm.policy_config.is_some());
+        let bucket_targets = bm.bucket_target_config.unwrap();
+        assert_eq!(bucket_targets.targets.len(), 1);
+        assert_eq!(bucket_targets.targets[0].endpoint, "s3.amazonaws.com");
+        assert_eq!(bucket_targets.targets[0].target_bucket, "target-bucket");
+    }
+
     #[tokio::test]
     async fn marshal_msg_complete_example() {
         // Create a complete BucketMetadata with various configurations

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -56,7 +56,7 @@ use rustfs_filemeta::{
     ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision, VersionPurgeStatusType,
     get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
 };
-use rustfs_s3_common::EventName;
+use rustfs_s3_types::EventName;
 use rustfs_utils::http::{
     AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_TAGGING, AMZ_TAGGING_DIRECTIVE, CONTENT_ENCODING, HeaderExt as _,
     SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP,
@@ -929,7 +929,7 @@ impl ReplicationResyncer {
 }
 
 fn heal_should_use_check_replicate_delete(oi: &ObjectInfo) -> bool {
-    oi.delete_marker || (!oi.replication_status.is_empty() && oi.replication_status != ReplicationStatusType::Failed)
+    oi.delete_marker || !oi.version_purge_status.is_empty()
 }
 
 pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationConfig) -> ReplicateObjectInfo {
@@ -939,8 +939,8 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
     if let Some(rc) = rcfg.config.as_ref()
         && !rc.role.is_empty()
     {
-        if !oi.replication_status.is_empty() {
-            oi.replication_status_internal = Some(format!("{}={};", rc.role, oi.replication_status.as_str()));
+        if !oi.version_purge_status.is_empty() {
+            oi.version_purge_status_internal = Some(format!("{}={};", rc.role, oi.version_purge_status.as_str()));
         }
 
         if !oi.replication_status.is_empty() {
@@ -3742,7 +3742,7 @@ mod tests {
     }
 
     #[test]
-    fn test_heal_should_use_check_replicate_delete_pending_uses_delete_path() {
+    fn test_heal_should_use_check_replicate_delete_pending_non_delete_marker_uses_must_replicate_path() {
         let oi = ObjectInfo {
             bucket: "b".to_string(),
             name: "obj".to_string(),
@@ -3751,8 +3751,8 @@ mod tests {
             ..Default::default()
         };
         assert!(
-            heal_should_use_check_replicate_delete(&oi),
-            "Pending (non-Failed) status with non-empty replication uses check_replicate_delete path"
+            !heal_should_use_check_replicate_delete(&oi),
+            "Pending non-delete-marker object must use must_replicate path to evaluate current target set"
         );
     }
 
@@ -3771,6 +3771,36 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_heal_should_use_check_replicate_delete_version_purge_status_uses_delete_path() {
+        let oi = ObjectInfo {
+            bucket: "b".to_string(),
+            name: "obj".to_string(),
+            delete_marker: false,
+            version_purge_status: VersionPurgeStatusType::Pending,
+            ..Default::default()
+        };
+        assert!(
+            heal_should_use_check_replicate_delete(&oi),
+            "Version purge entries must use check_replicate_delete path"
+        );
+    }
+
+    #[test]
+    fn test_heal_should_use_check_replicate_delete_completed_non_delete_marker_uses_must_replicate_path() {
+        let oi = ObjectInfo {
+            bucket: "b".to_string(),
+            name: "obj".to_string(),
+            delete_marker: false,
+            replication_status: ReplicationStatusType::Completed,
+            ..Default::default()
+        };
+        assert!(
+            !heal_should_use_check_replicate_delete(&oi),
+            "Completed non-delete-marker object must use must_replicate path so new targets can be evaluated"
+        );
+    }
+
     #[test]
     fn test_delete_replication_object_opts_marks_replica_deletes() {
         let dobj = ObjectToDelete {
@@ -4051,6 +4081,32 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn test_get_heal_replicate_object_info_maps_version_purge_status_for_role() {
+        let role = "arn:rustfs:replication::target:bucket";
+        let oi = ObjectInfo {
+            bucket: "test-bucket".to_string(),
+            name: "key".to_string(),
+            delete_marker: false,
+            version_purge_status: VersionPurgeStatusType::Pending,
+            version_id: Some(Uuid::nil()),
+            mod_time: Some(OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+        let rcfg = ReplicationConfig::new(
+            Some(ReplicationConfiguration {
+                role: role.to_string(),
+                rules: vec![],
+            }),
+            None,
+        );
+        let roi = get_heal_replicate_object_info(&oi, &rcfg).await;
+
+        assert_eq!(roi.replication_status_internal, None);
+        assert_eq!(roi.version_purge_status_internal.as_deref(), Some(format!("{role}=PENDING;").as_str()));
+        assert_eq!(roi.target_purge_statuses.get(role), Some(&VersionPurgeStatusType::Pending));
+    }
+
     #[tokio::test]
     async fn test_cancel_marks_only_matching_bucket_target_token() {
         let resyncer = ReplicationResyncer::new().await;

crates/ecstore/src/cache_value/metacache_set.rs

@@ -19,9 +19,10 @@ use futures::future::join_all;
 use metrics::counter;
 use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
 use std::{
+    collections::VecDeque,
     future::Future,
     pin::Pin,
-    sync::{Arc, Mutex},
+    sync::{Arc, OnceLock},
     time::Duration,
 };
 use tokio::io::AsyncRead;
@@ -112,9 +113,9 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
 
     let mut jobs: Vec<tokio::task::JoinHandle<std::result::Result<(), DiskError>>> = Vec::new();
     let mut readers = Vec::with_capacity(opts.disks.len());
-    let fds = opts.fallback_disks.iter().flatten().cloned().collect::<Vec<_>>();
+    let fds = opts.fallback_disks.iter().flatten().cloned().collect::<VecDeque<_>>();
     let max_disk_failures = opts.disks.len().saturating_sub(opts.min_disks);
-    let producer_errs = Arc::new(Mutex::new(vec![None; opts.disks.len()]));
+    let producer_errs: Arc<[OnceLock<DiskError>]> = (0..opts.disks.len()).map(|_| OnceLock::new()).collect::<Vec<_>>().into();
 
     let cancel_rx = CancellationToken::new();
 
@@ -137,14 +138,14 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                         return Ok(());
                     }
                     TestReaderBehavior::ProducerError(err) => {
-                        producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                        record_producer_error(&producer_errs_clone, disk_idx, &err);
                         return Err(err);
                     }
                     TestReaderBehavior::PartialThenTimeout(entries) => {
                         let mut wr = wr;
                         let mut out = rustfs_filemeta::MetacacheWriter::new(&mut wr);
                         let err = DiskError::Timeout;
-                        producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                        record_producer_error(&producer_errs_clone, disk_idx, &err);
                         let _ = out.write(&entries).await;
                         drop(out);
                         return Err(err);
@@ -187,8 +188,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
 
             while need_fallback {
                 let mut disk_op = None;
-                while !fds_clone.is_empty() {
-                    let disk = fds_clone.remove(0);
+                while let Some(disk) = fds_clone.pop_front() {
                     if disk.is_online().await {
                         disk_op = Some(disk);
                         break;
@@ -198,7 +198,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 let Some(disk) = disk_op else {
                     warn!("list_path_raw: fallback disk is none");
                     let err = last_err.unwrap_or(DiskError::DiskNotFound);
-                    producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                    record_producer_error(&producer_errs_clone, disk_idx, &err);
                     return Err(err);
                 };
 
@@ -281,7 +281,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                             // info!("read entry disk: {}, name: {}", i, entry.name);
                             entry
                         } else {
-                            if let Some(err) = producer_errs.lock().expect("producer error mutex poisoned")[i].clone() {
+                            if let Some(err) = producer_error(&producer_errs, i) {
                                 has_err += 1;
                                 errs[i] = Some(err);
                                 continue;
@@ -293,7 +293,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                         }
                     }
                     PeekOutcome::Error(err) => {
-                        if let Some(err) = producer_errs.lock().expect("producer error mutex poisoned")[i].clone() {
+                        if let Some(err) = producer_error(&producer_errs, i) {
                             has_err += 1;
                             errs[i] = Some(err);
                             continue;
@@ -510,10 +510,21 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
     Ok(())
 }
 
+#[inline]
+fn record_producer_error(producer_errs: &[OnceLock<DiskError>], idx: usize, err: &DiskError) {
+    let _ = producer_errs[idx].set(err.clone());
+}
+
+#[inline]
+fn producer_error(producer_errs: &[OnceLock<DiskError>], idx: usize) -> Option<DiskError> {
+    producer_errs[idx].get().cloned()
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
     use rustfs_filemeta::MetacacheWriter;
+    use std::sync::Mutex;
 
     #[tokio::test]
     async fn list_path_raw_empty_disks_returns_read_quorum() {

crates/ecstore/src/client/transition_api.rs

@@ -51,6 +51,7 @@ use md5::Md5;
 use rand::{Rng, RngExt};
 use rustfs_config::MAX_S3_CLIENT_RESPONSE_SIZE;
 use rustfs_rio::HashReader;
+use rustfs_tls_runtime::{load_global_outbound_tls_state, record_tls_generation};
 use rustfs_utils::HashAlgorithm;
 use rustfs_utils::{
     net::get_endpoint_url,
@@ -58,6 +59,8 @@ use rustfs_utils::{
         DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, MAX_RETRY, RetryTimer, is_http_status_retryable, is_s3code_retryable,
     },
 };
+use rustls_pki_types::PrivateKeyDer;
+use rustls_pki_types::pem::PemObject;
 use s3s::S3ErrorCode;
 use s3s::dto::Owner;
 use s3s::dto::ReplicationStatus;
@@ -152,24 +155,11 @@ pub enum BucketLookupType {
     BucketLookupPath,
 }
 
-fn load_root_store_from_tls_path() -> Option<rustls::RootCertStore> {
-    // Load the root certificate bundle from the path specified by the
-    // RUSTFS_TLS_PATH environment variable.
-    let tp = rustfs_utils::get_env_str(rustfs_config::ENV_RUSTFS_TLS_PATH, rustfs_config::DEFAULT_RUSTFS_TLS_PATH);
-    // If no TLS path is configured, do not fall back to a CA bundle in the current directory.
-    if tp.is_empty() {
-        return None;
-    }
-    let ca = std::path::Path::new(&tp).join(rustfs_config::RUSTFS_CA_CERT);
-    if !ca.exists() {
-        return None;
-    }
-
-    let der_list = rustfs_utils::load_cert_bundle_der_bytes(ca.to_str().unwrap_or_default()).ok()?;
+fn build_root_store_from_der_list(der_list: Vec<Vec<u8>>) -> Option<rustls::RootCertStore> {
     let mut store = rustls::RootCertStore::empty();
     for der in der_list {
         if let Err(e) = store.add(der.into()) {
-            warn!("Warning: failed to add certificate from '{}' to root store: {e}", ca.display());
+            warn!("Warning: failed to add certificate to root store: {e}");
         }
     }
     Some(store)
@@ -199,18 +189,38 @@ where
     })
 }
 
-fn build_tls_config() -> Result<rustls::ClientConfig, std::io::Error> {
-    with_rustls_init_guard(|| {
-        let config = if let Some(store) = load_root_store_from_tls_path() {
-            rustls::ClientConfig::builder()
-                .with_root_certificates(store)
-                .with_no_client_auth()
-        } else {
-            rustls::ClientConfig::builder().with_native_roots()?.with_no_client_auth()
-        };
+async fn build_tls_config() -> Result<rustls::ClientConfig, std::io::Error> {
+    with_rustls_init_guard(|| Ok(()))?;
 
-        Ok(config)
-    })
+    let outbound_tls = load_global_outbound_tls_state().await;
+    record_tls_generation("ecstore_transition_client", outbound_tls.generation.0);
+    let builder = if let Some(root_ca_pem) = outbound_tls.root_ca_pem.as_ref() {
+        let mut reader = std::io::BufReader::new(root_ca_pem.as_slice());
+        let certs_der = rustls_pki_types::CertificateDer::pem_reader_iter(&mut reader)
+            .collect::<Result<Vec<_>, _>>()
+            .map_err(|e| std::io::Error::other(format!("failed to parse published root CA PEM: {e}")))?;
+
+        let root_store = build_root_store_from_der_list(certs_der.into_iter().map(|cert| cert.to_vec()).collect::<Vec<_>>())
+            .ok_or_else(|| std::io::Error::other("published outbound root CA material could not build root store"))?;
+        rustls::ClientConfig::builder().with_root_certificates(root_store)
+    } else {
+        rustls::ClientConfig::builder().with_native_roots()?
+    };
+
+    let config = if let Some(identity) = outbound_tls.mtls_identity.as_ref() {
+        let certs = rustls_pki_types::CertificateDer::pem_reader_iter(&mut std::io::BufReader::new(identity.cert_pem.as_slice()))
+            .collect::<Result<Vec<_>, _>>()
+            .map_err(|e| std::io::Error::other(format!("failed to parse published client cert PEM: {e}")))?;
+        let key = PrivateKeyDer::from_pem_reader(&mut std::io::BufReader::new(identity.key_pem.as_slice()))
+            .map_err(|e| std::io::Error::other(format!("failed to parse published client key PEM: {e}")))?;
+        builder
+            .with_client_auth_cert(certs, key)
+            .map_err(|e| std::io::Error::other(format!("failed to build client mTLS identity: {e}")))?
+    } else {
+        builder.with_no_client_auth()
+    };
+
+    Ok(config)
 }
 
 impl TransitionClient {
@@ -234,7 +244,7 @@ impl TransitionClient {
 
         let endpoint_url = get_endpoint_url(endpoint, opts.secure)?;
 
-        let tls = build_tls_config()?;
+        let tls = build_tls_config().await?;
 
         let https = hyper_rustls::HttpsConnectorBuilder::new()
             .with_tls_config(tls)
@@ -1374,9 +1384,7 @@ pub struct CreateBucketConfiguration {
 
 #[cfg(test)]
 mod tests {
-    use super::{
-        build_tls_config, load_root_store_from_tls_path, signer_error_to_io_error, validate_header_values, with_rustls_init_guard,
-    };
+    use super::{build_tls_config, signer_error_to_io_error, validate_header_values, with_rustls_init_guard};
     use http::{HeaderMap, HeaderValue};
 
     #[test]
@@ -1395,22 +1403,6 @@ mod tests {
         assert!(outcome.is_ok(), "TLS config creation should not panic");
     }
 
-    /// When RUSTFS_TLS_PATH is not set, `load_root_store_from_tls_path` must return `None`
-    /// (i.e. it must not silently look for a CA bundle in the current working directory).
-    #[test]
-    fn tls_path_unset_returns_none() {
-        let result = temp_env::with_var_unset(rustfs_config::ENV_RUSTFS_TLS_PATH, || load_root_store_from_tls_path());
-        assert!(result.is_none(), "expected None when RUSTFS_TLS_PATH is unset, but got a root store");
-    }
-
-    /// When RUSTFS_TLS_PATH is set to an empty string, `load_root_store_from_tls_path` must
-    /// return `None` to avoid accidentally trusting a CA bundle in the current directory.
-    #[test]
-    fn tls_path_empty_returns_none() {
-        let result = temp_env::with_var(rustfs_config::ENV_RUSTFS_TLS_PATH, Some(""), || load_root_store_from_tls_path());
-        assert!(result.is_none(), "expected None when RUSTFS_TLS_PATH is empty, but got a root store");
-    }
-
     /// Installing the rustls crypto provider when one is already set must not panic or return
     /// an error that surfaces to callers (the race-safe `get_default` check guards the install).
     #[test]

crates/ecstore/src/data_usage.rs

@@ -27,7 +27,7 @@ pub use local_snapshot::{
     data_usage_dir, data_usage_state_dir, ensure_data_usage_layout, read_snapshot as read_local_snapshot, snapshot_file_name,
     snapshot_object_path, snapshot_path, write_snapshot as write_local_snapshot,
 };
-use rustfs_common::data_usage::{
+use rustfs_data_usage::{
     BucketTargetUsageInfo, BucketUsageInfo, DataUsageCache, DataUsageEntry, DataUsageInfo, DiskUsageStatus, SizeSummary,
 };
 use rustfs_io_metrics::record_system_path_failure;
@@ -48,7 +48,14 @@ const DATA_USAGE_BLOOM_NAME: &str = ".bloomcycle.bin";
 pub const DATA_USAGE_CACHE_NAME: &str = ".usage-cache.bin";
 const DATA_USAGE_CACHE_TTL_SECS: u64 = 30;
 
-type UsageMemoryCache = Arc<RwLock<HashMap<String, (u64, SystemTime)>>>;
+#[derive(Debug, Clone)]
+struct CachedBucketUsage {
+    usage: BucketUsageInfo,
+    refreshed_at: SystemTime,
+    usage_updated_at: SystemTime,
+}
+
+type UsageMemoryCache = Arc<RwLock<HashMap<String, CachedBucketUsage>>>;
 type CacheUpdating = Arc<RwLock<bool>>;
 
 static USAGE_MEMORY_CACHE: OnceLock<UsageMemoryCache> = OnceLock::new();
@@ -403,21 +410,90 @@ pub async fn compute_bucket_usage(store: Arc<ECStore>, bucket_name: &str) -> Res
     Ok(usage)
 }
 
-/// Fast in-memory increment for immediate quota consistency
-pub async fn increment_bucket_usage_memory(bucket: &str, size_increment: u64) {
+async fn ensure_bucket_usage_cached(bucket: &str) {
+    let cache = memory_cache().read().await;
+    if cache.contains_key(bucket) {
+        return;
+    }
+    drop(cache);
+
+    update_usage_cache_if_needed().await;
+}
+
+fn cached_bucket_usage_from_backend(usage: BucketUsageInfo, updated_at: SystemTime) -> CachedBucketUsage {
+    CachedBucketUsage {
+        usage,
+        refreshed_at: SystemTime::now(),
+        usage_updated_at: updated_at,
+    }
+}
+
+fn cached_bucket_usage_now(usage: BucketUsageInfo) -> CachedBucketUsage {
+    let now = SystemTime::now();
+    CachedBucketUsage {
+        usage,
+        refreshed_at: now,
+        usage_updated_at: now,
+    }
+}
+
+fn data_usage_info_updated_at(data_usage_info: &DataUsageInfo) -> SystemTime {
+    data_usage_info.last_update.unwrap_or(SystemTime::UNIX_EPOCH)
+}
+
+/// Fast in-memory update for immediate quota and admin usage consistency.
+pub async fn record_bucket_object_write_memory(bucket: &str, previous_current_size: Option<u64>, new_size: u64) {
+    ensure_bucket_usage_cached(bucket).await;
+
     let mut cache = memory_cache().write().await;
-    let current = cache.entry(bucket.to_string()).or_insert_with(|| (0, SystemTime::now()));
-    current.0 += size_increment;
-    current.1 = SystemTime::now();
+    let entry = cache
+        .entry(bucket.to_string())
+        .or_insert_with(|| cached_bucket_usage_now(BucketUsageInfo::default()));
+
+    match previous_current_size {
+        Some(previous_size) => {
+            entry.usage.size = entry.usage.size.saturating_sub(previous_size).saturating_add(new_size);
+        }
+        None => {
+            entry.usage.size = entry.usage.size.saturating_add(new_size);
+            entry.usage.objects_count = entry.usage.objects_count.saturating_add(1);
+            entry.usage.versions_count = entry.usage.versions_count.saturating_add(1);
+        }
+    }
+
+    let now = SystemTime::now();
+    entry.refreshed_at = now;
+    entry.usage_updated_at = now;
+}
+
+/// Fast in-memory increment for immediate quota consistency.
+pub async fn increment_bucket_usage_memory(bucket: &str, size_increment: u64) {
+    record_bucket_object_write_memory(bucket, None, size_increment).await;
+}
+
+/// Fast in-memory update for successful object deletes.
+pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) {
+    ensure_bucket_usage_cached(bucket).await;
+
+    let mut cache = memory_cache().write().await;
+    let entry = cache
+        .entry(bucket.to_string())
+        .or_insert_with(|| cached_bucket_usage_now(BucketUsageInfo::default()));
+
+    entry.usage.size = entry.usage.size.saturating_sub(deleted_size);
+    if removed_current_object {
+        entry.usage.objects_count = entry.usage.objects_count.saturating_sub(1);
+        entry.usage.versions_count = entry.usage.versions_count.saturating_sub(1);
+    }
+
+    let now = SystemTime::now();
+    entry.refreshed_at = now;
+    entry.usage_updated_at = now;
 }
 
 /// Fast in-memory decrement for immediate quota consistency
 pub async fn decrement_bucket_usage_memory(bucket: &str, size_decrement: u64) {
-    let mut cache = memory_cache().write().await;
-    if let Some(current) = cache.get_mut(bucket) {
-        current.0 = current.0.saturating_sub(size_decrement);
-        current.1 = SystemTime::now();
-    }
+    record_bucket_object_delete_memory(bucket, size_decrement, size_decrement > 0).await;
 }
 
 /// Get bucket usage from in-memory cache
@@ -425,7 +501,7 @@ pub async fn get_bucket_usage_memory(bucket: &str) -> Option<u64> {
     update_usage_cache_if_needed().await;
 
     let cache = memory_cache().read().await;
-    cache.get(bucket).map(|(usage, _)| *usage)
+    cache.get(bucket).map(|cached| cached.usage.size)
 }
 
 async fn update_usage_cache_if_needed() {
@@ -434,7 +510,7 @@ async fn update_usage_cache_if_needed() {
     let now = SystemTime::now();
 
     let cache = memory_cache().read().await;
-    let earliest_timestamp = cache.values().map(|(_, ts)| *ts).min();
+    let earliest_timestamp = cache.values().map(|cached| cached.refreshed_at).min();
     drop(cache);
 
     let age = match earliest_timestamp {
@@ -461,8 +537,12 @@ async fn update_usage_cache_if_needed() {
                 && let Ok(data_usage_info) = load_data_usage_from_backend(store.clone()).await
             {
                 let mut cache = cache_clone.write().await;
+                let usage_updated_at = data_usage_info_updated_at(&data_usage_info);
                 for (bucket_name, bucket_usage) in data_usage_info.buckets_usage.iter() {
-                    cache.insert(bucket_name.clone(), (bucket_usage.size, SystemTime::now()));
+                    cache.insert(
+                        bucket_name.clone(),
+                        cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at),
+                    );
                 }
             }
             let mut updating = updating_clone.write().await;
@@ -488,8 +568,12 @@ async fn update_usage_cache_if_needed() {
         && let Ok(data_usage_info) = load_data_usage_from_backend(store.clone()).await
     {
         let mut cache = memory_cache().write().await;
+        let usage_updated_at = data_usage_info_updated_at(&data_usage_info);
         for (bucket_name, bucket_usage) in data_usage_info.buckets_usage.iter() {
-            cache.insert(bucket_name.clone(), (bucket_usage.size, SystemTime::now()));
+            cache.insert(
+                bucket_name.clone(),
+                cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at),
+            );
         }
     }
 
@@ -497,15 +581,62 @@ async fn update_usage_cache_if_needed() {
     *updating = false;
 }
 
+pub async fn replace_bucket_usage_memory_from_info(data_usage_info: &DataUsageInfo) {
+    let usage_updated_at = data_usage_info_updated_at(data_usage_info);
+    let mut cache = memory_cache().write().await;
+    let mut next_cache = HashMap::new();
+
+    for (bucket, bucket_usage) in data_usage_info.buckets_usage.iter() {
+        if let Some(existing) = cache.get(bucket)
+            && existing.usage_updated_at > usage_updated_at
+        {
+            next_cache.insert(bucket.clone(), existing.clone());
+            continue;
+        }
+
+        next_cache.insert(bucket.clone(), cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at));
+    }
+
+    for (bucket, existing) in cache.iter() {
+        if !data_usage_info.buckets_usage.contains_key(bucket) && existing.usage_updated_at > usage_updated_at {
+            next_cache.insert(bucket.clone(), existing.clone());
+        }
+    }
+
+    *cache = next_cache;
+}
+
+pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageInfo) {
+    let cache = memory_cache().read().await;
+    if cache.is_empty() {
+        return;
+    }
+
+    let persisted_update = data_usage_info.last_update;
+    let mut changed = false;
+
+    for (bucket, cached) in cache.iter() {
+        if persisted_update.is_some_and(|persisted| cached.usage_updated_at <= persisted) {
+            continue;
+        }
+
+        data_usage_info.buckets_usage.insert(bucket.clone(), cached.usage.clone());
+        data_usage_info.bucket_sizes.insert(bucket.clone(), cached.usage.size);
+        changed = true;
+    }
+
+    if changed {
+        data_usage_info.buckets_count = data_usage_info.buckets_usage.len() as u64;
+        data_usage_info.calculate_totals();
+    }
+}
+
 /// Sync memory cache with backend data (called by scanner)
 pub async fn sync_memory_cache_with_backend() -> Result<(), Error> {
     if let Some(store) = crate::global::GLOBAL_OBJECT_API.get() {
         match load_data_usage_from_backend(store.clone()).await {
             Ok(data_usage_info) => {
-                let mut cache = memory_cache().write().await;
-                for (bucket, bucket_usage) in data_usage_info.buckets_usage.iter() {
-                    cache.insert(bucket.clone(), (bucket_usage.size, SystemTime::now()));
-                }
+                replace_bucket_usage_memory_from_info(&data_usage_info).await;
             }
             Err(e) => {
                 debug!("Failed to sync memory cache with backend: {}", e);
@@ -686,7 +817,33 @@ pub async fn save_data_usage_cache(cache: &DataUsageCache, name: &str) -> crate:
 #[cfg(test)]
 mod tests {
     use super::*;
-    use rustfs_common::data_usage::BucketUsageInfo;
+    use rustfs_data_usage::BucketUsageInfo;
+    use serial_test::serial;
+
+    async fn clear_usage_memory_cache_for_test() {
+        memory_cache().write().await.clear();
+        *cache_updating().write().await = false;
+    }
+
+    fn data_usage_info_for_test(bucket: &str, objects_count: u64, size: u64, last_update: SystemTime) -> DataUsageInfo {
+        let mut info = DataUsageInfo {
+            last_update: Some(last_update),
+            ..Default::default()
+        };
+        info.buckets_usage.insert(
+            bucket.to_string(),
+            BucketUsageInfo {
+                objects_count,
+                versions_count: objects_count,
+                size,
+                ..Default::default()
+            },
+        );
+        info.bucket_sizes.insert(bucket.to_string(), size);
+        info.buckets_count = info.buckets_usage.len() as u64;
+        info.calculate_totals();
+        info
+    }
 
     fn aggregate_for_test(
         inputs: Vec<(DiskUsageStatus, Result<Option<LocalUsageSnapshot>, Error>)>,
@@ -772,4 +929,101 @@ mod tests {
         assert_eq!(aggregated.buckets_count, 1);
         assert_eq!(aggregated.buckets_usage.get("bucket-a").map(|b| (b.objects_count, b.size)), Some((3, 42)));
     }
+
+    #[tokio::test]
+    #[serial]
+    async fn memory_overlay_reflects_recent_delete_before_scanner_persists() {
+        clear_usage_memory_cache_for_test().await;
+
+        let persisted = data_usage_info_for_test("bucket-a", 1, 42, SystemTime::now() - Duration::from_secs(10));
+        replace_bucket_usage_memory_from_info(&persisted).await;
+        record_bucket_object_delete_memory("bucket-a", 42, true).await;
+
+        let mut response = persisted.clone();
+        apply_bucket_usage_memory_overlay(&mut response).await;
+
+        assert_eq!(response.objects_total_count, 0);
+        assert_eq!(response.objects_total_size, 0);
+        assert_eq!(
+            response
+                .buckets_usage
+                .get("bucket-a")
+                .map(|usage| (usage.objects_count, usage.size)),
+            Some((0, 0))
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn memory_overlay_preserves_object_count_for_overwrite() {
+        clear_usage_memory_cache_for_test().await;
+
+        let persisted = data_usage_info_for_test("bucket-a", 1, 10, SystemTime::now() - Duration::from_secs(10));
+        replace_bucket_usage_memory_from_info(&persisted).await;
+        record_bucket_object_write_memory("bucket-a", Some(10), 20).await;
+
+        let mut response = persisted.clone();
+        apply_bucket_usage_memory_overlay(&mut response).await;
+
+        assert_eq!(response.objects_total_count, 1);
+        assert_eq!(response.objects_total_size, 20);
+        assert_eq!(
+            response
+                .buckets_usage
+                .get("bucket-a")
+                .map(|usage| (usage.objects_count, usage.size)),
+            Some((1, 20))
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn memory_overlay_does_not_replace_newer_persisted_usage() {
+        clear_usage_memory_cache_for_test().await;
+
+        let now = SystemTime::now();
+        let old_persisted = data_usage_info_for_test("bucket-a", 1, 42, now - Duration::from_secs(10));
+        replace_bucket_usage_memory_from_info(&old_persisted).await;
+        record_bucket_object_delete_memory("bucket-a", 42, true).await;
+
+        let mut newer_persisted = data_usage_info_for_test("bucket-a", 2, 84, now + Duration::from_secs(10));
+        apply_bucket_usage_memory_overlay(&mut newer_persisted).await;
+
+        assert_eq!(newer_persisted.objects_total_count, 2);
+        assert_eq!(newer_persisted.objects_total_size, 84);
+        assert_eq!(
+            newer_persisted
+                .buckets_usage
+                .get("bucket-a")
+                .map(|usage| (usage.objects_count, usage.size)),
+            Some((2, 84))
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn scanner_sync_preserves_newer_memory_update() {
+        clear_usage_memory_cache_for_test().await;
+
+        let now = SystemTime::now();
+        let old_persisted = data_usage_info_for_test("bucket-a", 1, 42, now - Duration::from_secs(10));
+        replace_bucket_usage_memory_from_info(&old_persisted).await;
+        record_bucket_object_delete_memory("bucket-a", 42, true).await;
+
+        let scanner_snapshot = data_usage_info_for_test("bucket-a", 1, 42, now - Duration::from_secs(5));
+        replace_bucket_usage_memory_from_info(&scanner_snapshot).await;
+
+        let mut response = scanner_snapshot.clone();
+        apply_bucket_usage_memory_overlay(&mut response).await;
+
+        assert_eq!(response.objects_total_count, 0);
+        assert_eq!(response.objects_total_size, 0);
+        assert_eq!(
+            response
+                .buckets_usage
+                .get("bucket-a")
+                .map(|usage| (usage.objects_count, usage.size)),
+            Some((0, 0))
+        );
+    }
 }

crates/ecstore/src/disk/disk_store.rs

@@ -27,6 +27,8 @@ use crate::global::GLOBAL_LOCAL_DISK_ID_MAP;
 use bytes::Bytes;
 use metrics::counter;
 use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
+#[cfg(not(test))]
+use std::sync::OnceLock;
 use std::{
     path::PathBuf,
     sync::{
@@ -44,10 +46,60 @@ use uuid::Uuid;
 const DISK_HEALTH_OK: u32 = 0;
 const DISK_HEALTH_FAULTY: u32 = 1;
 
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum TimeoutHealthAction {
+    MarkFailure,
+    IgnoreFailure,
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum TimeoutHealthPolicy {
+    MarkFailure,
+    IgnoreScanner,
+}
+
+impl TimeoutHealthPolicy {
+    fn parse(raw: &str) -> Option<Self> {
+        match raw.trim().to_ascii_lowercase().as_str() {
+            rustfs_config::DRIVE_TIMEOUT_HEALTH_ACTION_MARK_FAILURE => Some(Self::MarkFailure),
+            rustfs_config::DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER => Some(Self::IgnoreScanner),
+            _ => None,
+        }
+    }
+
+    fn scanner_timeout_health_action(self) -> TimeoutHealthAction {
+        match self {
+            Self::MarkFailure => TimeoutHealthAction::MarkFailure,
+            Self::IgnoreScanner => TimeoutHealthAction::IgnoreFailure,
+        }
+    }
+}
+
 pub const ENV_RUSTFS_DRIVE_ACTIVE_MONITORING: &str = "RUSTFS_DRIVE_ACTIVE_MONITORING";
 pub const DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING: bool = true;
 pub const SKIP_IF_SUCCESS_BEFORE: Duration = Duration::from_secs(5);
 
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum DriveTimeoutProfile {
+    Default,
+    HighLatency,
+}
+
+impl DriveTimeoutProfile {
+    fn parse(raw: &str) -> Option<Self> {
+        match raw.trim().to_ascii_lowercase().as_str() {
+            rustfs_config::DRIVE_TIMEOUT_PROFILE_DEFAULT => Some(Self::Default),
+            rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY => Some(Self::HighLatency),
+            _ => None,
+        }
+    }
+}
+
+#[cfg(not(test))]
+static DRIVE_TIMEOUT_PROFILE_CACHE: OnceLock<DriveTimeoutProfile> = OnceLock::new();
+#[cfg(not(test))]
+static DRIVE_TIMEOUT_HEALTH_POLICY_CACHE: OnceLock<TimeoutHealthPolicy> = OnceLock::new();
+
 lazy_static::lazy_static! {
     static ref TEST_DATA: Bytes = Bytes::from(vec![42u8; 2048]);
     static ref TEST_BUCKET: String = ".rustfs.sys/tmp".to_string();
@@ -60,10 +112,39 @@ pub fn get_max_timeout_duration() -> Duration {
     ))
 }
 
-fn get_drive_timeout_duration(env_key: &str, default_secs: u64) -> Duration {
+fn resolve_drive_timeout_profile_from_env() -> DriveTimeoutProfile {
+    let raw = rustfs_utils::get_env_str(rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE, rustfs_config::DEFAULT_DRIVE_TIMEOUT_PROFILE);
+    if let Some(profile) = DriveTimeoutProfile::parse(&raw) {
+        return profile;
+    }
+    warn!(
+        env = rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE,
+        value = %raw,
+        default = rustfs_config::DEFAULT_DRIVE_TIMEOUT_PROFILE,
+        "Invalid drive timeout profile; falling back to default"
+    );
+    DriveTimeoutProfile::parse(rustfs_config::DEFAULT_DRIVE_TIMEOUT_PROFILE).unwrap_or(DriveTimeoutProfile::Default)
+}
+
+fn get_drive_timeout_profile() -> DriveTimeoutProfile {
+    #[cfg(test)]
+    {
+        resolve_drive_timeout_profile_from_env()
+    }
+    #[cfg(not(test))]
+    {
+        *DRIVE_TIMEOUT_PROFILE_CACHE.get_or_init(resolve_drive_timeout_profile_from_env)
+    }
+}
+
+fn get_drive_timeout_duration(env_key: &str, default_secs: u64, high_latency_secs: Option<u64>) -> Duration {
+    let fallback_default = match (get_drive_timeout_profile(), high_latency_secs) {
+        (DriveTimeoutProfile::HighLatency, Some(secs)) => secs,
+        _ => default_secs,
+    };
     Duration::from_secs(
         rustfs_utils::get_env_opt_u64_with_aliases(env_key, &[rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION])
-            .unwrap_or(default_secs),
+            .unwrap_or(fallback_default),
     )
 }
 
@@ -71,6 +152,7 @@ pub fn get_drive_metadata_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -78,6 +160,7 @@ pub fn get_drive_disk_info_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_DISK_INFO_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_DISK_INFO_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -85,6 +168,7 @@ pub fn get_drive_list_dir_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_LIST_DIR_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_LIST_DIR_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -92,6 +176,7 @@ pub fn get_drive_walkdir_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_WALKDIR_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -99,6 +184,7 @@ pub fn get_drive_walkdir_stall_timeout() -> Duration {
     get_drive_timeout_duration(
         rustfs_config::ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS,
         rustfs_config::DEFAULT_DRIVE_WALKDIR_STALL_TIMEOUT_SECS,
+        Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS),
     )
 }
 
@@ -116,6 +202,34 @@ pub fn get_drive_active_check_timeout() -> Duration {
     ))
 }
 
+fn resolve_drive_timeout_health_policy_from_env() -> TimeoutHealthPolicy {
+    let raw = rustfs_utils::get_env_str(
+        rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION,
+        rustfs_config::DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION,
+    );
+    if let Some(policy) = TimeoutHealthPolicy::parse(&raw) {
+        return policy;
+    }
+    warn!(
+        env = rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION,
+        value = %raw,
+        default = rustfs_config::DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION,
+        "Invalid drive timeout health action policy; falling back to default"
+    );
+    TimeoutHealthPolicy::parse(rustfs_config::DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION).unwrap_or(TimeoutHealthPolicy::MarkFailure)
+}
+
+fn get_drive_timeout_health_policy() -> TimeoutHealthPolicy {
+    #[cfg(test)]
+    {
+        resolve_drive_timeout_health_policy_from_env()
+    }
+    #[cfg(not(test))]
+    {
+        *DRIVE_TIMEOUT_HEALTH_POLICY_CACHE.get_or_init(resolve_drive_timeout_health_policy_from_env)
+    }
+}
+
 /// DiskHealthTracker tracks the health status of a disk.
 /// Similar to Go's diskHealthTracker.
 #[derive(Debug)]
@@ -464,6 +578,8 @@ pub struct LocalDiskWrapper {
     cancel_token: CancellationToken,
     /// Disk ID for stale checking
     disk_id: Arc<RwLock<Option<Uuid>>>,
+    /// Timeout policy for scanner-sensitive operations, loaded once on wrapper initialization.
+    timeout_health_policy: TimeoutHealthPolicy,
 }
 
 impl LocalDiskWrapper {
@@ -480,6 +596,7 @@ impl LocalDiskWrapper {
             health_check: health_check && env_health_check,
             cancel_token: CancellationToken::new(),
             disk_id: Arc::new(RwLock::new(None)),
+            timeout_health_policy: get_drive_timeout_health_policy(),
         };
         record_drive_runtime_state(&wrapper.disk.endpoint(), RuntimeDriveHealthState::Online);
         wrapper
@@ -505,6 +622,10 @@ impl LocalDiskWrapper {
         self.health.record_capacity_probe(total, used, free);
     }
 
+    fn scanner_timeout_health_action(&self) -> TimeoutHealthAction {
+        self.timeout_health_policy.scanner_timeout_health_action()
+    }
+
     #[cfg(test)]
     pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
         self.health.force_runtime_state_for_test(state);
@@ -551,9 +672,8 @@ impl LocalDiskWrapper {
 
     /// Monitor disk writability periodically
     async fn monitor_disk_writable(disk: Arc<LocalDisk>, health: Arc<DiskHealthTracker>, cancel_token: CancellationToken) {
-        // TODO: config interval
-
         let mut interval = time::interval(get_drive_active_check_interval());
+        let active_check_timeout = get_drive_active_check_timeout();
 
         loop {
             tokio::select! {
@@ -592,7 +712,7 @@ impl LocalDiskWrapper {
                         &test_obj,
                         &TEST_DATA,
                         true,
-                        get_drive_active_check_timeout(),
+                        active_check_timeout,
                     )
                     .await
                     .is_err()
@@ -687,6 +807,7 @@ impl LocalDiskWrapper {
     /// Monitor disk status and try to bring it back online
     async fn monitor_disk_status(disk: Arc<LocalDisk>, health: Arc<DiskHealthTracker>, cancel_token: CancellationToken) {
         let check_every = get_drive_returning_probe_interval();
+        let active_check_timeout = get_drive_active_check_timeout();
 
         let mut interval = time::interval(check_every);
 
@@ -707,7 +828,7 @@ impl LocalDiskWrapper {
                         &test_obj,
                         &TEST_DATA,
                         false,
-                        get_drive_active_check_timeout(),
+                        active_check_timeout,
                     )
                     .await
                     {
@@ -804,6 +925,21 @@ impl LocalDiskWrapper {
         operation: F,
         timeout_duration: Duration,
     ) -> Result<T>
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = Result<T>>,
+    {
+        self.track_disk_health_with_op_and_timeout_action(op, operation, timeout_duration, TimeoutHealthAction::MarkFailure)
+            .await
+    }
+
+    async fn track_disk_health_with_op_and_timeout_action<T, F, Fut>(
+        &self,
+        op: &'static str,
+        operation: F,
+        timeout_duration: Duration,
+        timeout_health_action: TimeoutHealthAction,
+    ) -> Result<T>
     where
         F: FnOnce() -> Fut,
         Fut: std::future::Future<Output = Result<T>>,
@@ -848,7 +984,9 @@ impl LocalDiskWrapper {
             Err(_) => {
                 // Timeout occurred, mark disk as potentially faulty and decrement waiting counter
                 self.health.decrement_waiting();
-                if self.health.mark_failure(&self.endpoint(), "operation_timeout") {
+                if timeout_health_action == TimeoutHealthAction::MarkFailure
+                    && self.health.mark_failure(&self.endpoint(), "operation_timeout")
+                {
                     self.spawn_recovery_monitor_if_needed();
                 }
                 counter!(
@@ -872,10 +1010,11 @@ impl LocalDiskWrapper {
 #[async_trait::async_trait]
 impl DiskAPI for LocalDiskWrapper {
     async fn read_metadata(&self, volume: &str, path: &str) -> Result<Bytes> {
-        self.track_disk_health_with_op(
+        self.track_disk_health_with_op_and_timeout_action(
             "read_metadata",
             || async { self.disk.read_metadata(volume, path).await },
             get_drive_metadata_timeout(),
+            self.scanner_timeout_health_action(),
         )
         .await
     }
@@ -952,7 +1091,7 @@ impl DiskAPI for LocalDiskWrapper {
             return Err(DiskError::FaultyDisk);
         }
 
-        self.track_disk_health_with_op(
+        self.track_disk_health_with_op_and_timeout_action(
             "disk_info",
             || async {
                 let result = self.disk.disk_info(opts).await?;
@@ -966,6 +1105,7 @@ impl DiskAPI for LocalDiskWrapper {
                 Ok(result)
             },
             get_drive_disk_info_timeout(),
+            self.scanner_timeout_health_action(),
         )
         .await
     }
@@ -996,8 +1136,13 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn walk_dir<W: tokio::io::AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
-        self.track_disk_health_with_op("walk_dir", || async { self.disk.walk_dir(opts, wr).await }, get_drive_walkdir_timeout())
-            .await
+        self.track_disk_health_with_op_and_timeout_action(
+            "walk_dir",
+            || async { self.disk.walk_dir(opts, wr).await },
+            get_drive_walkdir_timeout(),
+            self.scanner_timeout_health_action(),
+        )
+        .await
     }
 
     async fn delete_version(
@@ -1104,10 +1249,11 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn list_dir(&self, origvolume: &str, volume: &str, dir_path: &str, count: i32) -> Result<Vec<String>> {
-        self.track_disk_health_with_op(
+        self.track_disk_health_with_op_and_timeout_action(
             "list_dir",
             || async { self.disk.list_dir(origvolume, volume, dir_path, count).await },
             get_drive_list_dir_timeout(),
+            self.scanner_timeout_health_action(),
         )
         .await
     }
@@ -1203,19 +1349,75 @@ mod tests {
     use super::*;
     use crate::disk::endpoint::Endpoint;
     use crate::disk::health_state::RuntimeDriveHealthState;
+    use std::{
+        io,
+        pin::Pin,
+        task::{Context, Poll},
+    };
+    use tokio::io::AsyncWrite;
+
+    struct PendingWriter;
+
+    impl AsyncWrite for PendingWriter {
+        fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<io::Result<usize>> {
+            Poll::Pending
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
 
     #[test]
     fn drive_metadata_timeout_uses_default_when_unset() {
         temp_env::with_var_unset(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, || {
             temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
-                assert_eq!(
-                    get_drive_metadata_timeout(),
-                    Duration::from_secs(rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS)
+                temp_env::with_var_unset(rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE, || {
+                    assert_eq!(
+                        get_drive_metadata_timeout(),
+                        Duration::from_secs(rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS)
+                    );
+                });
+            });
+        });
+    }
+
+    #[test]
+    fn drive_metadata_timeout_uses_high_latency_profile_when_unset() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, || {
+            temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                temp_env::with_var(
+                    rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE,
+                    Some(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY),
+                    || {
+                        assert_eq!(
+                            get_drive_metadata_timeout(),
+                            Duration::from_secs(rustfs_config::DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS)
+                        );
+                    },
                 );
             });
         });
     }
 
+    #[test]
+    fn drive_metadata_timeout_invalid_profile_falls_back_to_default() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, || {
+            temp_env::with_var_unset(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, || {
+                temp_env::with_var(rustfs_config::ENV_DRIVE_TIMEOUT_PROFILE, Some("invalid"), || {
+                    assert_eq!(
+                        get_drive_metadata_timeout(),
+                        Duration::from_secs(rustfs_config::DEFAULT_DRIVE_METADATA_TIMEOUT_SECS)
+                    );
+                });
+            });
+        });
+    }
+
     #[test]
     fn drive_metadata_timeout_uses_legacy_fallback_when_canonical_unset() {
         temp_env::with_var_unset(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, || {
@@ -1323,6 +1525,182 @@ mod tests {
         assert!(health.offline_duration().is_none());
     }
 
+    #[tokio::test]
+    async fn ignored_timeout_does_not_mark_drive_failure() {
+        let dir = tempfile::tempdir().expect("temp dir should be created");
+        let endpoint =
+            Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+        let wrapper = LocalDiskWrapper::new(disk, false);
+
+        let result = wrapper
+            .track_disk_health_with_op_and_timeout_action(
+                "walk_dir",
+                || async {
+                    tokio::time::sleep(Duration::from_millis(20)).await;
+                    Ok(())
+                },
+                Duration::from_millis(1),
+                TimeoutHealthAction::IgnoreFailure,
+            )
+            .await;
+
+        assert_eq!(result.expect_err("operation should time out"), DiskError::Timeout);
+        assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+        assert!(!wrapper.health.is_faulty());
+    }
+
+    #[tokio::test]
+    async fn walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure() {
+        temp_env::async_with_vars(
+            [
+                (rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1")),
+                (
+                    rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION,
+                    Some(rustfs_config::DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER),
+                ),
+            ],
+            async {
+                let dir = tempfile::tempdir().expect("temp dir should be created");
+                let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8"))
+                    .expect("endpoint should parse");
+                let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+                let wrapper = LocalDiskWrapper::new(disk, false);
+                let bucket = "test-bucket";
+                let object = "test-object";
+
+                wrapper.make_volume(bucket).await.expect("bucket should be created");
+
+                let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+                file_info.volume = bucket.to_string();
+                file_info.name = object.to_string();
+                file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+                file_info.erasure.index = 1;
+
+                wrapper
+                    .write_metadata("", bucket, object, file_info)
+                    .await
+                    .expect("object metadata should be written");
+
+                let mut writer = PendingWriter;
+                let result = wrapper
+                    .walk_dir(
+                        WalkDirOptions {
+                            bucket: bucket.to_string(),
+                            recursive: true,
+                            ..Default::default()
+                        },
+                        &mut writer,
+                    )
+                    .await;
+
+                assert_eq!(result.expect_err("walk_dir should time out"), DiskError::Timeout);
+                assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+                assert!(!wrapper.health.is_faulty());
+            },
+        )
+        .await;
+    }
+
+    #[tokio::test]
+    async fn walk_dir_writer_backpressure_timeout_marks_drive_failure_by_default() {
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let dir = tempfile::tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+            let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+            let wrapper = LocalDiskWrapper::new(disk, false);
+            let bucket = "test-bucket";
+            let object = "test-object";
+
+            wrapper.make_volume(bucket).await.expect("bucket should be created");
+
+            let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+            file_info.volume = bucket.to_string();
+            file_info.name = object.to_string();
+            file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+            file_info.erasure.index = 1;
+
+            wrapper
+                .write_metadata("", bucket, object, file_info)
+                .await
+                .expect("object metadata should be written");
+
+            let mut writer = PendingWriter;
+            let result = wrapper
+                .walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                )
+                .await;
+
+            assert_eq!(result.expect_err("walk_dir should time out"), DiskError::Timeout);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Suspect);
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    async fn default_timeout_marks_drive_failure() {
+        let dir = tempfile::tempdir().expect("temp dir should be created");
+        let endpoint =
+            Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+        let wrapper = LocalDiskWrapper::new(disk, false);
+
+        let result = wrapper
+            .track_disk_health_with_op(
+                "read_metadata",
+                || async {
+                    tokio::time::sleep(Duration::from_millis(20)).await;
+                    Ok(())
+                },
+                Duration::from_millis(1),
+            )
+            .await;
+
+        assert_eq!(result.expect_err("operation should time out"), DiskError::Timeout);
+        assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Suspect);
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn drive_timeout_health_policy_defaults_to_mark_failure() {
+        temp_env::with_var_unset(rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION, || {
+            let policy = get_drive_timeout_health_policy();
+            assert_eq!(policy, TimeoutHealthPolicy::MarkFailure);
+            assert_eq!(policy.scanner_timeout_health_action(), TimeoutHealthAction::MarkFailure);
+        });
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn drive_timeout_health_policy_respects_ignore_scanner() {
+        temp_env::with_var(
+            rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION,
+            Some(rustfs_config::DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER),
+            || {
+                let policy = get_drive_timeout_health_policy();
+                assert_eq!(policy, TimeoutHealthPolicy::IgnoreScanner);
+                assert_eq!(policy.scanner_timeout_health_action(), TimeoutHealthAction::IgnoreFailure);
+            },
+        );
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn drive_timeout_health_policy_invalid_value_falls_back_to_default() {
+        temp_env::with_var(rustfs_config::ENV_DRIVE_TIMEOUT_HEALTH_ACTION, Some("invalid"), || {
+            let policy = get_drive_timeout_health_policy();
+            assert_eq!(policy, TimeoutHealthPolicy::MarkFailure);
+            assert_eq!(policy.scanner_timeout_health_action(), TimeoutHealthAction::MarkFailure);
+        });
+    }
+
     #[test]
     fn reset_for_store_init_retry_clears_faulty_and_back_online() {
         let endpoint = Endpoint::try_from("/tmp/reset-store-init-retry").expect("endpoint should parse");

crates/ecstore/src/disk/endpoint.rs

@@ -82,17 +82,22 @@ impl TryFrom<&str> for Endpoint {
                 #[cfg(not(windows))]
                 let path = Path::new(&path).absolutize()?;
 
-                // On windows having a preceding SlashSeparator will cause problems, if the
-                // command line already has C:/<export-folder/ in it. Final resulting
-                // path on windows might become C:/C:/ this will cause problems
-                // of starting rustfs server properly in distributed mode on windows.
-                // As a special case make sure to trim the separator.
                 #[cfg(windows)]
-                let path = Path::new(&path[1..]).absolutize()?;
+                let path = if has_leading_slash_windows_drive(&path) {
+                    // Url::path() exposes file-like Windows paths as `/C:/...`.
+                    // Strip only that synthetic leading slash; plain URL paths
+                    // such as `/export1` must stay URL paths, not become
+                    // relative paths under the current drive.
+                    Path::new(&path[1..]).absolutize()?.to_string_lossy().into_owned()
+                } else {
+                    path
+                };
+                #[cfg(windows)]
+                let path = Path::new(&path);
 
                 debug!("endpoint try_from: path={}", path.display());
 
-                if path.parent().is_none() || Path::new("").eq(&path) {
+                if path.parent().is_none() || path.as_os_str().is_empty() {
                     return Err(Error::other("empty or root path is not supported in URL endpoint"));
                 }
 
@@ -217,6 +222,12 @@ impl Endpoint {
     }
 }
 
+#[cfg(windows)]
+fn has_leading_slash_windows_drive(path: &str) -> bool {
+    let bytes = path.as_bytes();
+    bytes.len() >= 4 && bytes[0] == b'/' && bytes[1].is_ascii_alphabetic() && bytes[2] == b':' && bytes[3] == b'/'
+}
+
 /// parse a file path into a URL.
 fn url_parse_from_file_path(value: &str) -> Result<Url> {
     // Only check if the arg is an ip address and ask for scheme since its absent.
@@ -242,6 +253,14 @@ fn url_parse_from_file_path(value: &str) -> Result<Url> {
 mod test {
     use super::*;
 
+    fn expected_file_path(path: &str) -> String {
+        Path::new(path).absolutize().unwrap().to_string_lossy().replace('\\', "/")
+    }
+
+    fn expected_file_url(path: &str) -> Url {
+        url_parse_from_file_path(path).unwrap()
+    }
+
     #[test]
     fn test_new_endpoint() {
         #[derive(Default)]
@@ -255,7 +274,7 @@ mod test {
         let u2 = Url::parse("https://example.org/path").unwrap();
         let u4 = Url::parse("http://192.168.253.200/path").unwrap();
         let u6 = Url::parse("http://server:/path").unwrap();
-        let root_slash_foo = Url::from_file_path("/foo").unwrap();
+        let root_slash_foo = expected_file_url("/foo");
 
         let test_cases = [
             TestCase {
@@ -416,7 +435,7 @@ mod test {
         // Test file path display
         let file_endpoint = Endpoint::try_from("/tmp/data").unwrap();
         let display_str = format!("{file_endpoint}");
-        assert_eq!(display_str, "/tmp/data");
+        assert_eq!(display_str, expected_file_path("/tmp/data"));
 
         // Test URL display
         let url_endpoint = Endpoint::try_from("http://example.com:9000/path").unwrap();
@@ -479,12 +498,25 @@ mod test {
     #[test]
     fn test_endpoint_get_file_path() {
         let file_endpoint = Endpoint::try_from("/tmp/data").unwrap();
-        assert_eq!(file_endpoint.get_file_path(), "/tmp/data");
+        assert_eq!(file_endpoint.get_file_path(), expected_file_path("/tmp/data"));
 
         let url_endpoint = Endpoint::try_from("http://example.com:9000/path/to/data").unwrap();
         assert_eq!(url_endpoint.get_file_path(), "/path/to/data");
     }
 
+    #[cfg(windows)]
+    #[test]
+    fn test_windows_url_drive_path_requires_separator_after_colon() {
+        let drive_path_endpoint = Endpoint::try_from("http://host/C:/data").unwrap();
+        assert_eq!(drive_path_endpoint.get_type(), EndpointType::Url);
+        assert!(has_leading_slash_windows_drive(Url::parse("http://host/C:/data").unwrap().path()));
+
+        let url_path_endpoint = Endpoint::try_from("http://host/C:foo").unwrap();
+        assert_eq!(url_path_endpoint.get_type(), EndpointType::Url);
+        assert!(!has_leading_slash_windows_drive(Url::parse("http://host/C:foo").unwrap().path()));
+        assert_eq!(url_path_endpoint.get_file_path(), "/C:foo");
+    }
+
     #[test]
     fn test_endpoint_clone_and_equality() {
         let endpoint1 = Endpoint::try_from("/tmp/data").unwrap();
@@ -503,7 +535,7 @@ mod test {
         // Test with complex paths
         let complex_path = "/var/lib/rustfs/data/bucket1";
         let endpoint = Endpoint::try_from(complex_path).unwrap();
-        assert_eq!(endpoint.get_file_path(), complex_path);
+        assert_eq!(endpoint.get_file_path(), expected_file_path(complex_path));
         assert!(endpoint.is_local);
         assert_eq!(endpoint.get_type(), EndpointType::Path);
     }
@@ -512,7 +544,7 @@ mod test {
     fn test_endpoint_with_spaces_in_path() {
         let path_with_spaces = "/Users/test/Library/Application Support/rustfs/data";
         let endpoint = Endpoint::try_from(path_with_spaces).unwrap();
-        assert_eq!(endpoint.get_file_path(), path_with_spaces);
+        assert_eq!(endpoint.get_file_path(), expected_file_path(path_with_spaces));
         assert!(endpoint.is_local);
         assert_eq!(endpoint.get_type(), EndpointType::Path);
     }
@@ -532,7 +564,7 @@ mod test {
         // Verify that get_file_path() decodes the percent-encoded path correctly
         assert_eq!(
             endpoint.get_file_path(),
-            "/Users/test/Library/Application Support/rustfs/data",
+            expected_file_path("/Users/test/Library/Application Support/rustfs/data"),
             "get_file_path() should decode percent-encoded spaces"
         );
     }
@@ -544,7 +576,7 @@ mod test {
         let endpoint = Endpoint::try_from(path_with_special).unwrap();
 
         // get_file_path() should return the original path with decoded characters
-        assert_eq!(endpoint.get_file_path(), path_with_special);
+        assert_eq!(endpoint.get_file_path(), expected_file_path(path_with_special));
     }
 
     #[test]

crates/ecstore/src/disk/fs.rs

@@ -545,7 +545,7 @@ mod tests {
 
         // Create two different files
         tokio::fs::write(&file1_path, b"content1").await.unwrap();
-        tokio::fs::write(&file2_path, b"content2").await.unwrap();
+        tokio::fs::write(&file2_path, b"different content").await.unwrap();
 
         // Get metadata
         let metadata1 = tokio::fs::metadata(&file1_path).await.unwrap();

crates/ecstore/src/disk/local.rs

@@ -1163,33 +1163,45 @@ impl LocalDisk {
     }
     // write_all_internal do write file
     async fn write_all_internal(&self, file_path: &Path, data: InternalBuf<'_>, sync: bool, skip_parent: &Path) -> Result<()> {
-        let flags = O_CREATE | O_WRONLY | O_TRUNC;
-
-        let mut f = {
-            if sync {
-                // TODO: support sync
-                self.open_file(file_path, flags, skip_parent).await?
-            } else {
-                self.open_file(file_path, flags, skip_parent).await?
-            }
+        let skip_parent = if skip_parent.as_os_str().is_empty() {
+            self.root.as_path()
+        } else {
+            skip_parent
         };
 
         match data {
             InternalBuf::Ref(buf) => {
+                let mut f = self.open_file(file_path, O_CREATE | O_WRONLY | O_TRUNC, skip_parent).await?;
                 f.write_all(buf).await.map_err(to_file_error)?;
             }
             InternalBuf::Owned(buf) => {
-                // Reduce one copy by using the owned buffer directly.
-                // It may be more efficient for larger writes.
-                let mut f = f.into_std().await;
-                let task = tokio::task::spawn_blocking(move || {
-                    use std::io::Write as _;
-                    f.write_all(buf.as_ref()).map_err(to_file_error)
-                });
-                task.await??;
+                let path = file_path.to_path_buf();
+                if let Some(parent) = path.parent()
+                    && parent != skip_parent
+                {
+                    os::make_dir_all(parent, skip_parent).await?;
+                }
+
+                tokio::task::spawn_blocking(move || {
+                    let mut f = std::fs::OpenOptions::new()
+                        .create(true)
+                        .write(true)
+                        .truncate(true)
+                        .open(path)
+                        .map_err(to_file_error)?;
+
+                    std::io::Write::write_all(&mut f, buf.as_ref()).map_err(to_file_error)?;
+
+                    Ok::<(), std::io::Error>(())
+                })
+                .await
+                .map_err(DiskError::from)??;
             }
         }
 
+        // Keep existing durability contract: this path intentionally ignores `sync`.
+        let _ = sync;
+
         Ok(())
     }
 
@@ -1199,7 +1211,9 @@ impl LocalDisk {
             skip_parent = self.root.as_path();
         }
 
-        if let Some(parent) = path.as_ref().parent() {
+        if let Some(parent) = path.as_ref().parent()
+            && parent != skip_parent
+        {
             os::make_dir_all(parent, skip_parent).await?;
         }
 
@@ -1208,6 +1222,11 @@ impl LocalDisk {
         Ok(f)
     }
 
+    async fn open_file_read_only(&self, path: impl AsRef<Path>) -> Result<File> {
+        let f = super::fs::open_file(path.as_ref(), O_RDONLY).await.map_err(to_file_error)?;
+        Ok(f)
+    }
+
     #[allow(dead_code)]
     fn get_metrics(&self) -> DiskMetrics {
         DiskMetrics::default()
@@ -1234,6 +1253,7 @@ impl LocalDisk {
     }
 
     #[async_recursion::async_recursion]
+    #[allow(clippy::too_many_arguments)]
     async fn scan_dir<W>(
         &self,
         mut current: String,
@@ -1242,6 +1262,7 @@ impl LocalDisk {
         out: &mut MetacacheWriter<W>,
         objs_returned: &mut i32,
         skip_current_dir_object: bool,
+        multipart_dir_to_skip: Option<HashSet<String>>,
     ) -> Result<()>
     where
         W: AsyncWrite + Unpin + Send,
@@ -1298,6 +1319,14 @@ impl LocalDisk {
             if opts.limit > 0 && *objs_returned >= opts.limit {
                 return Ok(());
             }
+            // check multipart dir
+            if skip_current_dir_object
+                && let Some(ref dir_to_skip) = multipart_dir_to_skip
+                && dir_to_skip.contains(entry.trim_end_matches(SLASH_SEPARATOR))
+            {
+                *item = "".to_owned();
+                continue;
+            }
             // check prefix
             if !prefix.is_empty() && !entry.starts_with(prefix.as_str()) {
                 *item = "".to_owned();
@@ -1367,15 +1396,25 @@ impl LocalDisk {
             }
         }
 
-        let mut dir_stack: Vec<(String, bool)> = Vec::with_capacity(5);
+        let mut dir_stack: Vec<(String, bool, Option<HashSet<String>>)> = Vec::with_capacity(5);
         // Explicit directory markers and real directories can resolve to the same logical path.
-        let schedule_dir = |dir_stack: &mut Vec<(String, bool)>, dir_name: String, skip_object: bool| {
-            if let Some((last_dir_name, existing_skip_object)) = dir_stack.last_mut()
+        let schedule_dir = |dir_stack: &mut Vec<(String, bool, Option<HashSet<String>>)>,
+                            dir_name: String,
+                            skip_object: bool,
+                            dir_to_skip: Option<HashSet<String>>| {
+            if let Some((last_dir_name, existing_skip_object, existing_dir_to_skip)) = dir_stack.last_mut()
                 && *last_dir_name == dir_name
             {
                 *existing_skip_object |= skip_object;
+                if let Some(existing_dir_to_skip) = existing_dir_to_skip {
+                    if let Some(new_dir_to_skip) = &dir_to_skip {
+                        existing_dir_to_skip.extend(new_dir_to_skip.iter().cloned());
+                    }
+                } else {
+                    *existing_dir_to_skip = dir_to_skip;
+                }
             } else {
-                dir_stack.push((dir_name, skip_object));
+                dir_stack.push((dir_name, skip_object, dir_to_skip));
             }
         };
         prefix = "".to_owned();
@@ -1391,9 +1430,10 @@ impl LocalDisk {
 
             let name = path_join_buf(&[current.as_str(), entry.as_str()]);
 
-            while let Some((pop, skip_object)) = dir_stack.last().cloned()
-                && pop < name
+            while let Some((last_name, _, _)) = dir_stack.last()
+                && *last_name < name
             {
+                let (pop, skip_object, dir_to_skip) = dir_stack.pop().unwrap();
                 out.write_obj(&MetaCacheEntry {
                     name: pop.clone(),
                     ..Default::default()
@@ -1401,11 +1441,11 @@ impl LocalDisk {
                 .await?;
 
                 if opts.recursive
-                    && let Err(er) = Box::pin(self.scan_dir(pop, prefix.clone(), opts, out, objs_returned, skip_object)).await
+                    && let Err(er) =
+                        Box::pin(self.scan_dir(pop, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await
                 {
                     error!("scan_dir err {:?}", er);
                 }
-                dir_stack.pop();
             }
 
             let mut meta = MetaCacheEntry {
@@ -1442,11 +1482,24 @@ impl LocalDisk {
                     // }
 
                     if opts.recursive {
+                        let mut dir_to_skip = HashSet::new();
+                        if let Ok(file_meta) = FileMeta::load(&res)
+                            && let Ok(data_dirs) = file_meta.get_data_dirs()
+                        {
+                            for data_dir in data_dirs.iter().flatten() {
+                                dir_to_skip.insert(data_dir.to_string());
+                            }
+                        }
                         let mut dir_name = meta.name.clone();
                         if !dir_name.ends_with(SLASH_SEPARATOR) {
                             dir_name.push_str(SLASH_SEPARATOR);
                         }
-                        schedule_dir(&mut dir_stack, dir_name, true);
+                        schedule_dir(
+                            &mut dir_stack,
+                            dir_name,
+                            true,
+                            if dir_to_skip.is_empty() { None } else { Some(dir_to_skip) },
+                        );
                     }
                 }
                 Err(err) => {
@@ -1455,7 +1508,7 @@ impl LocalDisk {
                         // If dirObject, but no metadata (which is unexpected) we skip it.
                         if !is_dir_obj && !is_empty_dir(self.get_object_path(&opts.bucket, &meta.name)?).await {
                             meta.name.push_str(SLASH_SEPARATOR);
-                            schedule_dir(&mut dir_stack, meta.name, false);
+                            schedule_dir(&mut dir_stack, meta.name, false, None);
                         }
                     }
 
@@ -1464,7 +1517,7 @@ impl LocalDisk {
             };
         }
 
-        while let Some((dir, skip_object)) = dir_stack.pop() {
+        while let Some((dir, skip_object, dir_to_skip)) = dir_stack.pop() {
             if opts.limit > 0 && *objs_returned >= opts.limit {
                 return Ok(());
             }
@@ -1476,7 +1529,8 @@ impl LocalDisk {
             .await?;
 
             if opts.recursive
-                && let Err(er) = Box::pin(self.scan_dir(dir, prefix.clone(), opts, out, objs_returned, skip_object)).await
+                && let Err(er) =
+                    Box::pin(self.scan_dir(dir, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await
             {
                 warn!("scan_dir err {:?}", &er);
             }
@@ -1958,11 +2012,7 @@ impl DiskAPI for LocalDisk {
             let meta_op = match lstat_std(&src_file_path).map_err(|e| to_file_error(e).into()) {
                 Ok(meta) => Some(meta),
                 Err(e) => {
-                    if e != DiskError::FileNotFound {
-                        return Err(e);
-                    }
-
-                    None
+                    return Err(e);
                 }
             };
 
@@ -1974,10 +2024,22 @@ impl DiskAPI for LocalDisk {
             }
 
             remove_std(&dst_file_path).map_err(to_file_error)?;
+        } else {
+            let meta = lstat_std(&src_file_path).map_err(|e| -> DiskError { to_file_error(e).into() })?;
+            if meta.is_dir() {
+                warn!("rename_part src is dir {:?}", &src_file_path);
+                return Err(DiskError::FileAccessDenied);
+            }
         }
 
         rename_all(&src_file_path, &dst_file_path, &dst_volume_dir).await?;
 
+        let dst_meta = lstat_std(&dst_file_path).map_err(|e| -> DiskError { to_file_error(e).into() })?;
+        if src_is_dir != dst_meta.is_dir() {
+            warn!("rename_part dst type changed after rename {:?}", &dst_file_path);
+            return Err(DiskError::FileAccessDenied);
+        }
+
         self.write_all(dst_volume, format!("{dst_path}.meta").as_str(), meta).await?;
 
         if let Some(parent) = src_file_path.parent() {
@@ -2107,7 +2169,7 @@ impl DiskAPI for LocalDisk {
         let file_path = self.get_object_path(volume, path)?;
         check_path_length(file_path.to_string_lossy().as_ref())?;
 
-        let f = self.open_file(file_path, O_RDONLY, volume_dir).await?;
+        let f = self.open_file_read_only(file_path).await?;
 
         Ok(Box::new(f))
     }
@@ -2124,7 +2186,7 @@ impl DiskAPI for LocalDisk {
         let file_path = self.get_object_path(volume, path)?;
         check_path_length(file_path.to_string_lossy().as_ref())?;
 
-        let mut f = self.open_file(file_path, O_RDONLY, volume_dir).await?;
+        let mut f = self.open_file_read_only(file_path).await?;
 
         let meta = f.metadata().await?;
         let end_offset = offset.checked_add(length).ok_or(DiskError::FileCorrupt)?;
@@ -2153,9 +2215,6 @@ impl DiskAPI for LocalDisk {
     #[allow(unsafe_code)]
     #[tracing::instrument(level = "debug", skip(self))]
     async fn read_file_zero_copy(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<Bytes> {
-        use std::time::Instant;
-
-        let start = Instant::now();
         let volume_dir = self.get_bucket_path(volume)?;
         if !skip_access_checks(volume) {
             access(&volume_dir)
@@ -2188,6 +2247,9 @@ impl DiskAPI for LocalDisk {
         #[cfg(unix)]
         {
             use memmap2::MmapOptions;
+            use std::time::Instant;
+
+            let start = Instant::now();
             let file_path_clone = file_path.clone();
 
             let should_reclaim_after_read = should_reclaim_file_cache_after_read(length);
@@ -2272,8 +2334,7 @@ impl DiskAPI for LocalDisk {
                 f.seek(SeekFrom::Start(offset as u64)).await?;
             }
 
-            let mut buffer = Vec::with_capacity(length);
-            buffer.resize(length, 0);
+            let mut buffer = vec![0; length];
             f.read_exact(&mut buffer).await?;
 
             Ok(Bytes::from(buffer))
@@ -2329,6 +2390,7 @@ impl DiskAPI for LocalDisk {
         let mut objs_returned = 0;
 
         let mut skip_current_dir_object = false;
+        let mut multipart_dir_to_skip: HashSet<String> = HashSet::new();
         if opts.base_dir.ends_with(SLASH_SEPARATOR) {
             if let Ok(data) = self
                 .read_metadata(
@@ -2352,10 +2414,23 @@ impl DiskAPI for LocalDisk {
                 let fpath =
                     self.get_object_path(&opts.bucket, path_join_buf(&[opts.base_dir.as_str(), STORAGE_FORMAT_FILE]).as_str())?;
 
-                if let Ok(meta) = tokio::fs::metadata(fpath).await
+                if let Ok(meta) = tokio::fs::metadata(&fpath).await
                     && meta.is_file()
                 {
                     skip_current_dir_object = true;
+                    if let Ok(meta_bytes) = self
+                        .read_metadata(
+                            opts.bucket.as_str(),
+                            path_join_buf(&[opts.base_dir.as_str(), STORAGE_FORMAT_FILE]).as_str(),
+                        )
+                        .await
+                        && let Ok(file_meta) = FileMeta::load(&meta_bytes)
+                        && let Ok(data_dirs) = file_meta.get_data_dirs()
+                    {
+                        for data_dir in data_dirs.iter().flatten() {
+                            multipart_dir_to_skip.insert(data_dir.to_string());
+                        }
+                    }
                 }
             }
         }
@@ -2367,6 +2442,11 @@ impl DiskAPI for LocalDisk {
             &mut out,
             &mut objs_returned,
             skip_current_dir_object,
+            if multipart_dir_to_skip.is_empty() {
+                None
+            } else {
+                Some(multipart_dir_to_skip)
+            },
         )
         .await?;
 
@@ -2465,32 +2545,41 @@ impl DiskAPI for LocalDisk {
 
         // TODO: Healing
 
-        let search_version_id = fi.version_id.or(Some(Uuid::nil()));
+        let version_id = fi.version_id.unwrap_or_default();
+        let search_version_id = Some(version_id);
+        let no_inline = fi.data.is_none() && fi.size > 0;
 
         // Check if there's an existing version with the same version_id that has a data_dir to clean up
-        let has_old_data_dir = {
-            xlmeta.find_version(search_version_id).ok().and_then(|(_, ver)| {
-                // shard_count == 0 means no other version shares this data_dir
-                ver.get_data_dir()
-                    .filter(|&data_dir| xlmeta.shard_data_dir_count(&search_version_id, &Some(data_dir)) == 0)
-            })
-        };
+        // Reuse one metadata scan to find the version data_dir and determine whether it is shared.
+        let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(search_version_id);
         if let Some(old_data_dir) = has_old_data_dir.as_ref() {
-            let _ = xlmeta.data.remove(vec![search_version_id.unwrap_or_default(), *old_data_dir]);
+            let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
         }
 
-        xlmeta.add_version(fi.clone())?;
+        xlmeta.add_version(fi)?;
 
         if xlmeta.versions.len() <= 10 {
             // TODO: Sign
         }
 
-        let new_dst_buf = xlmeta.marshal_msg()?;
-
-        self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf.into())
-            .await?;
         if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref() {
-            let no_inline = fi.data.is_none() && fi.size > 0;
+            let src_file_parent = src_file_path.parent().unwrap_or(src_volume_dir.as_path());
+            let meta_skip_parent = if no_inline {
+                src_file_parent
+            } else {
+                src_volume_dir.as_path()
+            };
+            let new_dst_buf = xlmeta.marshal_msg()?;
+
+            self.write_all_private(
+                src_volume,
+                format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(),
+                new_dst_buf.into(),
+                true,
+                meta_skip_parent,
+            )
+            .await?;
+
             if no_inline && let Err(err) = rename_all(&src_data_path, &dst_data_path, &skip_parent).await {
                 let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
                 info!(
@@ -2499,6 +2588,10 @@ impl DiskAPI for LocalDisk {
                 );
                 return Err(err);
             }
+        } else {
+            let new_dst_buf = xlmeta.marshal_msg()?;
+            self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf.into())
+                .await?;
         }
 
         if let Some(old_data_dir) = has_old_data_dir {
@@ -2985,7 +3078,6 @@ impl DiskAPI for LocalDisk {
     #[tracing::instrument(skip(self))]
     async fn disk_info(&self, _: &DiskInfoOptions) -> Result<DiskInfo> {
         let mut info = Cache::get(self.disk_info_cache.clone()).await?;
-        // TODO: nr_requests, rotational
         info.nr_requests = self.nrrequests;
         info.rotational = self.rotational;
         info.mount_path = self.path().to_str().unwrap().to_string();
@@ -3152,7 +3244,7 @@ mod test {
         };
         let mut objs_returned = 0;
 
-        disk.scan_dir("".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false)
+        disk.scan_dir("".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false, None)
             .await
             .unwrap();
         out.close().await.unwrap();
@@ -3207,7 +3299,7 @@ mod test {
         };
         let mut objs_returned = 0;
 
-        disk.scan_dir("marker/".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false)
+        disk.scan_dir("marker/".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false, None)
             .await
             .unwrap();
         out.close().await.unwrap();
@@ -3267,7 +3359,7 @@ mod test {
             };
             let mut objs_returned = 0;
 
-            disk.scan_dir(base_dir.to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false)
+            disk.scan_dir(base_dir.to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false, None)
                 .await
                 .unwrap();
             out.close().await.unwrap();
@@ -3328,6 +3420,155 @@ mod test {
         assert_eq!(double_count as usize, double_names.len());
     }
 
+    #[tokio::test]
+    async fn test_walk_dir_ignore_multipart_dirs() {
+        use rustfs_filemeta::MetacacheReader;
+        use tempfile::tempdir;
+
+        const UUID_MULTIPART_1: &str = "8b262d24-fcf9-473d-a4cd-f9b27f24f60e";
+        const UUID_MULTIPART_2: &str = "fbf3183c-63be-45cc-b3bf-424ddb7f95f8";
+        const UUID_OBJ: &str = "db8b9b74-9016-4f9e-83e9-82a772947d28";
+        const VER_ID_1: &str = "c683f9f8-c0a1-4bc5-8a67-0faafa839a1a";
+        const VER_ID_2: &str = "a4b84f6e-c8ba-461b-8f9d-43feb0893efb";
+        const VER_ID_3: &str = "892c9ae7-2bb3-44ee-9a71-bc7ddf08d765";
+        const BASE_DIR: &str = "dir1/obj/";
+        const MULTIPART_DIR: &str = "multipart-file";
+        const DIR_IN_MULTIPART_DIR: &str = "dir-in-multipart";
+        const EMPTY_STR: &str = "";
+
+        let parse_uuid = |s: &str| Uuid::parse_str(s).unwrap();
+        let create_file_info = |version_id: &str, data_dir: &str| FileInfo {
+            version_id: Some(parse_uuid(version_id)),
+            data_dir: Some(parse_uuid(data_dir)),
+            mod_time: Some(OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+
+        let dir = tempdir().unwrap();
+        let obj_base = dir.path().join("test-bucket").join(BASE_DIR);
+        let multipart_base = obj_base.join(MULTIPART_DIR);
+        let dir_in_multipart_base = multipart_base.join(DIR_IN_MULTIPART_DIR);
+
+        fs::create_dir_all(&multipart_base).await.unwrap();
+        for uuid in &[UUID_MULTIPART_1, UUID_MULTIPART_2] {
+            fs::create_dir_all(multipart_base.join(uuid)).await.unwrap();
+            fs::write(multipart_base.join(uuid).join("part.1"), b"part").await.unwrap();
+        }
+        fs::create_dir_all(obj_base.join(UUID_OBJ)).await.unwrap();
+        fs::write(obj_base.join(UUID_OBJ).join("part.1"), b"part").await.unwrap();
+
+        fs::create_dir_all(&dir_in_multipart_base).await.unwrap();
+        fs::write(dir_in_multipart_base.join(STORAGE_FORMAT_FILE), b"meta")
+            .await
+            .unwrap();
+
+        let mut fm = FileMeta::default();
+        fm.add_version(create_file_info(VER_ID_1, UUID_MULTIPART_1)).unwrap();
+        fm.add_version(create_file_info(VER_ID_2, UUID_MULTIPART_2)).unwrap();
+        fs::write(multipart_base.join(STORAGE_FORMAT_FILE), fm.marshal_msg().unwrap())
+            .await
+            .unwrap();
+
+        let mut fm = FileMeta::default();
+        fm.add_version(create_file_info(VER_ID_3, UUID_OBJ)).unwrap();
+        fs::write(obj_base.join(STORAGE_FORMAT_FILE), fm.marshal_msg().unwrap())
+            .await
+            .unwrap();
+
+        let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap();
+        let disk = LocalDisk::new(&endpoint, false).await.unwrap();
+
+        let (reader, mut writer) = tokio::io::duplex(4096);
+        disk.walk_dir(
+            WalkDirOptions {
+                bucket: "test-bucket".to_string(),
+                base_dir: BASE_DIR.to_string(),
+                recursive: true,
+                filter_prefix: Some(EMPTY_STR.to_string()),
+                ..Default::default()
+            },
+            &mut writer,
+        )
+        .await
+        .unwrap();
+        MetacacheWriter::new(&mut writer).close().await.unwrap();
+
+        let mut reader = MetacacheReader::new(reader);
+        let entries = reader.read_all().await.unwrap();
+        let names: Vec<String> = entries.into_iter().map(|entry| entry.name).collect();
+
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}", BASE_DIR, MULTIPART_DIR))
+                .count(),
+            1
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/", BASE_DIR, MULTIPART_DIR))
+                .count(),
+            1
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}", BASE_DIR, MULTIPART_DIR, DIR_IN_MULTIPART_DIR))
+                .count(),
+            1
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}/", BASE_DIR, MULTIPART_DIR, DIR_IN_MULTIPART_DIR))
+                .count(),
+            1
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}", BASE_DIR, MULTIPART_DIR, UUID_MULTIPART_1))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}", BASE_DIR, MULTIPART_DIR, UUID_MULTIPART_2))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}", BASE_DIR, UUID_OBJ))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}/", BASE_DIR, MULTIPART_DIR, UUID_MULTIPART_1))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}/", BASE_DIR, MULTIPART_DIR, UUID_MULTIPART_2))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/", BASE_DIR, UUID_OBJ))
+                .count(),
+            0
+        );
+    }
+
     #[tokio::test]
     async fn test_make_volume() {
         let p = "./testv0";
@@ -3497,13 +3738,15 @@ mod test {
         let disk_info = disk.disk_info(&disk_info_opts).await.unwrap();
 
         // Basic checks on disk info
-        // Note: On macOS and some other Unix systems, fs_type may be empty
+        // Note: On macOS, Windows, and some other systems, fs_type may be empty
         // because statvfs does not provide filesystem type information.
         // This is a platform limitation, not a bug.
-        #[cfg(not(target_os = "macos"))]
+        #[cfg(not(any(target_os = "macos", windows)))]
         assert!(!disk_info.fs_type.is_empty(), "fs_type should not be empty on this platform");
         assert!(disk_info.total > 0);
         assert!(disk_info.free <= disk_info.total);
+        assert_eq!(disk_info.nr_requests, disk.nrrequests);
+        assert_eq!(disk_info.rotational, disk.rotational);
         assert!(!disk_info.mount_path.is_empty());
         assert!(!disk_info.endpoint.is_empty());
 

crates/ecstore/src/disk/mod.rs

@@ -37,6 +37,7 @@ use crate::disk::disk_store::LocalDiskWrapper;
 use crate::disk::health_state::RuntimeDriveHealthState;
 use crate::disk::local::ScanGuard;
 use crate::rpc::RemoteDisk;
+use crate::rpc::build_internode_data_transport_from_env;
 use bytes::Bytes;
 use endpoint::Endpoint;
 use error::DiskError;
@@ -476,7 +477,8 @@ pub async fn new_disk(ep: &Endpoint, opt: &DiskOption) -> Result<DiskStore> {
         let s = LocalDisk::new(ep, opt.cleanup).await?;
         Ok(Arc::new(Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(s), opt.health_check)))))
     } else {
-        let remote_disk = RemoteDisk::new(ep, opt).await?;
+        let data_transport = build_internode_data_transport_from_env();
+        let remote_disk = RemoteDisk::new(ep, opt, data_transport?).await?;
         Ok(Arc::new(Disk::Remote(Box::new(remote_disk))))
     }
 }
@@ -789,7 +791,6 @@ mod tests {
     use super::*;
     use endpoint::Endpoint;
     use local::LocalDisk;
-    use std::path::PathBuf;
     use tokio::fs;
     use uuid::Uuid;
 
@@ -1094,7 +1095,7 @@ mod tests {
         assert!(disk.is_ok());
 
         let disk = disk.unwrap();
-        assert_eq!(disk.path(), PathBuf::from(test_dir).canonicalize().unwrap());
+        assert_eq!(disk.path(), rustfs_utils::canonicalize(test_dir).unwrap());
         assert!(disk.is_local());
         // Note: is_online() might return false for local disks without proper initialization
         // This is expected behavior for test environments
@@ -1155,6 +1156,7 @@ mod tests {
                 cleanup: false,
                 health_check: false,
             },
+            Arc::new(crate::rpc::TcpHttpInternodeDataTransport),
         )
         .await
         .unwrap();

crates/ecstore/src/disk/os.rs

@@ -215,24 +215,29 @@ pub async fn os_mkdir_all(dir_path: impl AsRef<Path>, base_dir: impl AsRef<Path>
         return Ok(());
     }
 
-    if let Some(parent) = dir_path.as_ref().parent() {
-        // Without recursion support, fall back to create_dir_all
-        if let Err(e) = super::fs::make_dir_all(&parent).await {
-            if e.kind() == io::ErrorKind::AlreadyExists {
-                return Ok(());
-            }
-
-            return Err(e);
-        }
-        // Box::pin(os_mkdir_all(&parent, &base_dir)).await?;
-    }
-
     if let Err(e) = super::fs::mkdir(dir_path.as_ref()).await {
         if e.kind() == io::ErrorKind::AlreadyExists {
             return Ok(());
         }
 
-        return Err(e);
+        if e.kind() != io::ErrorKind::NotFound {
+            return Err(e);
+        }
+
+        if let Some(parent) = dir_path.as_ref().parent() {
+            // Fall back to creating the missing parent chain only when the direct mkdir proves it is required.
+            if let Err(parent_err) = super::fs::make_dir_all(parent).await
+                && parent_err.kind() != io::ErrorKind::AlreadyExists
+            {
+                return Err(parent_err);
+            }
+        }
+
+        if let Err(retry_err) = super::fs::mkdir(dir_path.as_ref()).await
+            && retry_err.kind() != io::ErrorKind::AlreadyExists
+        {
+            return Err(retry_err);
+        }
     }
 
     Ok(())

crates/ecstore/src/endpoints.rs

@@ -749,6 +749,7 @@ fn validate_local_cross_device_mounts(local_paths: &[String]) -> Result<()> {
 
 #[cfg(test)]
 mod test {
+    use path_absolutize::Absolutize;
     use rustfs_utils::must_get_local_ips;
 
     use super::*;
@@ -1452,9 +1453,10 @@ mod test {
     }
 
     fn must_file_path(s: impl AsRef<Path>) -> url::Url {
-        let url = url::Url::from_file_path(s.as_ref());
+        let path = s.as_ref().absolutize().expect("absolute test path");
+        let url = url::Url::from_file_path(&path);
 
-        assert!(url.is_ok(), "failed to convert path to URL: {}", s.as_ref().display());
+        assert!(url.is_ok(), "failed to convert path to URL: {}", path.display());
 
         url.unwrap()
     }

crates/ecstore/src/erasure_coding/erasure.rs

@@ -106,7 +106,7 @@ impl LegacyReedSolomonEncoder {
         Ok(())
     }
 
-    fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+    fn reconstruct_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         let shard_len = shards
             .iter()
             .find_map(|s| s.as_ref().map(|v| v.len()))
@@ -172,6 +172,15 @@ impl LegacyReedSolomonEncoder {
 
         Ok(())
     }
+
+    fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        self.reconstruct_data(shards)?;
+        self.encode_parity(shards)
+    }
+
+    fn encode_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        encode_parity_shards(shards, self.data_shards, self.parity_shards, |shards| self.encode(shards))
+    }
 }
 
 /// Reed-Solomon encoder using reed-solomon-erasure
@@ -224,8 +233,8 @@ impl ReedSolomonEncoder {
         }
     }
 
-    /// Reconstruct missing shards.
-    pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+    /// Reconstruct missing data shards.
+    pub fn reconstruct_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         if let Some(ref rs) = self.encoder {
             rs.reconstruct_data(shards)
                 .map_err(|e| io::Error::other(format!("Reed-Solomon reconstruct failed: {e:?}")))
@@ -233,6 +242,58 @@ impl ReedSolomonEncoder {
             Ok(())
         }
     }
+
+    /// Reconstruct missing data shards and regenerate parity shards.
+    pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        self.reconstruct_data(shards)?;
+        self.encode_parity(shards)
+    }
+
+    fn encode_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        encode_parity_shards(shards, self.data_shards, self.parity_shards, |shards| self.encode(shards))
+    }
+}
+
+fn encode_parity_shards<F>(shards: &mut [Option<Vec<u8>>], data_shards: usize, parity_shards: usize, encode: F) -> io::Result<()>
+where
+    F: FnOnce(SmallVec<[&mut [u8]; 16]>) -> io::Result<()>,
+{
+    let expected_shards = data_shards + parity_shards;
+    if shards.len() != expected_shards {
+        return Err(io::Error::other(format!(
+            "invalid shard count: got {}, expected {}",
+            shards.len(),
+            expected_shards
+        )));
+    }
+
+    let shard_len = shards
+        .iter()
+        .find_map(|s| s.as_ref().map(Vec::len))
+        .ok_or_else(|| io::Error::other("No valid shards found for parity encoding"))?;
+
+    for shard in shards.iter_mut().skip(data_shards) {
+        if shard.is_none() {
+            *shard = Some(vec![0; shard_len]);
+        }
+    }
+
+    let mut shard_refs: SmallVec<[&mut [u8]; 16]> = SmallVec::new();
+    for (index, shard) in shards.iter_mut().enumerate() {
+        let shard = shard
+            .as_mut()
+            .ok_or_else(|| io::Error::other(format!("missing shard {index} after data reconstruction")))?;
+        if shard.len() != shard_len {
+            return Err(io::Error::other(format!(
+                "inconsistent shard length at index {index}: got {}, expected {}",
+                shard.len(),
+                shard_len
+            )));
+        }
+        shard_refs.push(shard.as_mut_slice());
+    }
+
+    encode(shard_refs)
 }
 
 /// Erasure coding utility for data reliability using Reed-Solomon codes.
@@ -392,7 +453,7 @@ impl Erasure {
         Ok(shards)
     }
 
-    /// Decode and reconstruct missing shards in-place.
+    /// Decode and reconstruct missing data shards in-place.
     ///
     /// # Arguments
     /// * `shards` - Mutable slice of optional shard data. Missing shards should be `None`.
@@ -400,6 +461,25 @@ impl Erasure {
     /// # Returns
     /// Ok if reconstruction succeeds, error otherwise.
     pub fn decode_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
+        if self.parity_shards > 0 {
+            if self.uses_legacy {
+                if let Some(encoder) = self.legacy_encoder.as_ref() {
+                    encoder.reconstruct_data(shards)?;
+                } else {
+                    warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
+                }
+            } else if let Some(encoder) = self.encoder.as_ref() {
+                encoder.reconstruct_data(shards)?;
+            } else {
+                warn!("parity_shards > 0, but encoder is None");
+            }
+        }
+
+        Ok(())
+    }
+
+    /// Decode and reconstruct missing data shards, then regenerate parity shards.
+    pub fn decode_data_and_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
         if self.parity_shards > 0 {
             if self.uses_legacy {
                 if let Some(encoder) = self.legacy_encoder.as_ref() {
@@ -534,6 +614,131 @@ mod tests {
 
     use super::*;
 
+    fn optional_shards(shards: &[Bytes]) -> Vec<Option<Vec<u8>>> {
+        shards.iter().map(|shard| Some(shard.to_vec())).collect()
+    }
+
+    #[test]
+    fn decode_data_keeps_missing_parity_shard_unreconstructed() {
+        let erasure = Erasure::new(2, 2, 64);
+        let data = b"read decode should not rebuild parity";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        let missing_parity = erasure.data_shards;
+        shards[missing_parity] = None;
+
+        erasure.decode_data(&mut shards).expect("decode should succeed");
+
+        assert!(shards[missing_parity].is_none(), "read decode should leave parity missing");
+        for index in 0..erasure.data_shards {
+            assert_eq!(
+                shards[index].as_deref(),
+                Some(encoded[index].as_ref()),
+                "data shard {index} should remain unchanged"
+            );
+        }
+    }
+
+    #[test]
+    fn legacy_decode_data_keeps_missing_parity_shard_unreconstructed() {
+        let erasure = Erasure::new_with_options(2, 2, 64, true);
+        let data = b"legacy read decode should not rebuild parity";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        let missing_parity = erasure.data_shards + 1;
+        shards[missing_parity] = None;
+
+        erasure.decode_data(&mut shards).expect("decode should succeed");
+
+        assert!(shards[missing_parity].is_none(), "legacy read decode should leave parity missing");
+        for index in 0..erasure.data_shards {
+            assert_eq!(
+                shards[index].as_deref(),
+                Some(encoded[index].as_ref()),
+                "legacy data shard {index} should remain unchanged"
+            );
+        }
+    }
+
+    #[test]
+    fn decode_data_and_parity_leaves_complete_shards_unchanged() {
+        let erasure = Erasure::new(4, 2, 128);
+        let data = b"complete shards should not be changed";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let original = optional_shards(&encoded);
+        let mut shards = original.clone();
+
+        erasure
+            .decode_data_and_parity(&mut shards)
+            .expect("decode should succeed without missing shards");
+
+        assert_eq!(shards, original);
+    }
+
+    #[test]
+    fn decode_data_and_parity_reconstructs_missing_parity_shard() {
+        let erasure = Erasure::new(2, 2, 64);
+        let data = b"parity shard must be rebuilt";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        shards[2] = None;
+
+        erasure
+            .decode_data_and_parity(&mut shards)
+            .expect("decode should rebuild parity");
+
+        for (index, shard) in shards.iter().enumerate() {
+            assert_eq!(
+                shard.as_deref(),
+                Some(encoded[index].as_ref()),
+                "shard {index} should match encoded source"
+            );
+        }
+    }
+
+    #[test]
+    fn decode_data_and_parity_reconstructs_missing_data_and_parity_shards() {
+        let erasure = Erasure::new(4, 2, 128);
+        let data = b"data and parity shards should both be reconstructed";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        shards[1] = None;
+        shards[4] = None;
+
+        erasure
+            .decode_data_and_parity(&mut shards)
+            .expect("decode should rebuild all missing shards");
+
+        for (index, shard) in shards.iter().enumerate() {
+            assert_eq!(
+                shard.as_deref(),
+                Some(encoded[index].as_ref()),
+                "shard {index} should match encoded source"
+            );
+        }
+    }
+
+    #[test]
+    fn legacy_decode_data_and_parity_reconstructs_missing_parity_shard() {
+        let erasure = Erasure::new_with_options(2, 2, 64, true);
+        let data = b"legacy parity shard must be rebuilt";
+        let encoded = erasure.encode_data(data).expect("encode should succeed");
+        let mut shards = optional_shards(&encoded);
+        shards[3] = None;
+
+        erasure
+            .decode_data_and_parity(&mut shards)
+            .expect("decode should rebuild parity");
+
+        for (index, shard) in shards.iter().enumerate() {
+            assert_eq!(
+                shard.as_deref(),
+                Some(encoded[index].as_ref()),
+                "shard {index} should match encoded source"
+            );
+        }
+    }
+
     #[test]
     fn test_shard_file_size_cases2() {
         let erasure = Erasure::new(12, 4, 1024 * 1024);