chore(deps): refresh workspace deps and linux fs_type gating (#3030)

* chore(deps): refresh workspace deps and linux fs_type gating

- refresh workspace dependency pins and lockfile updates

- remove now-unused crate dependency entries in multiple Cargo.toml files

- enable profiling export defaults in config and scripts/run.sh

- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings

* fix(utils): simplify fs_type linux gating

- keep fs_type module-level linux cfg in os::mod

- remove redundant linux cfg on get_fs_type and test module

* chore(deps): bump s3s git revision

- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5

- refresh Cargo.lock source entry for s3s

* chore(dev): allow mysql_async git source and env overrides

- allow mysql_async git source in deny.toml allow-git list

- make scripts/run.sh core env vars overrideable via existing shell env

* fix(utils): import get_fs_type in fs_type tests

- add explicit super::get_fs_type import in fs_type test module

- fix Linux E0425 unresolved function errors in unit tests

* chore(dev): tune run script observability defaults

- make profiling export env overrideable in scripts/run.sh

- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0

- update allow-git review window comments in deny.toml

* test(obs): stabilize profiling env alias tests

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,22 +1,33 @@
 ---
 name: rustfs-release-version-bump
-description: Prepare a RustFS release branch like PR #2957 by bumping versioned files, aligning release assets, running required verification, and finishing with commit, push, and gh PR creation. Use when publishing a new RustFS alpha, beta, or stable release.
+description: Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation.
 ---
 
 # RustFS Release Version Bump
 
-Use this skill when the task is to prepare a new RustFS version release branch following the pattern validated in PR `#2957`.
+Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
 
-## Read first
+Validated baseline: release pattern used in PR `#2957`.
 
-- Read `AGENTS.md`.
-- Read `.github/pull_request_template.md`.
-- Inspect the current branch diff against `origin/main`.
-- Do not assume every release file should change in the same way every time.
+## Required inputs
 
-## Scope validated by PR #2957
+- Exact target version, for example `1.0.0-beta.4`.
+- Delivery scope:
+- Local only (`edit/verify`).
+- Local + git (`commit/push`).
+- Full GitHub flow (`commit/push/PR`).
 
-The `1.0.0-beta.3` release branch updated these files:
+If target version is missing or ambiguous, stop and ask before editing.
+
+## Read before editing
+
+- `AGENTS.md` (root and nearest path-specific files).
+- `.github/pull_request_template.md`.
+- Current branch status and diff against `origin/main`.
+
+## Default release file scope
+
+Treat the following file list as the default checklist for each release bump:
 
 - `Cargo.toml`
 - `Cargo.lock`
@@ -26,78 +37,89 @@ The `1.0.0-beta.3` release branch updated these files:
 - `helm/rustfs/Chart.yaml`
 - `rustfs.spec`
 
-Treat this file list as the default checklist for future release bumps. Only drop a file if the repository's current release pattern clearly says it is no longer part of the publish flow.
+Only drop a file when the current repository release process clearly no longer requires it.
 
-## Workflow
+## Hard release policy
 
-1. Confirm release intent
-- Identify the target version string exactly, for example `1.0.0-beta.4`.
-- Check whether the user wants only local commit preparation or the full `commit + push + PR` flow.
-- Compare the branch against `origin/main` and isolate only release-related edits.
-- If the target version is not explicit, stop and ask for it before editing.
+- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
+- Helm chart version mapping follows `beta.N -> 0.N.0`.
+- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
+- Do not change these rules without explicit confirmation.
 
-2. Update core Rust workspace versions
-- Bump the workspace version in `Cargo.toml`.
-- Bump all internal workspace crate versions in `Cargo.toml`.
-- Ensure `Cargo.lock` reflects the same release version for workspace packages.
-- Re-read the diff and verify there are no partial version leftovers.
+## Step-by-step workflow
 
-3. Align release assets
-- Update versioned Docker examples in `README.md` and `README_ZH.md` using the `<version>` tag form, for example `rustfs/rustfs:1.0.0-beta.4`.
-- Update `flake.nix` package version.
-- Update `helm/rustfs/Chart.yaml` `appVersion`.
-- Update `helm/rustfs/Chart.yaml` `version` using the release-chart rule:
+1. Confirm intent and isolate scope
+- Confirm target version string exactly.
+- Confirm whether user requested local-only or full GitHub flow.
+- Inspect current branch and ensure only release-related files are touched for this task.
+
+2. Update workspace versions
+- Bump `[workspace.package].version` in `Cargo.toml`.
+- Bump internal workspace crate dependency versions in `Cargo.toml`.
+- Update `Cargo.lock` so workspace package versions match target version.
+- Re-scan for partial leftovers.
+
+3. Update release assets
+- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
+- `flake.nix`: update package version to target version.
+- `helm/rustfs/Chart.yaml`:
+- `appVersion` = target version.
+- `version` follows chart mapping rule, for example:
 - `1.0.0-beta.3` -> `0.3.0`
 - `1.0.0-beta.4` -> `0.4.0`
-- Follow the same pattern for later beta releases unless the repository rule changes.
-- Update `rustfs.spec` release metadata and changelog entry.
-- Set `rustfs.spec` `Release` to the prerelease suffix without the base version, for example `beta.3` for `1.0.0-beta.3`.
-- Keep the release asset changes in a separate commit from the core Rust workspace version bump when the branch contains both.
+- `rustfs.spec`:
+- Set `Release` to prerelease suffix (example `beta.4`).
+- Add/update top changelog entry with exact format:
+- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
+- `- Update RPM package to RustFS 1.0.0-beta.4`
+- Changelog identity and time must come from current environment:
+- `git config --get user.name`
+- `git config --get user.email`
+- `date '+%a %b %d %Y'`
+- Changelog version text must match target release version exactly.
 
-4. Stop and discuss before changing release policy
-- Ask before changing the established Docker tag style away from `<version>`.
-- Ask before changing the established Helm chart version mapping away from `beta.N -> 0.N.0`.
-- Ask before changing the established `rustfs.spec` `Release` rule away from the release suffix form such as `beta.N`.
-- Ask before widening the release scope beyond the files already validated in PR `#2957`.
+4. Verify before shipping
+- Run:
+- `cargo fmt --all`
+- `cargo fmt --all --check`
+- `make pre-commit`
+- If verification passes, run `cargo clean`.
+- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
 
-5. Verify before shipping
-- Run `make pre-commit`.
-- If verification succeeds, run `cargo clean` to remove generated build artifacts before wrapping up.
-- If `make pre-commit` fails, stop and return `BLOCKED`.
+5. Commit strategy
+- Preferred split when both parts changed:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
+- `chore(release): align release assets for <version>` for docs and packaging files.
+- If user asks for one commit, use one commit.
+- Stage only intended release files; do not include unrelated working tree changes.
 
-6. Commit structure
-- Prefer two commits when the change naturally splits:
-- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`
-- `chore(release): align release assets for <version>` for docs and packaging metadata
-- If the user asks for a single commit, follow that request.
+6. Push and PR
+- Push branch:
+- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
+- Create PR with template headings unchanged:
+- `gh pr create --base main --head <branch> --title ... --body-file ...`
+- PR title/body must be English.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands and any `BLOCKED` reason clearly.
 
-7. Push and PR
-- Push the release branch with `git push -u origin <branch>` or `git push` if upstream already exists.
-- Create the PR with `gh pr create --base main --head <branch> --title ... --body-file ...`.
-- Keep the PR title and body in English.
-- Keep the `.github/pull_request_template.md` headings exactly.
-
-## Ready-to-check commands
+## Recommended check commands
 
+- `git status --short --branch`
 - `git diff --name-only origin/main...HEAD`
 - `git diff --stat origin/main...HEAD`
+- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
 - `cargo clean`
-- `git status --short --branch`
 
-## Output expectations
+## Output contract
 
-When using this skill, return:
+When using this skill, always report:
 
-- The files changed for the release bump
-- Any uncertainty that needs user confirmation before editing
-- Verification status
-- Commit messages used
-- Push status and PR URL when the GitHub flow was requested
-
-## Established release policy
-
-- Docs use Docker tags in `<version>` form, not `v<version>`.
-- `helm/rustfs/Chart.yaml` `version` follows `beta.N -> 0.N.0` based on the current release policy.
-- `rustfs.spec` `Release` follows the release suffix form, for example `beta.3`.
-- If any of these rules need to change in the future, pause and confirm before editing.
+- Target version.
+- Files changed.
+- Any assumptions or uncertainties requiring confirmation.
+- Verification result (`PASSED` or `BLOCKED`) with key evidence.
+- Commit message(s) used.
+- Push status and PR URL when GitHub flow is requested.

.agents/skills/security-advisory-lessons/SKILL.md

@@ -91,7 +91,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
 - Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
 - Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
-- License/version-like metadata endpoints should expose only coarse public data unless authenticated.
+- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
 
 ### Profiling, debug, and health endpoints
 - Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -20,7 +20,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
 - `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
 - `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-- `GHSA-x5xv-223c-8vm7`: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
+- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
 
 ### IAM import, service accounts, and privilege boundaries
 

.config/make/help.mak

@@ -4,7 +4,7 @@
 .PHONY: help
 help: ## Shows This Help Menu
 	echo -e "$$HEADER"
-	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} ; {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}' | sed -e 's/\[36m##/\n[32m##/'
+	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
 
 .PHONY: help-build
 help-build: ## Shows RustFS build help

.docker/compose/README.md

@@ -54,6 +54,46 @@ Strict mode is available when you explicitly want `/health/ready == 200` as the
 WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
 ```
 
+### Profiling + Trace Validation
+
+The profiling-focused 4-node compose keeps profiling enabled and points RustFS
+to an OTLP/HTTP collector endpoint:
+
+```bash
+docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
+```
+
+Important behavior notes:
+
+- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
+  traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
+- Startup usually produces logs and metrics first. That does not guarantee
+  visible traces yet.
+- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
+- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
+  many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
+  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
+
+Minimal trace verification flow:
+
+```bash
+# 1. Start the profiling compose with richer span visibility.
+RUSTFS_OBS_LOGGER_LEVEL=debug \
+docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
+
+# 2. Generate real request traffic after startup.
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 3. Then inspect Tempo or Jaeger.
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+If logs and metrics are present but traces are sparse, the most common cause is
+"no real request traffic yet" or "`info` level filtered nested spans", not an
+OTLP routing failure.
+
 ### (Deprecated) Minimal Observability
 
 ```bash

.docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml

@@ -0,0 +1,236 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Profiling-first 4-node local-build compose.
+#
+# Goals:
+# - force linux/amd64 runtime/build on Apple Silicon hosts;
+# - enable RustFS built-in CPU profiling;
+# - keep all tuning knobs host-overridable via env.
+#
+# Observability notes:
+# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
+#   `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
+# - Logs and metrics usually appear during startup. Traces mainly appear after
+#   real HTTP/S3/gRPC requests create spans.
+# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
+#   filters many `debug`-level nested spans. Use `debug` when validating trace
+#   richness rather than collector reachability.
+
+services:
+  node1:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node1_data_0:/data/rustfs0
+      - node1_data_1:/data/rustfs1
+      - node1_data_2:/data/rustfs2
+      - node1_data_3:/data/rustfs3
+    ports:
+      - "9000:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node2:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node2_data_0:/data/rustfs0
+      - node2_data_1:/data/rustfs1
+      - node2_data_2:/data/rustfs2
+      - node2_data_3:/data/rustfs3
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node3:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node3_data_0:/data/rustfs0
+      - node3_data_1:/data/rustfs1
+      - node3_data_2:/data/rustfs2
+      - node3_data_3:/data/rustfs3
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node4:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node4
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node4_data_0:/data/rustfs0
+      - node4_data_1:/data/rustfs1
+      - node4_data_2:/data/rustfs2
+      - node4_data_3:/data/rustfs3
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-cluster-net
+
+volumes:
+  node1_data_0:
+  node1_data_1:
+  node1_data_2:
+  node1_data_3:
+  node2_data_0:
+  node2_data_1:
+  node2_data_2:
+  node2_data_3:
+  node3_data_0:
+  node3_data_1:
+  node3_data_2:
+  node3_data_3:
+  node4_data_0:
+  node4_data_1:
+  node4_data_2:
+  node4_data_3:
+
+networks:
+  rustfs-cluster-net:
+    driver: bridge

.docker/compose/docker-compose.cluster.local-build.yml

@@ -23,8 +23,8 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
@@ -50,8 +50,8 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
@@ -77,8 +77,8 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
@@ -104,8 +104,8 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-latency-stable.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-throughput-max.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60

.docker/compose/docker-compose.cluster.perf-round1-linux.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40

.docker/compose/docker-compose.cluster.perf-round1.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30

.docker/compose/docker-compose.cluster.yaml

@@ -21,8 +21,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
+      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
     platform: linux/amd64
     ports:
       - "9000:9000" # Map port 9001 of the host to port 9000 of the container
@@ -38,8 +38,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
+      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
     platform: linux/amd64
     ports:
       - "9001:9000" # Map port 9002 of the host to port 9000 of the container
@@ -55,8 +55,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
+      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
     platform: linux/amd64
     ports:
       - "9002:9000" # Map port 9003 of the host to port 9000 of the container
@@ -72,8 +72,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=rustfs-cluster-admin
+      - RUSTFS_SECRET_KEY=rustfs-cluster-secret
     platform: linux/amd64
     ports:
       - "9003:9000" # Map port 9004 of the host to port 9000 of the container

.docker/observability/README.md

@@ -90,6 +90,54 @@ docker compose down -v
 - **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
 - **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
 
+### Verifying RustFS Traces
+
+When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
+OTLP/HTTP base URL, for example:
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+RustFS automatically expands that base URL to:
+
+- `/v1/traces`
+- `/v1/metrics`
+- `/v1/logs`
+
+Important behavior notes:
+
+- Logs and metrics usually appear during startup, so seeing those two signals
+  first is expected.
+- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
+  startup, because request-path spans are created on demand.
+- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
+  many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
+  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
+
+Minimal validation flow:
+
+```bash
+# 1. Start this observability stack.
+docker compose up -d
+
+# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+export RUSTFS_OBS_LOGGER_LEVEL=debug
+
+# 3. Generate real request traffic.
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 4. Inspect Grafana or Jaeger.
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+If logs and metrics are present but traces are sparse, the most common cause is
+"no real request traffic yet" or "`info` level filtered nested spans", not an
+OTLP routing failure.
+
 ## Troubleshooting
 
 - **Service Health**: Check the health of services using `docker compose ps`.

.docker/observability/README_ZH.md

@@ -90,6 +90,50 @@ docker compose down -v
 -   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
 -   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
 
+### 验证 RustFS Trace
+
+当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
+OTLP/HTTP 的基础 URL，例如：
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+RustFS 会自动在该基础 URL 后补全：
+
+- `/v1/traces`
+- `/v1/metrics`
+- `/v1/logs`
+
+需要注意：
+
+- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
+- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
+- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
+  如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
+
+最小验证流程：
+
+```bash
+# 1. 启动本目录下的可观测性技术栈。
+docker compose up -d
+
+# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+export RUSTFS_OBS_LOGGER_LEVEL=debug
+
+# 3. 产生真实请求流量。
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 4. 到 Grafana 或 Jaeger 中检查。
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
+“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
+
 ## 故障排除
 
 -   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。

.docker/observability/grafana/dashboards/rustfs.json

@@ -9037,13 +9037,13 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "{__name__=~\"rustfs_(notification|audit)_.*\",job=~\"$job\"}",
+          "expr": "{__name__=~\"rustfs_(notification|audit|log_chain)_.*\",job=~\"$job\"}",
           "legendFormat": "{{__name__}}",
           "range": true,
           "refId": "A"
         }
       ],
-      "title": "Notification / Audit (All)",
+      "title": "Notification / Audit / LogChain (All)",
       "type": "timeseries"
     },
     {

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -38,3 +38,16 @@ groups:
         expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
       - record: rustfs:scanner_cycles_success:rate5m
         expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
+
+      - record: rustfs:log_chain_op_event_mismatch:rate5m
+        expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
+
+      - alert: RustFSLogChainOpEventMismatchDetected
+        expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
+        for: 10m
+        labels:
+          severity: warning
+          component: s3-log-chain
+        annotations:
+          summary: "RustFS log-chain op/event mismatch detected"
+          description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."

.github/actions/setup/action.yml

@@ -53,18 +53,19 @@ runs:
           musl-tools \
           build-essential \
           pkg-config \
-          libssl-dev
+          libssl-dev \
+          protobuf-compiler
 
     - name: Install protoc
-      uses: arduino/setup-protoc@v3
+      uses: rustfs/setup-protoc@v3.0.1
       with:
-        version: "33.1"
-        repo-token: ${{ inputs.github-token }}
+        version: "29.3"
+        repo-token: ${{ github.token }}
 
     - name: Install flatc
-      uses: Nugine/setup-flatc@v1.2.4
+      uses: Nugine/setup-flatc@v1
       with:
-        version: "25.9.23"
+        version: "25.12.19"
 
     - name: Install Rust toolchain
       uses: dtolnay/rust-toolchain@stable

.github/workflows/build.yml

@@ -169,7 +169,7 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-15-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -234,8 +234,7 @@ jobs:
         run: |
           mkdir -p ./rustfs/static
           if [[ "${{ matrix.platform }}" == "windows" ]]; then
-            curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300
-            if [[ $? -eq 0 ]]; then
+            if curl.exe --fail -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300; then
               unzip -o console.zip -d ./rustfs/static
               rm console.zip
             else
@@ -244,9 +243,8 @@ jobs:
             fi
           else
             chmod +w ./rustfs/static/LICENSE || true
-            curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-              -o console.zip --retry 3 --retry-delay 5 --max-time 300
-            if [[ $? -eq 0 ]]; then
+            if curl --fail -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
+              -o console.zip --retry 3 --retry-delay 5 --max-time 300; then
               unzip -o console.zip -d ./rustfs/static
               rm console.zip
             else

.github/workflows/ci.yml

@@ -204,7 +204,7 @@ jobs:
         with:
           tool: s3s-e2e
           git: https://github.com/s3s-project/s3s.git
-          rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
+          rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
 
       - name: Run end-to-end tests
         run: |

.github/workflows/e2e-s3tests.yml

@@ -40,8 +40,8 @@ on:
 
 env:
   # main user
-  S3_ACCESS_KEY: rustfsadmin
-  S3_SECRET_KEY: rustfsadmin
+  S3_ACCESS_KEY: rustfs-ci-admin
+  S3_SECRET_KEY: rustfs-ci-secret
   # alt user (must be different from main for many s3-tests)
   S3_ALT_ACCESS_KEY: rustfsalt
   S3_ALT_SECRET_KEY: rustfsalt

.github/workflows/nix-flake-update.yml

@@ -31,7 +31,9 @@ jobs:
   update-flake:
     name: Update flake.lock
     runs-on: ubuntu-latest
-    timeout-minutes: 45
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -39,6 +41,9 @@ jobs:
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@v3
 
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@v3.20.0
+
       - name: Check Nix flake inputs
         uses: DeterminateSystems/flake-checker-action@v12
 
@@ -56,7 +61,7 @@ jobs:
             nix
             automated
           commit-msg: "chore(deps): update flake.lock"
-          pr-reviewers: houseme, overtrue, majinghe
+          pr-reviewers: overtrue, majinghe
           token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
 
       - name: Log PR details

.github/workflows/nix.yml

@@ -43,7 +43,9 @@ jobs:
   nix-validation:
     name: Nix Build & Check
     runs-on: ubuntu-latest
-    timeout-minutes: 45
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6

.github/workflows/performance.yml

@@ -86,7 +86,7 @@ jobs:
 
       - name: Build with profiling optimizations
         run: |
-          RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off" \
+          RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off --cfg tokio_unstable" \
           cargo +nightly build --profile profiling -p rustfs --bins
 
       - name: Run performance profiling

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 ## 📋 Code Quality Requirements
 
-For instructions on setting up and running the local development environment, please see [Development Guide](docs/DEVELOPMENT.md).
+This guide covers the local development environment and the checks expected before contributing.
 
 ### 🔧 Code Formatting Rules
 

Cargo.toml

@@ -15,10 +15,10 @@
 [workspace]
 members = [
     "rustfs", # Core file system implementation
-    "crates/appauth", # Application authentication and authorization
     "crates/audit", # Audit target management system with multi-target fan-out
     "crates/checksums", # client checksums
     "crates/common", # Shared utilities and data structures
+    "crates/data-usage", # Shared data usage models and algorithms
     "crates/config", # Configuration management
     "crates/credentials", # Credential management system
     "crates/crypto", # Cryptography and security features
@@ -39,7 +39,8 @@ members = [
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
-    "crates/s3-common", # Common utilities and data structures for S3 compatibility
+    "crates/s3-types", # S3 event type definitions
+    "crates/s3-ops", # S3 operation definitions and mapping
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
     "crates/scanner", # Scanner for data integrity checks and health monitoring
@@ -47,7 +48,6 @@ members = [
     "crates/targets", # Target-specific configurations and utilities
     "crates/trusted-proxies", # Trusted proxies management
     "crates/utils", # Utility functions and helpers
-    "crates/workers", # Worker thread pools and task scheduling
     "crates/io-metrics", # Zero-copy metrics collection for performance analysis
     "crates/io-core", # Zero-copy core reader and writer implementations
     "crates/zip", # ZIP file handling and compression
@@ -59,7 +59,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.95.0"
-version = "1.0.0-beta.3"
+version = "1.0.0-beta.4"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -76,46 +76,46 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.3" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.3" }
-rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.3" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.3" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.3" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.3" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.3" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.3" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.3" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.3" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.3" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.3" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.3" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.3" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.3" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.3" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.3" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.3" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.3" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.3" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.3" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.3" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.3" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.3" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.3" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.3" }
-rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.3" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.3" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.3" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.3" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.3" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.3" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.3" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.3" }
-rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.3" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.3" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.4" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.4" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.4" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.4" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.4" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.4" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.4" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.4" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.4" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.4" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.4" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.4" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.4" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.4" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.4" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.4" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.4" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.4" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.4" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.4" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.4" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.4" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.4" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.4" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.4" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.4" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.4" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.4" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.4" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.4" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.4" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.4" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.4" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.4" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.4" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.4" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "98d3d8067efdf97d3e93cdca7b9231753c904aca" }
+mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "2bad388283bc3ce48801fc2ffcd22445eb6f3d24" }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
@@ -126,7 +126,7 @@ futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
 pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
-lapin = { version = "4.7.2", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
+lapin = { version = "4.7.4", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
 hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
@@ -145,7 +145,7 @@ tonic = { version = "0.14.6", features = ["gzip"] }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3", features = ["timeout"] }
-tower-http = { version = "0.6.10", features = ["cors"] }
+tower-http = { version = "0.6.11", features = ["cors"] }
 
 # Serialization and Data Formats
 bytes = { version = "1.11.1", features = ["serde"] }
@@ -154,7 +154,7 @@ byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
 prost = "0.14.3"
-quick-xml = "0.39.4"
+quick-xml = "0.40.1"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
@@ -166,9 +166,9 @@ aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
 argon2 = { version = "0.6.0-rc.8" }
 blake2 = "0.11.0-rc.6"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
-crc-fast = "1.9.0"
+crc-fast = "1.10.0"
 hmac = { version = "0.13.0" }
-jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
+jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
 openidconnect = { version = "4.0", default-features = false }
 pbkdf2 = "0.13.0"
 rsa = { version = "0.10.0-rc.18" }
@@ -194,15 +194,14 @@ tokio-postgres-rustls = "0.13"
 # Utilities and Tools
 anyhow = "1.0.102"
 arc-swap = "1.9.1"
-astral-tokio-tar = "0.6.1"
+astral-tokio-tar = "0.6.2"
 atoi = "2.0.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.8.16" }
 aws-credential-types = { version = "1.2.14" }
 aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.7" }
-backtrace = "0.3.76"
+aws-smithy-types = { version = "1.4.8" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.2"
@@ -216,7 +215,7 @@ crossbeam-deque = "0.8.6"
 crossbeam-utils = "0.8.21"
 datafusion = "53.1.0"
 derive_builder = "0.20.2"
-enumset = "1.1.12"
+enumset = "1.1.13"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
@@ -230,7 +229,7 @@ ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
 libc = "0.2.186"
 libsystemd = "0.7.2"
-local-ip-address = "0.6.12"
+local-ip-address = "0.6.13"
 memmap2 = "0.9.10"
 lz4 = "1.28.1"
 matchit = "0.9.2"
@@ -254,12 +253,12 @@ rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.12.3" }
-rumqttc = { package = "rumqttc-next", version = "0.33.0", features = ["websocket"] }
+rumqttc = { package = "rumqttc-next", version = "0.33.1", features = ["websocket"] }
 redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
-s3s = { git = "https://github.com/rustfs/s3s", rev = "a3b16608df35aaeed8fff08b4988d03f4ca9445b", features = ["minio"] }
+s3s = { git = "https://github.com/rustfs/s3s", rev = "507e1312b211c3ddc214b03875d6fabd15d22ed5", features = ["minio"] }
 serial_test = "3.4.0"
 shadow-rs = { version = "2.0.0", default-features = false }
 siphasher = "1.0.3"
@@ -267,9 +266,9 @@ smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
 snafu = "0.9.0"
 snap = "1.1.1"
-starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
+starshard = { version = "2.2.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.39.0"
+sysinfo = "0.39.2"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -277,7 +276,7 @@ thiserror = "2.0.18"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
-tracing-opentelemetry = "0.32.1"
+tracing-opentelemetry = { version = "0.33" }
 tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
@@ -292,22 +291,22 @@ zip = "8.6.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
-metrics = "0.24.5"
+metrics = "0.24.6"
 dial9-tokio-telemetry = "0.3"
-opentelemetry = { version = "0.31.0" }
-opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
-opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rustls"] }
-opentelemetry_sdk = { version = "0.31.0" }
-opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
-opentelemetry-stdout = { version = "0.31.0" }
-pyroscope = { version = "2.0.3", features = ["backend-pprof-rs"] }
+opentelemetry = { version = "0.32.0" }
+opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
+opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
+opentelemetry_sdk = { version = "0.32.0", features = ["rt-tokio"] }
+opentelemetry-semantic-conventions = { version = "0.32.0", features = ["semconv_experimental"] }
+opentelemetry-stdout = { version = "0.32.0" }
+pyroscope = { version = "2.0.4", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
 suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
-rcgen = "0.14.7"
-russh = "0.60.2"
+rcgen = "0.14.8"
+russh = { version = "0.60.3", git = "https://github.com/Eugeny/russh", rev = "fc6e3ab4cd4338e94ae64e17aeed2acee9335e6b" }
 russh-sftp = "2.1.2"
 
 # WebDAV
@@ -324,17 +323,11 @@ jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
 # pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
 # Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
-pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
+pprof = { package = "pprof-pyroscope-fork", version = "0.1500.4", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs"]
 
-[patch.crates-io]
-# Pinned to the simon-escapecode/russh fork carrying the upstream fix at
-# https://github.com/Eugeny/russh/pull/702. Drops out when a russh release
-# containing the fix is published.
-russh = { git = "https://github.com/simon-escapecode/russh", rev = "5cac2ed84945f9b80a52b673e058f2032bbe98ec" }
-
 [profile.release]
 opt-level = 3
 

Dockerfile.decommission-local

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM rust:1.93-trixie
+FROM rust:1.95-trixie
 
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \

Dockerfile.glibc

@@ -105,8 +105,6 @@ RUN groupadd -g 10001 rustfs && \
 
 ENV RUSTFS_ADDRESS=":9000" \
     RUSTFS_CONSOLE_ADDRESS=":9001" \
-    RUSTFS_ACCESS_KEY="rustfsadmin" \
-    RUSTFS_SECRET_KEY="rustfsadmin" \
     RUSTFS_CONSOLE_ENABLE="true" \
     RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
     RUSTFS_VOLUMES="/data" \

Dockerfile.source

@@ -31,7 +31,7 @@ ARG TARGETARCH
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.93-trixie AS builder
+FROM rust:1.95-trixie AS builder
 
 # Re-declare args after FROM
 ARG TARGETPLATFORM

INTERNODE_DATA_TRANSPORT_RFC.md

@@ -0,0 +1,400 @@
+# RFC: Pluggable Internode Data Transport
+
+> Status: draft
+> Last updated: 2026-05-19
+> Scope: internode data-path analysis, benchmark baseline, and transport boundary
+
+## Summary
+
+RustFS does not currently include RDMA, RoCE, InfiniBand, DPU, BlueField/DOCA,
+DPDK, SPDK, or SmartNIC offload support. The current distributed internode
+paths use TCP-based HTTP/gRPC transports:
+
+- `tonic` gRPC `NodeService` for most control, metadata, lock, health, and
+  peer operations.
+- HTTP streaming routes under `/rustfs/rpc/` for remote disk file streams.
+
+RDMA/RoCE is still a plausible future optimization for large internode disk
+data transfers, but it should not replace the whole internode RPC surface.
+The correct first step is to isolate the data plane, establish a TCP baseline,
+and introduce a pluggable transport boundary only around high-volume streams.
+
+## Goals
+
+- Document the current internode control plane and data plane.
+- Identify the existing transfer paths that could benefit from a future
+  high-throughput backend.
+- Define the minimum benchmark baseline required before transport changes.
+- Sketch a pluggable transport boundary that preserves the current TCP/HTTP
+  behavior as the default backend.
+- Reserve explicit boundaries for future RDMA/RoCE/InfiniBand work without
+  committing RustFS to a specific vendor stack.
+
+## Non-Goals
+
+- Implement RDMA, RoCE, InfiniBand, DPU, DOCA, DPDK, SPDK, or SmartNIC support.
+- Replace `tonic` gRPC for control-plane RPCs.
+- Redesign erasure coding, quorum handling, disk health tracking, or object
+  correctness semantics.
+- Require RDMA-capable hardware for default development, CI, or ordinary
+  RustFS deployments.
+
+## Current Internode Architecture
+
+### Server-side entry points
+
+The main HTTP server builds a hybrid service per connection:
+
+- `rustfs/src/server/http.rs` wires a `NodeServiceServer` for gRPC.
+- `rustfs/src/storage/rpc/InternodeRpcService` intercepts HTTP paths under
+  `/rustfs/rpc/`.
+- Other HTTP/S3 traffic continues through the normal S3 service.
+
+Compression logic already treats `/rustfs/rpc/` and `/rustfs/peer/` as internode
+RPC paths and skips normal response compression for them.
+
+### gRPC channel management
+
+`crates/protos/src/lib.rs` creates internode gRPC channels with `tonic`
+`Endpoint`:
+
+- connect timeout
+- TCP keepalive
+- HTTP/2 keepalive interval and timeout
+- request timeout
+- optional TLS configuration
+- global channel caching and failed-connection eviction
+
+This confirms the current gRPC transport is TCP/HTTP2-based.
+
+### NodeService layout
+
+`crates/protos/src/node.proto` defines one `NodeService` that mixes several
+classes of RPCs:
+
+- meta service: bucket and metadata operations
+- disk service: local/remote disk operations
+- lock service: distributed lock operations
+- peer rest service: node health, metrics, IAM/policy reload, rebalance,
+  profiling, events, and admin-style operations
+
+The service layout is practical today, but it is too broad to become an RDMA
+surface. A future high-throughput transport should target only disk data
+streams and keep this gRPC service as the control plane.
+
+## Control Plane vs Data Plane
+
+### Control plane
+
+These paths carry coordination, metadata, health, and administrative state.
+They should remain on gRPC/TCP:
+
+| Area | Client/server code | Examples | Notes |
+| --- | --- | --- | --- |
+| Bucket peer ops | `crates/ecstore/src/rpc/peer_s3_client.rs`, `rustfs/src/storage/rpc/bucket.rs` | `MakeBucket`, `ListBucket`, `DeleteBucket`, `GetBucketInfo`, `HealBucket` | Small metadata/control payloads. |
+| Locking | `crates/ecstore/src/rpc/remote_locker.rs`, `rustfs/src/storage/rpc/lock.rs` | `Lock`, `UnLock`, `Refresh`, batch lock/unlock | Latency-sensitive but not bulk data; correctness and timeout semantics matter more than transport bandwidth. |
+| Peer/admin state | `crates/ecstore/src/rpc/peer_rest_client.rs`, `rustfs/src/storage/rpc/health.rs`, `metrics.rs`, `event.rs` | `LocalStorageInfo`, `ServerInfo`, `GetMetrics`, `GetLiveEvents`, reload APIs, rebalance APIs | Operational control plane. |
+| Disk metadata/control | `crates/ecstore/src/rpc/remote_disk.rs`, `rustfs/src/storage/rpc/disk.rs` | `DiskInfo`, `ReadXL`, `ReadVersion`, `ReadMetadata`, `WriteMetadata`, `RenameFile`, `RenamePart`, `Delete*`, `VerifyFile`, `CheckParts` | Usually metadata, integrity checks, or namespace mutations. |
+| Connection health | `RemoteDisk`, `RemotePeerS3Client`, `PeerRestClient` | TCP connectivity probes and fault/recovery state | Must remain available even if an optional data backend is unavailable. |
+
+### Data plane candidates
+
+These paths move object shard bytes or stream potentially large disk data and
+are the only reasonable first candidates for a pluggable transport.
+
+| Priority | Path | Current client | Current server | Current transport | Why it matters |
+| --- | --- | --- | --- | --- | --- |
+| P0 | `read_file_stream` | `RemoteDisk::read_file_stream` | `handle_read_file` in `http_service.rs` | HTTP `GET /rustfs/rpc/read_file_stream` with a streaming response body | Main remote disk read stream used by bitrot readers and erasure reads. |
+| P0 | `put_file_stream` | `RemoteDisk::create_file` and `RemoteDisk::append_file` | `handle_put_file` in `http_service.rs` | HTTP `PUT /rustfs/rpc/put_file_stream` with a streaming request body | Main remote disk write stream used by bitrot writers and erasure writes. |
+| P1 | `walk_dir` | `RemoteDisk::walk_dir` | `handle_walk_dir` in `http_service.rs` | HTTP `GET /rustfs/rpc/walk_dir` with a streamed metadata listing | Can be high-volume during scans/healing, but it is metadata-oriented rather than object byte data. |
+| P1 | `ReadAll` / `WriteAll` | `RemoteDisk::read_all` / `write_all` | gRPC unary disk handlers | gRPC unary `bytes` payload | Moves bytes today, but should be measured before treating it as a high-throughput data path. |
+| P2 | proto `WriteStream` / `ReadAt` | currently not used | currently returns unimplemented | gRPC streaming definitions exist but are not implemented | Possible future API shape, not a current production path. |
+
+## Current Object Write Path
+
+For object PUTs in distributed erasure mode, the relevant flow is:
+
+1. Upper storage layers prepare object data and erasure metadata.
+2. `SetDisks` selects local and remote disks.
+3. `create_bitrot_writer` calls `disk.create_file(...)` for each shard writer.
+4. For a remote disk, `RemoteDisk::create_file` returns an `HttpWriter`.
+5. `HttpWriter` sends an HTTP `PUT` to `/rustfs/rpc/put_file_stream`.
+6. The remote node's `handle_put_file` opens the local file writer and copies
+   incoming body chunks into it.
+7. `Erasure::encode` writes shards through `MultiWriter` to all selected
+   writers while enforcing write quorum.
+
+This is the primary write data-plane candidate.
+
+## Current Object Read Path
+
+For object GETs and repair reads in distributed erasure mode, the relevant flow is:
+
+1. `SetDisks` prepares shard readers for the selected disks.
+2. `create_bitrot_reader` uses local zero-copy only when `disk.is_local()`.
+3. For a remote disk, it calls `disk.read_file_stream(...)`.
+4. `RemoteDisk::read_file_stream` returns an `HttpReader`.
+5. `HttpReader` sends an HTTP `GET` to `/rustfs/rpc/read_file_stream`.
+6. The remote node's `handle_read_file` opens the local disk stream and returns
+   it as an HTTP streaming body.
+7. The erasure decoder reads from the shard streams and reconstructs the object.
+
+This is the primary read data-plane candidate.
+
+## Existing Metrics and Benchmark Surface
+
+RustFS already has coarse internode metrics in `crates/common/src/internode_metrics.rs`:
+
+- sent bytes
+- received bytes
+- outgoing requests
+- incoming requests
+- errors
+- dial errors
+- average dial time
+
+These metrics are useful as a starting point, but they are not enough for a
+transport RFC. A transport benchmark needs route-level and operation-level
+measurements for at least:
+
+- `read_file_stream`
+- `put_file_stream`
+- `walk_dir`
+- gRPC `ReadAll` / `WriteAll`
+- gRPC control-plane request volume
+
+Existing benchmark assets:
+
+- `scripts/run_object_batch_bench.sh`
+- `scripts/run_object_batch_bench_enhanced.sh`
+- `scripts/run_object_batch_bench_abc.sh`
+- `scripts/run_four_node_cluster_failover_bench.sh`
+- `scripts/run_internode_transport_baseline.sh` (scenario matrix wrapper for local vs distributed TCP baseline artifacts)
+- Criterion benches under `crates/ecstore/benches/`
+
+These mostly cover S3/object workload or erasure coding performance. They do
+not yet isolate internode transport cost.
+
+## Required TCP Baseline
+
+Before adding a transport abstraction or any RDMA backend, collect a baseline
+for the current TCP/HTTP/gRPC implementation.
+
+### Topology
+
+Minimum:
+
+- 1-node local erasure deployment, to measure local disk and erasure overhead.
+- 4-node distributed erasure deployment, to measure internode overhead.
+
+Preferred:
+
+- Same host count and disk layout for every run.
+- Dedicated network interface or isolated VLAN.
+- Fixed CPU governor and no unrelated background load.
+- Recorded kernel version, NIC model, MTU, RustFS commit, Rust toolchain, and
+  benchmark tool versions.
+
+### Workloads
+
+| Workload | Sizes | Concurrency | Main signal |
+| --- | --- | --- | --- |
+| S3 PUT | 4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB | 1, 16, 64, 128 | End-to-end write throughput and tail latency. |
+| S3 GET | 4 KiB, 1 MiB, 16 MiB, 128 MiB, 1 GiB | 1, 16, 64, 128 | End-to-end read throughput and tail latency. |
+| Remote disk stream read | shard-sized ranges from `read_file_stream` | 1, 16, 64 | Isolated internode read path. |
+| Remote disk stream write | shard-sized writes through `put_file_stream` | 1, 16, 64 | Isolated internode write path. |
+| Healing / repair | missing disk or missing shard scenario | controlled | Rebuild throughput and read/write amplification. |
+| Scanner walk | large bucket/object namespace | controlled | Metadata streaming pressure, not primary RDMA target. |
+
+### Measurements
+
+Collect:
+
+- throughput in bytes/s and objects/s
+- p50, p95, p99, and max latency
+- CPU utilization per process and per core
+- memory RSS and allocation pressure where available
+- `rustfs_system_network_internode_*` metrics
+- TCP retransmits, socket errors, and NIC throughput
+- disk throughput and utilization
+- failure/retry/fallback counts
+
+The baseline should produce a machine-readable artifact, for example
+`target/bench/internode-transport/<timestamp>/summary.csv`, plus the exact
+commands and configuration used.
+
+### Baseline runner entry point
+
+Use `scripts/run_internode_transport_baseline.sh` to execute a reproducible
+S3 PUT/GET matrix against `local` and `distributed` scenarios and export:
+
+- `summary.csv` (throughput/latency summary per workload and object size)
+- `internode_metric_deltas.csv` (operation-level internode metric deltas when
+  `--metrics-url` is provided)
+
+## Transport Abstraction Proposal
+
+### Design principle
+
+Keep `NodeService` as the control plane. Introduce a separate data transport
+only below `RemoteDisk`, where remote disk byte streams are opened today.
+
+The first implementation should be a no-behavior-change TCP/HTTP backend that
+wraps the current `HttpReader`, `HttpWriter`, and `/rustfs/rpc/*` handlers.
+Only after that wrapper is benchmarked should an experimental RDMA/RoCE backend
+be considered.
+
+### Candidate boundary
+
+The narrowest useful boundary is remote disk stream transfer:
+
+```rust
+#[async_trait::async_trait]
+pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
+    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
+    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
+    async fn walk_dir(&self, request: WalkDirStreamRequest, writer: &mut dyn AsyncWrite) -> Result<()>;
+    fn capabilities(&self) -> InternodeTransportCapabilities;
+}
+```
+
+Initial request fields should mirror the current HTTP query parameters:
+
+- peer endpoint
+- disk reference
+- volume
+- path
+- offset
+- length
+- append/create mode
+- expected size
+- auth or transfer token material
+
+The initial TCP backend can keep the current signed HTTP URLs internally.
+
+### Integration point
+
+`RemoteDisk` should delegate only these methods to the data transport:
+
+- `read_file_stream`
+- `read_file_zero_copy` as a wrapper over `read_file_stream` unless the backend
+  supports a stronger zero-copy API
+- `append_file`
+- `create_file`
+- optionally `walk_dir`
+
+All other `RemoteDisk` methods should continue using the current gRPC client
+until measurements prove otherwise.
+
+### Capability model
+
+Avoid hard-coding RDMA assumptions into the generic interface. Use capabilities:
+
+- stream read
+- stream write
+- bounded range read
+- bidirectional streaming
+- registered memory support
+- scatter/gather support
+- zero-copy receive into caller-owned buffers
+- authenticated out-of-band transfer
+- transport fallback support
+
+The first TCP backend should report only capabilities that it actually provides.
+
+## TCP Fallback Requirements
+
+TCP/HTTP/gRPC must remain the default and required backend.
+
+Fallback rules:
+
+- If no explicit data transport is configured, use the current TCP/HTTP
+  implementation.
+- If an experimental backend fails initialization, either fail fast with a clear
+  error or fall back to TCP only when the configured policy allows fallback.
+- Runtime fallback must preserve object correctness and quorum semantics.
+- Fallback events must be logged and counted in metrics.
+- CI and local development must not require RDMA-capable hardware.
+
+Suggested future configuration shape:
+
+```text
+RUSTFS_INTERNODE_DATA_TRANSPORT=tcp
+RUSTFS_INTERNODE_DATA_TRANSPORT_FALLBACK=tcp
+```
+
+Do not add these settings until there is an implementation PR that uses them.
+
+## Future RDMA/RoCE/InfiniBand Boundary
+
+A future RDMA backend should be experimental and feature-gated. It should be
+designed as an optional data-plane backend, not as a replacement for the gRPC
+control plane.
+
+Required design areas:
+
+- peer capability discovery over the existing gRPC control plane
+- connection management and health mapping into existing disk fault handling
+- memory registration lifecycle and registration cache
+- buffer ownership, pinning, alignment, and lifetime rules
+- scatter/gather behavior for erasure shards
+- authentication and authorization for out-of-band data transfers
+- encryption/TLS-equivalent story or a documented deployment boundary
+- timeout, cancellation, retry, and fallback behavior
+- metrics for registration cost, transfer latency, bytes, queue depth, retries,
+  fallback, and errors
+- hardware and kernel compatibility matrix
+
+The first RDMA prototype should target `read_file_stream` and `put_file_stream`
+only. `walk_dir`, metadata RPCs, locks, admin RPCs, and bucket coordination
+should remain on gRPC unless a later benchmark identifies a specific bottleneck.
+
+## DPU, DOCA, DPDK, SPDK, and SmartNIC Notes
+
+These technologies should not drive the first abstraction:
+
+- DPU/BlueField/DOCA may become relevant for TLS, checksum, compression, or
+  storage/network offload, but they are vendor- and deployment-specific.
+- DPDK is a poor first fit because RustFS is currently an HTTP/S3 object store
+  and does not have a custom packet data plane.
+- SPDK may be relevant only if RustFS adds a raw block or NVMe-oriented local
+  storage backend. The current disk model is filesystem-based.
+- SmartNIC offload should be discussed only after the data-plane boundary and
+  baseline metrics show where CPU is spent.
+
+## Suggested PR Sequence
+
+1. Add this RFC and the current-path classification.
+2. Add route-level internode metrics for `/rustfs/rpc/read_file_stream`,
+   `/rustfs/rpc/put_file_stream`, `/rustfs/rpc/walk_dir`, and gRPC disk byte
+   calls.
+3. Add an internode transport benchmark harness that can run against a local
+   multi-node cluster and produce repeatable artifacts.
+4. Introduce an `InternodeDataTransport` wrapper with a TCP/HTTP backend that
+   preserves current behavior.
+5. Move `RemoteDisk` stream methods to the transport wrapper without changing
+   default behavior.
+6. Add an experimental feature-gated RDMA/RoCE backend only after the baseline
+   proves that internode byte transfer is a limiting factor.
+
+## Open Questions
+
+- Which production workload is the primary target: large-object throughput,
+  small-object tail latency, healing throughput, or rebalance throughput?
+- Should `ReadAll` and `WriteAll` stay as gRPC unary calls, or should large
+  payloads be redirected to the data transport?
+- Is `walk_dir` a metadata control stream or a secondary data-plane stream for
+  scanner/healing workloads?
+- What is the acceptable fallback policy for an explicitly configured
+  experimental backend?
+- How should an RDMA backend preserve authentication and encryption guarantees
+  currently provided by signed HTTP requests and TLS-capable gRPC/HTTP clients?
+- What hardware matrix is required before accepting a non-default RDMA backend?
+
+## Immediate Next Steps
+
+- Create a focused issue from this RFC.
+- Add route-level internode metrics before changing transport code.
+- Extend existing benchmark scripts or add a new script to isolate remote disk
+  stream read/write throughput.
+- Keep the first code PR behavior-preserving and TCP-only.

README.md

@@ -115,7 +115,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command

README_ZH.md

@@ -1,4 +1,4 @@
-[![RustFS](https://github.com/user-attachments/assets/1b5afcd6-a2c3-47ff-8bc3-ce882b0ddca7)](https://rustfs.com.cn)
+[![RustFS](https://repository-images.githubusercontent.com/722597620/0fa936a2-8164-4f53-867f-def4beb64b21)](https://rustfs.com.cn)
 
 <p align="center">RustFS 是一个基于 Rust 构建的高性能分布式对象存储系统。</p>
 
@@ -113,7 +113,7 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
  docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
  # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
+ docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.4
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：

build-rustfs.sh

@@ -217,7 +217,7 @@ setup_rust_environment() {
     # Set up environment variables for musl targets
     if [[ "$PLATFORM" == *"musl"* ]]; then
         print_message $YELLOW "Setting up environment for musl target..."
-        export RUSTFLAGS="'--cfg tokio_unstable -C target-feature=-crt-static'"
+        export RUSTFLAGS="--cfg tokio_unstable -C target-feature=-crt-static"
 
         # For cargo-zigbuild, set up additional environment variables
         if command -v cargo-zigbuild &> /dev/null; then

crates/appauth/README.md

@@ -1,37 +0,0 @@
-[![RustFS](https://rustfs.com/images/rustfs-github.png)](https://rustfs.com)
-
-# RustFS AppAuth - Application Authentication
-
-<p align="center">
-  <strong>Application-level authentication and authorization module for RustFS distributed object storage</strong>
-</p>
-
-<p align="center">
-  <a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
-  <a href="https://docs.rustfs.com/">📖 Documentation</a>
-  · <a href="https://github.com/rustfs/rustfs/issues">🐛 Bug Reports</a>
-  · <a href="https://github.com/rustfs/rustfs/discussions">💬 Discussions</a>
-</p>
-
----
-
-## 📖 Overview
-
-**RustFS AppAuth** provides application-level authentication and authorization capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
-
-## ✨ Features
-
-- JWT-based authentication with secure token management
-- RBAC (Role-Based Access Control) for fine-grained permissions
-- Multi-tenant application isolation and management
-- OAuth 2.0 and OpenID Connect integration
-- API key management and rotation
-- Session management with configurable expiration
-
-## 📚 Documentation
-
-For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
-
-## 📄 License
-
-This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.

crates/audit/Cargo.toml

@@ -29,7 +29,7 @@ categories = ["web-programming", "development-tools", "asynchronous", "api-bindi
 rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
-rustfs-s3-common = { workspace = true }
+rustfs-s3-types = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }
 futures = { workspace = true }

crates/audit/src/entity.rs

@@ -14,7 +14,7 @@
 
 use chrono::{DateTime, Utc};
 use hashbrown::HashMap;
-use rustfs_s3_common::EventName;
+use rustfs_s3_types::EventName;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
 

crates/checksums/Cargo.toml

@@ -23,7 +23,7 @@ homepage.workspace = true
 description = "Checksum calculation and verification callbacks for HTTP request and response bodies sent by service clients generated by RustFS, ensuring data integrity and authenticity."
 keywords = ["checksum-calculation", "verification", "integrity", "authenticity", "rustfs"]
 categories = ["web-programming", "development-tools", "network-programming"]
-documentation = "https://docs.rs/rustfs-signer/latest/rustfs_checksum/"
+documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
 
 [dependencies]
 bytes = { workspace = true }

crates/common/Cargo.toml

@@ -33,12 +33,8 @@ tonic = { workspace = true }
 uuid = { workspace = true }
 chrono = { workspace = true }
 metrics = { workspace = true }
-rustfs-madmin = { workspace = true }
-rustfs-filemeta = { workspace = true }
 serde = { workspace = true }
-path-clean = { workspace = true }
 rmp-serde = { workspace = true }
-async-trait = { workspace = true }
 s3s = { workspace = true }
 tracing = { workspace = true }
 

crates/common/src/heal_channel.rs

@@ -379,7 +379,9 @@ fn heal_response_sender() -> &'static HealResponseSender {
 
 /// Publish a heal response to subscribers.
 pub fn publish_heal_response(response: HealChannelResponse) -> Result<(), broadcast::error::SendError<HealChannelResponse>> {
-    heal_response_sender().send(response).map(|_| ())
+    let sender = heal_response_sender();
+    let _ = sender.send(response);
+    Ok(())
 }
 
 /// Subscribe to heal responses.
@@ -626,5 +628,10 @@ mod tests {
         let received = receiver.recv().await.expect("should receive heal response");
         assert_eq!(received.request_id, response.request_id);
         assert!(received.success);
+
+        drop(receiver);
+        let response = create_heal_response("req-no-subscriber".to_string(), true, None, None);
+
+        publish_heal_response(response).expect("publish without subscribers should be ignored");
     }
 }

crates/common/src/lib.rs

@@ -13,12 +13,9 @@
 // limitations under the License.
 
 pub mod bucket_stats;
-pub mod capacity_scope;
 // pub mod error;
-pub mod data_usage;
 pub mod globals;
 pub mod heal_channel;
-pub mod internode_metrics;
 pub mod last_minute;
 pub mod metrics;
 mod readiness;

crates/common/src/metrics.rs

@@ -14,7 +14,6 @@
 
 use crate::last_minute::{AccElem, LastMinuteLatency};
 use chrono::{DateTime, Utc};
-use rustfs_madmin::metrics::{ScannerMetrics as M_ScannerMetrics, TimedAction};
 use serde::{Deserialize, Serialize};
 use std::{
     collections::HashMap,
@@ -351,6 +350,32 @@ pub struct CurrentCycle {
     pub started: DateTime<Utc>,
 }
 
+#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
+pub struct ScannerTimedAction {
+    pub count: u64,
+    pub acc_time: u64,
+    pub bytes: u64,
+}
+
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
+pub struct ScannerLastMinute {
+    pub actions: HashMap<String, ScannerTimedAction>,
+    pub ilm: HashMap<String, ScannerTimedAction>,
+}
+
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
+pub struct ScannerMetricsReport {
+    pub collected_at: DateTime<Utc>,
+    pub current_cycle: u64,
+    pub current_started: DateTime<Utc>,
+    pub cycles_completed_at: Vec<DateTime<Utc>>,
+    pub ongoing_buckets: usize,
+    pub life_time_ops: HashMap<String, u64>,
+    pub life_time_ilm: HashMap<String, u64>,
+    pub last_minute: ScannerLastMinute,
+    pub active_paths: Vec<String>,
+}
+
 impl CurrentCycle {
     pub fn unmarshal(&mut self, buf: &[u8]) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         *self = rmp_serde::from_slice(buf)?;
@@ -572,8 +597,8 @@ impl Metrics {
     }
 
     /// Build a full metrics report snapshot.
-    pub async fn report(&self) -> M_ScannerMetrics {
-        let mut m = M_ScannerMetrics::default();
+    pub async fn report(&self) -> ScannerMetricsReport {
+        let mut m = ScannerMetricsReport::default();
 
         if let Some(cycle) = self.get_cycle().await {
             m.current_cycle = cycle.current;
@@ -606,7 +631,7 @@ impl Metrics {
             {
                 m.last_minute.actions.insert(
                     metric.as_str().to_string(),
-                    TimedAction {
+                    ScannerTimedAction {
                         count: last_min.n,
                         acc_time: last_min.total,
                         bytes: last_min.size,
@@ -633,7 +658,7 @@ impl Metrics {
             {
                 m.last_minute.ilm.insert(
                     action.as_str().to_string(),
-                    TimedAction {
+                    ScannerTimedAction {
                         count: last_min.n,
                         acc_time: last_min.total,
                         bytes: last_min.size,

crates/concurrency/src/backpressure.rs

@@ -14,15 +14,17 @@
 
 //! Backpressure management
 
-use rustfs_io_core::{BackpressureMonitor as CoreBackpressureMonitor, BackpressureState};
+use rustfs_io_core::{
+    BackpressureConfig as CoreBackpressureConfig, BackpressureMonitor as CoreBackpressureMonitor, BackpressureState,
+};
 use rustfs_io_metrics::backpressure_metrics;
 use std::sync::Arc;
 use std::time::Instant;
 use tokio::io::{DuplexStream, duplex};
 
-/// Backpressure configuration
-#[derive(Debug, Clone)]
-pub struct BackpressureConfig {
+/// Facade policy for duplex-pipe watermark backpressure.
+#[derive(Debug, Clone, Copy)]
+pub struct PipeBackpressurePolicy {
     /// Buffer size in bytes
     pub buffer_size: usize,
     /// High watermark percentage
@@ -31,7 +33,7 @@ pub struct BackpressureConfig {
     pub low_watermark: u32,
 }
 
-impl Default for BackpressureConfig {
+impl Default for PipeBackpressurePolicy {
     fn default() -> Self {
         Self {
             buffer_size: 4 * 1024 * 1024, // 4MB
@@ -41,7 +43,7 @@ impl Default for BackpressureConfig {
     }
 }
 
-impl BackpressureConfig {
+impl PipeBackpressurePolicy {
     /// Calculate high watermark threshold in bytes
     pub fn high_watermark_bytes(&self) -> usize {
         (self.buffer_size as u64 * self.high_watermark as u64 / 100) as usize
@@ -51,42 +53,59 @@ impl BackpressureConfig {
     pub fn low_watermark_bytes(&self) -> usize {
         (self.buffer_size as u64 * self.low_watermark as u64 / 100) as usize
     }
+
+    /// Convert the facade policy into the reusable io-core admission-pressure config.
+    ///
+    /// The concurrency layer still owns duplex buffer sizing, but the shared
+    /// overload/admission primitive lives in `io-core`.
+    pub fn to_core_config(&self) -> CoreBackpressureConfig {
+        CoreBackpressureConfig {
+            max_concurrent: 32,
+            high_water_mark: self.high_watermark as f64 / 100.0,
+            low_water_mark: self.low_watermark as f64 / 100.0,
+            cooldown: std::time::Duration::from_millis(100),
+            enabled: true,
+        }
+    }
 }
 
 /// Backpressure manager
 pub struct BackpressureManager {
-    config: BackpressureConfig,
+    config: PipeBackpressurePolicy,
+    core_config: CoreBackpressureConfig,
     monitor: Arc<CoreBackpressureMonitor>,
 }
 
 impl BackpressureManager {
     /// Create a new backpressure manager
     pub fn new(buffer_size: usize, high_watermark: u32, low_watermark: u32) -> Self {
-        let config = BackpressureConfig {
+        Self::from_policy(PipeBackpressurePolicy {
             buffer_size,
             high_watermark,
             low_watermark,
-        };
-
-        let core_config = rustfs_io_core::BackpressureConfig {
-            max_concurrent: 32,
-            high_water_mark: high_watermark as f64 / 100.0,
-            low_water_mark: low_watermark as f64 / 100.0,
-            cooldown: std::time::Duration::from_millis(100),
-            enabled: true,
-        };
+        })
+    }
 
+    /// Create a new backpressure manager from the facade policy type.
+    pub fn from_policy(config: PipeBackpressurePolicy) -> Self {
+        let core_config = config.to_core_config();
         Self {
             config,
+            core_config: core_config.clone(),
             monitor: Arc::new(CoreBackpressureMonitor::new(core_config)),
         }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &BackpressureConfig {
+    pub fn config(&self) -> &PipeBackpressurePolicy {
         &self.config
     }
 
+    /// Get the derived io-core admission-pressure configuration.
+    pub fn core_config(&self) -> &CoreBackpressureConfig {
+        &self.core_config
+    }
+
     /// Get the monitor
     pub fn monitor(&self) -> Arc<CoreBackpressureMonitor> {
         self.monitor.clone()
@@ -94,7 +113,7 @@ impl BackpressureManager {
 
     /// Create a backpressure pipe
     pub fn create_pipe(&self) -> BackpressurePipe {
-        BackpressurePipe::new(self.config.clone(), self.monitor.clone())
+        BackpressurePipe::new(self.config, self.monitor.clone())
     }
 
     /// Get current state
@@ -112,13 +131,24 @@ impl BackpressureManager {
 pub struct BackpressurePipe {
     reader: DuplexStream,
     writer: DuplexStream,
-    config: BackpressureConfig,
+    config: PipeBackpressurePolicy,
     monitor: Arc<CoreBackpressureMonitor>,
     created_at: Instant,
 }
 
+/// Shared pipe metadata snapshot for facade-level backpressure pipes.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct BackpressurePipeMeta {
+    /// Configured duplex buffer capacity in bytes.
+    pub buffer_capacity: usize,
+    /// Current backpressure state reported by the shared core monitor.
+    pub state: BackpressureState,
+    /// Age of the pipe since creation.
+    pub age: std::time::Duration,
+}
+
 impl BackpressurePipe {
-    fn new(config: BackpressureConfig, monitor: Arc<CoreBackpressureMonitor>) -> Self {
+    fn new(config: PipeBackpressurePolicy, monitor: Arc<CoreBackpressureMonitor>) -> Self {
         let (reader, writer) = duplex(config.buffer_size);
 
         Self {
@@ -146,7 +176,7 @@ impl BackpressurePipe {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &BackpressureConfig {
+    pub fn config(&self) -> &PipeBackpressurePolicy {
         &self.config
     }
 
@@ -160,6 +190,15 @@ impl BackpressurePipe {
         self.created_at.elapsed()
     }
 
+    /// Get a compact metadata snapshot for the pipe.
+    pub fn meta(&self) -> BackpressurePipeMeta {
+        BackpressurePipeMeta {
+            buffer_capacity: self.config.buffer_size,
+            state: self.state(),
+            age: self.age(),
+        }
+    }
+
     /// Check if should apply backpressure
     pub fn should_apply_backpressure(&self) -> bool {
         let should = self.monitor.should_apply_backpressure();
@@ -170,45 +209,26 @@ impl BackpressurePipe {
     }
 }
 
-/// Backpressure event
-#[allow(dead_code)]
-#[derive(Debug, Clone)]
-pub struct BackpressureEvent {
-    /// Event timestamp
-    pub timestamp: Instant,
-    /// Event type
-    pub event_type: BackpressureEventType,
-    /// Buffer usage
-    pub buffer_usage: usize,
-    /// Buffer capacity
-    pub buffer_capacity: usize,
-}
-
-/// Backpressure event type
-#[allow(dead_code)]
-#[derive(Debug, Clone, Copy)]
-pub enum BackpressureEventType {
-    /// High watermark reached
-    HighWatermarkReached,
-    /// High watermark exited
-    HighWatermarkExited,
-    /// Backpressure applied
-    BackpressureApplied,
-    /// Backpressure released
-    BackpressureReleased,
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
 
     #[test]
     fn test_backpressure_config() {
-        let config = BackpressureConfig::default();
+        let config = PipeBackpressurePolicy::default();
         assert_eq!(config.buffer_size, 4 * 1024 * 1024);
         assert!(config.high_watermark > config.low_watermark);
     }
 
+    #[test]
+    fn test_backpressure_policy_to_core_config() {
+        let policy = PipeBackpressurePolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.high_water_mark, policy.high_watermark as f64 / 100.0);
+        assert_eq!(core.low_water_mark, policy.low_watermark as f64 / 100.0);
+        assert!(core.enabled);
+    }
+
     #[test]
     fn test_backpressure_manager() {
         let manager = BackpressureManager::new(1024, 80, 50);
@@ -220,5 +240,6 @@ mod tests {
         let manager = BackpressureManager::new(1024, 80, 50);
         let pipe = manager.create_pipe();
         assert_eq!(pipe.state(), BackpressureState::Normal);
+        assert_eq!(pipe.meta().buffer_capacity, 1024);
     }
 }

crates/concurrency/src/config.rs

@@ -14,6 +14,10 @@
 
 //! Configuration for concurrency management
 
+use crate::{
+    backpressure::PipeBackpressurePolicy, deadlock::DeadlockMonitorPolicy, scheduler::SchedulerPolicy,
+    timeout::TimeoutManagerPolicy,
+};
 use std::time::Duration;
 
 /// Feature flags for concurrency modules
@@ -72,84 +76,39 @@ impl ConcurrencyFeatures {
     }
 }
 
+/// Facade policy for lock manager behavior.
+#[derive(Debug, Clone, Copy)]
+pub struct LockManagerPolicy {
+    /// Enable lock optimization.
+    pub enabled: bool,
+    /// Lock acquisition timeout.
+    pub acquire_timeout: Duration,
+}
+
+impl Default for LockManagerPolicy {
+    fn default() -> Self {
+        Self {
+            enabled: true,
+            acquire_timeout: Duration::from_secs(5),
+        }
+    }
+}
+
 /// Main configuration for concurrency management
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Default)]
 pub struct ConcurrencyConfig {
     /// Feature flags
     pub features: ConcurrencyFeatures,
-
-    // Timeout configuration
-    /// Default timeout duration
-    pub default_timeout: Duration,
-    /// Maximum timeout duration
-    pub max_timeout: Duration,
-    /// Enable dynamic timeout
-    pub enable_dynamic_timeout: bool,
-
-    // Lock configuration
-    /// Enable lock optimization
-    pub enable_lock_optimization: bool,
-    /// Lock acquisition timeout
-    pub lock_acquire_timeout: Duration,
-
-    // Deadlock configuration
-    /// Enable deadlock detection
-    pub enable_deadlock_detection: bool,
-    /// Deadlock check interval
-    pub deadlock_check_interval: Duration,
-    /// Hang threshold
-    pub hang_threshold: Duration,
-
-    // Backpressure configuration
-    /// Buffer size for backpressure
-    pub backpressure_buffer_size: usize,
-    /// High watermark percentage
-    pub high_watermark: u32,
-    /// Low watermark percentage
-    pub low_watermark: u32,
-
-    // Scheduler configuration
-    /// Base buffer size for I/O
-    pub io_buffer_size: usize,
-    /// Maximum buffer size
-    pub max_buffer_size: usize,
-    /// High priority size threshold
-    pub high_priority_threshold: usize,
-    /// Low priority size threshold
-    pub low_priority_threshold: usize,
-}
-
-impl Default for ConcurrencyConfig {
-    fn default() -> Self {
-        Self {
-            features: ConcurrencyFeatures::default(),
-
-            // Timeout defaults
-            default_timeout: Duration::from_secs(30),
-            max_timeout: Duration::from_secs(300),
-            enable_dynamic_timeout: true,
-
-            // Lock defaults
-            enable_lock_optimization: true,
-            lock_acquire_timeout: Duration::from_secs(5),
-
-            // Deadlock defaults
-            enable_deadlock_detection: false,
-            deadlock_check_interval: Duration::from_secs(10),
-            hang_threshold: Duration::from_secs(60),
-
-            // Backpressure defaults
-            backpressure_buffer_size: 4 * 1024 * 1024, // 4MB
-            high_watermark: 80,
-            low_watermark: 50,
-
-            // Scheduler defaults
-            io_buffer_size: 64 * 1024,                // 64KB
-            max_buffer_size: 4 * 1024 * 1024,         // 4MB
-            high_priority_threshold: 1024 * 1024,     // 1MB
-            low_priority_threshold: 10 * 1024 * 1024, // 10MB
-        }
-    }
+    /// Timeout facade policy.
+    pub timeout_policy: TimeoutManagerPolicy,
+    /// Lock facade policy.
+    pub lock_policy: LockManagerPolicy,
+    /// Deadlock facade policy.
+    pub deadlock_policy: DeadlockMonitorPolicy,
+    /// Backpressure facade policy.
+    pub backpressure_policy: PipeBackpressurePolicy,
+    /// Scheduler facade policy.
+    pub scheduler_policy: SchedulerPolicy,
 }
 
 impl ConcurrencyConfig {
@@ -161,25 +120,25 @@ impl ConcurrencyConfig {
         if let Ok(val) = std::env::var("RUSTFS_TIMEOUT_DEFAULT")
             && let Ok(secs) = val.parse::<u64>()
         {
-            config.default_timeout = Duration::from_secs(secs);
+            config.timeout_policy.default_timeout = Duration::from_secs(secs);
         }
 
         if let Ok(val) = std::env::var("RUSTFS_TIMEOUT_MAX")
             && let Ok(secs) = val.parse::<u64>()
         {
-            config.max_timeout = Duration::from_secs(secs);
+            config.timeout_policy.max_timeout = Duration::from_secs(secs);
         }
 
         if let Ok(val) = std::env::var("RUSTFS_BACKPRESSURE_BUFFER_SIZE")
             && let Ok(size) = val.parse::<usize>()
         {
-            config.backpressure_buffer_size = size;
+            config.backpressure_policy.buffer_size = size;
         }
 
         if let Ok(val) = std::env::var("RUSTFS_IO_BUFFER_SIZE")
             && let Ok(size) = val.parse::<usize>()
         {
-            config.io_buffer_size = size;
+            config.scheduler_policy.base_buffer_size = size;
         }
 
         config
@@ -187,18 +146,25 @@ impl ConcurrencyConfig {
 
     /// Validate configuration
     pub fn validate(&self) -> Result<(), ConfigError> {
-        if self.default_timeout > self.max_timeout {
+        if self.timeout_policy.default_timeout > self.timeout_policy.max_timeout {
             return Err(ConfigError::InvalidTimeout("default_timeout cannot exceed max_timeout".to_string()));
         }
+        if self.timeout_policy.min_timeout > self.timeout_policy.max_timeout {
+            return Err(ConfigError::InvalidTimeout("min_timeout cannot exceed max_timeout".to_string()));
+        }
 
-        if self.high_watermark <= self.low_watermark || self.high_watermark > 100 {
+        if self.backpressure_policy.high_watermark <= self.backpressure_policy.low_watermark
+            || self.backpressure_policy.high_watermark > 100
+        {
             return Err(ConfigError::InvalidBackpressure(
                 "high_watermark must be > low_watermark and <= 100".to_string(),
             ));
         }
 
-        if self.io_buffer_size > self.max_buffer_size {
-            return Err(ConfigError::InvalidScheduler("io_buffer_size cannot exceed max_buffer_size".to_string()));
+        if self.scheduler_policy.base_buffer_size > self.scheduler_policy.max_buffer_size {
+            return Err(ConfigError::InvalidScheduler(
+                "base_buffer_size cannot exceed max_buffer_size".to_string(),
+            ));
         }
 
         Ok(())
@@ -235,8 +201,12 @@ mod tests {
     #[test]
     fn test_invalid_timeout() {
         let config = ConcurrencyConfig {
-            default_timeout: Duration::from_secs(100),
-            max_timeout: Duration::from_secs(50),
+            timeout_policy: TimeoutManagerPolicy {
+                default_timeout: Duration::from_secs(100),
+                max_timeout: Duration::from_secs(50),
+                enable_dynamic: true,
+                ..Default::default()
+            },
             ..Default::default()
         };
         assert!(
@@ -245,6 +215,22 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_invalid_min_timeout() {
+        let config = ConcurrencyConfig {
+            timeout_policy: TimeoutManagerPolicy {
+                min_timeout: Duration::from_secs(100),
+                max_timeout: Duration::from_secs(50),
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+        assert!(
+            config.validate().is_err(),
+            "validate() should return an error when min_timeout > max_timeout"
+        );
+    }
+
     #[test]
     fn test_features() {
         let features = ConcurrencyFeatures::all();

crates/concurrency/src/deadlock.rs

@@ -14,15 +14,15 @@
 
 //! Deadlock detection management
 
-use rustfs_io_core::{DeadlockDetector as CoreDeadlockDetector, LockType};
+use rustfs_io_core::{DeadlockDetector as CoreDeadlockDetector, DeadlockDetectorConfig as CoreDeadlockConfig, LockType};
 use rustfs_io_metrics::deadlock_metrics;
 use std::collections::HashMap;
 use std::sync::Arc;
 use std::time::{Duration, Instant};
 
-/// Deadlock configuration
-#[derive(Debug, Clone)]
-pub struct DeadlockConfig {
+/// Facade policy for the concurrency-layer deadlock monitor.
+#[derive(Debug, Clone, Copy)]
+pub struct DeadlockMonitorPolicy {
     /// Enable deadlock detection
     pub enabled: bool,
     /// Check interval
@@ -31,7 +31,7 @@ pub struct DeadlockConfig {
     pub hang_threshold: Duration,
 }
 
-impl Default for DeadlockConfig {
+impl Default for DeadlockMonitorPolicy {
     fn default() -> Self {
         Self {
             enabled: false,
@@ -41,9 +41,20 @@ impl Default for DeadlockConfig {
     }
 }
 
+impl DeadlockMonitorPolicy {
+    /// Convert the facade policy into the reusable io-core deadlock config.
+    pub fn to_core_config(&self) -> CoreDeadlockConfig {
+        CoreDeadlockConfig {
+            enabled: self.enabled,
+            detection_interval: self.check_interval,
+            max_hold_time: self.hang_threshold,
+        }
+    }
+}
+
 /// Deadlock manager
 pub struct DeadlockManager {
-    config: DeadlockConfig,
+    config: DeadlockMonitorPolicy,
     detector: Arc<CoreDeadlockDetector>,
     running: Arc<tokio::sync::Mutex<bool>>,
 }
@@ -51,18 +62,16 @@ pub struct DeadlockManager {
 impl DeadlockManager {
     /// Create a new deadlock manager
     pub fn new(enabled: bool, check_interval: Duration, hang_threshold: Duration) -> Self {
-        let config = DeadlockConfig {
+        Self::from_policy(DeadlockMonitorPolicy {
             enabled,
             check_interval,
             hang_threshold,
-        };
-
-        let core_config = rustfs_io_core::DeadlockDetectorConfig {
-            enabled,
-            detection_interval: check_interval,
-            max_hold_time: hang_threshold,
-        };
+        })
+    }
 
+    /// Create a new deadlock manager from the facade policy type.
+    pub fn from_policy(config: DeadlockMonitorPolicy) -> Self {
+        let core_config = config.to_core_config();
         Self {
             config,
             detector: Arc::new(CoreDeadlockDetector::new(core_config)),
@@ -71,7 +80,7 @@ impl DeadlockManager {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &DeadlockConfig {
+    pub fn config(&self) -> &DeadlockMonitorPolicy {
         &self.config
     }
 
@@ -129,7 +138,11 @@ impl DeadlockManager {
     }
 }
 
-/// Request tracker for tracking resources
+/// Lightweight compatibility wrapper for request-scoped deadlock bookkeeping.
+///
+/// This type intentionally stays minimal in the concurrency layer. Rich
+/// request-level lock/resource diagnostics belong to
+/// `rustfs::storage::deadlock_detector::RequestResourceTracker`.
 pub struct RequestTracker {
     request_id: String,
     description: String,
@@ -174,6 +187,11 @@ impl RequestTracker {
         deadlock_metrics::record_lock_acquisition("read");
     }
 
+    /// Return a read-only view of tracked resource names.
+    pub fn resources(&self) -> &HashMap<String, Vec<String>> {
+        &self.resources
+    }
+
     /// Record a lock release
     pub fn record_lock_release(&mut self, lock_id: u64) {
         self.detector.record_release(lock_id);
@@ -196,12 +214,24 @@ mod tests {
         assert!(!manager.config().enabled);
     }
 
+    #[test]
+    fn test_deadlock_policy_to_core_config() {
+        let policy = DeadlockMonitorPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.enabled, policy.enabled);
+        assert_eq!(core.detection_interval, policy.check_interval);
+        assert_eq!(core.max_hold_time, policy.hang_threshold);
+    }
+
     #[tokio::test]
     async fn test_request_tracker() {
         let manager = DeadlockManager::new(true, Duration::from_secs(10), Duration::from_secs(60));
-        let tracker = manager.track_request("req-1".to_string(), "test request".to_string());
+        let mut tracker = manager.track_request("req-1".to_string(), "test request".to_string());
+        let lock_id = manager.register_lock(LockType::Mutex);
+        tracker.record_lock_acquire(lock_id, "bucket/key".to_string());
 
         assert_eq!(tracker.request_id(), "req-1");
         assert_eq!(tracker.description(), "test request");
+        assert_eq!(tracker.resources().get("locks").map(Vec::len), Some(1));
     }
 }

crates/concurrency/src/lib.rs

@@ -124,21 +124,23 @@ mod backpressure;
 #[cfg(feature = "scheduler")]
 mod scheduler;
 
+pub mod workers;
+
 // Public module exports with feature gates
 #[cfg(feature = "timeout")]
-pub use timeout::{TimeoutConfig, TimeoutGuard, TimeoutManager};
+pub use timeout::{TimeoutGuard, TimeoutManager, TimeoutManagerPolicy};
 
 #[cfg(feature = "lock")]
 pub use lock::{LockConfig, LockManager, LockScopeGuard, OptimizedLockGuard};
 
 #[cfg(feature = "deadlock")]
-pub use deadlock::{DeadlockConfig, DeadlockManager, RequestTracker};
+pub use deadlock::{DeadlockManager, DeadlockMonitorPolicy, RequestTracker};
 
 #[cfg(feature = "backpressure")]
-pub use backpressure::{BackpressureConfig, BackpressureManager, BackpressurePipe};
+pub use backpressure::{BackpressureManager, BackpressurePipe, PipeBackpressurePolicy};
 
 #[cfg(feature = "scheduler")]
-pub use scheduler::{IoStrategy, SchedulerConfig, SchedulerManager};
+pub use scheduler::{IoStrategy, SchedulerManager, SchedulerPolicy};
 
 // Configuration
 mod config;
@@ -153,19 +155,19 @@ pub mod prelude {
     //! Prelude module for convenient imports
 
     #[cfg(feature = "timeout")]
-    pub use crate::timeout::{TimeoutConfig, TimeoutGuard, TimeoutManager};
+    pub use crate::timeout::{TimeoutGuard, TimeoutManager, TimeoutManagerPolicy};
 
     #[cfg(feature = "lock")]
     pub use crate::lock::{LockConfig, LockManager, LockScopeGuard, OptimizedLockGuard};
 
     #[cfg(feature = "deadlock")]
-    pub use crate::deadlock::{DeadlockConfig, DeadlockManager, RequestTracker};
+    pub use crate::deadlock::{DeadlockManager, DeadlockMonitorPolicy, RequestTracker};
 
     #[cfg(feature = "backpressure")]
-    pub use crate::backpressure::{BackpressureConfig, BackpressureManager, BackpressurePipe};
+    pub use crate::backpressure::{BackpressureManager, BackpressurePipe, PipeBackpressurePolicy};
 
     #[cfg(feature = "scheduler")]
-    pub use crate::scheduler::{IoStrategy, SchedulerConfig, SchedulerManager};
+    pub use crate::scheduler::{IoStrategy, SchedulerManager, SchedulerPolicy};
 
     pub use crate::{ConcurrencyConfig, ConcurrencyFeatures, ConcurrencyManager};
 }

crates/concurrency/src/manager.rs

@@ -85,39 +85,22 @@ impl ConcurrencyManager {
 
         Self {
             #[cfg(feature = "timeout")]
-            timeout: Arc::new(crate::timeout::TimeoutManager::new(
-                config.default_timeout,
-                config.max_timeout,
-                config.enable_dynamic_timeout,
-            )),
+            timeout: Arc::new(crate::timeout::TimeoutManager::from_policy(config.timeout_policy)),
 
             #[cfg(feature = "lock")]
             lock: Arc::new(crate::lock::LockManager::new(
-                config.enable_lock_optimization,
-                config.lock_acquire_timeout,
+                config.lock_policy.enabled,
+                config.lock_policy.acquire_timeout,
             )),
 
             #[cfg(feature = "deadlock")]
-            deadlock: Arc::new(crate::deadlock::DeadlockManager::new(
-                config.enable_deadlock_detection,
-                config.deadlock_check_interval,
-                config.hang_threshold,
-            )),
+            deadlock: Arc::new(crate::deadlock::DeadlockManager::from_policy(config.deadlock_policy)),
 
             #[cfg(feature = "backpressure")]
-            backpressure: Arc::new(crate::backpressure::BackpressureManager::new(
-                config.backpressure_buffer_size,
-                config.high_watermark,
-                config.low_watermark,
-            )),
+            backpressure: Arc::new(crate::backpressure::BackpressureManager::from_policy(config.backpressure_policy)),
 
             #[cfg(feature = "scheduler")]
-            scheduler: Arc::new(crate::scheduler::SchedulerManager::new(
-                config.io_buffer_size,
-                config.max_buffer_size,
-                config.high_priority_threshold,
-                config.low_priority_threshold,
-            )),
+            scheduler: Arc::new(crate::scheduler::SchedulerManager::from_policy(config.scheduler_policy)),
 
             config,
         }
@@ -244,7 +227,7 @@ impl ConcurrencyManager {
     pub async fn start(&self) {
         #[cfg(feature = "deadlock")]
         {
-            if self.config.enable_deadlock_detection {
+            if self.config.deadlock_policy.enabled {
                 self.deadlock.start().await;
             }
         }

crates/concurrency/src/scheduler.rs

@@ -22,9 +22,9 @@ use rustfs_io_metrics::io_metrics;
 use std::sync::Arc;
 use std::time::Duration;
 
-/// Scheduler configuration
-#[derive(Debug, Clone)]
-pub struct SchedulerConfig {
+/// Facade policy for the concurrency-layer scheduler manager.
+#[derive(Debug, Clone, Copy)]
+pub struct SchedulerPolicy {
     /// Base buffer size
     pub base_buffer_size: usize,
     /// Maximum buffer size
@@ -35,7 +35,7 @@ pub struct SchedulerConfig {
     pub low_priority_threshold: usize,
 }
 
-impl Default for SchedulerConfig {
+impl Default for SchedulerPolicy {
     fn default() -> Self {
         Self {
             base_buffer_size: 64 * 1024,              // 64KB
@@ -46,9 +46,23 @@ impl Default for SchedulerConfig {
     }
 }
 
+impl SchedulerPolicy {
+    /// Convert facade policy to io-core scheduler config.
+    pub fn to_core_config(&self) -> rustfs_io_core::IoSchedulerConfig {
+        rustfs_io_core::IoSchedulerConfig {
+            base_buffer_size: self.base_buffer_size,
+            max_buffer_size: self.max_buffer_size,
+            high_priority_size_threshold: self.high_priority_threshold,
+            low_priority_size_threshold: self.low_priority_threshold,
+            ..rustfs_io_core::IoSchedulerConfig::default()
+        }
+    }
+}
+
 /// Scheduler manager
 pub struct SchedulerManager {
-    config: SchedulerConfig,
+    config: SchedulerPolicy,
+    core_config: rustfs_io_core::IoSchedulerConfig,
     scheduler: Arc<CoreIoScheduler>,
 }
 
@@ -60,26 +74,35 @@ impl SchedulerManager {
         high_priority_threshold: usize,
         low_priority_threshold: usize,
     ) -> Self {
-        let config = SchedulerConfig {
+        Self::from_policy(SchedulerPolicy {
             base_buffer_size,
             max_buffer_size,
             high_priority_threshold,
             low_priority_threshold,
-        };
+        })
+    }
 
-        let core_config = rustfs_io_core::IoSchedulerConfig::default();
+    /// Create a scheduler manager from facade policy.
+    pub fn from_policy(config: SchedulerPolicy) -> Self {
+        let core_config = config.to_core_config();
 
         Self {
             config,
+            core_config: core_config.clone(),
             scheduler: Arc::new(CoreIoScheduler::new(core_config)),
         }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &SchedulerConfig {
+    pub fn config(&self) -> &SchedulerPolicy {
         &self.config
     }
 
+    /// Get the derived io-core scheduler config.
+    pub fn core_config(&self) -> &rustfs_io_core::IoSchedulerConfig {
+        &self.core_config
+    }
+
     /// Get the scheduler
     pub fn scheduler(&self) -> Arc<CoreIoScheduler> {
         self.scheduler.clone()
@@ -87,7 +110,7 @@ impl SchedulerManager {
 
     /// Create an I/O strategy
     pub fn create_strategy(&self) -> IoStrategy {
-        IoStrategy::new(self.config.clone(), self.scheduler.clone())
+        IoStrategy::new(self.config, self.scheduler.clone())
     }
 
     /// Calculate buffer size
@@ -111,12 +134,12 @@ impl SchedulerManager {
 
 /// I/O strategy
 pub struct IoStrategy {
-    config: SchedulerConfig,
+    config: SchedulerPolicy,
     scheduler: Arc<CoreIoScheduler>,
 }
 
 impl IoStrategy {
-    fn new(config: SchedulerConfig, scheduler: Arc<CoreIoScheduler>) -> Self {
+    fn new(config: SchedulerPolicy, scheduler: Arc<CoreIoScheduler>) -> Self {
         Self { config, scheduler }
     }
 
@@ -191,7 +214,7 @@ impl IoStrategy {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &SchedulerConfig {
+    pub fn config(&self) -> &SchedulerPolicy {
         &self.config
     }
 }
@@ -202,10 +225,20 @@ mod tests {
 
     #[test]
     fn test_scheduler_config() {
-        let config = SchedulerConfig::default();
+        let config = SchedulerPolicy::default();
         assert!(config.base_buffer_size < config.max_buffer_size);
     }
 
+    #[test]
+    fn test_scheduler_policy_to_core_config() {
+        let policy = SchedulerPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.base_buffer_size, policy.base_buffer_size);
+        assert_eq!(core.max_buffer_size, policy.max_buffer_size);
+        assert_eq!(core.high_priority_size_threshold, policy.high_priority_threshold);
+        assert_eq!(core.low_priority_size_threshold, policy.low_priority_threshold);
+    }
+
     #[test]
     fn test_scheduler_manager() {
         let manager = SchedulerManager::new(1024, 4096, 512, 2048);

crates/concurrency/src/timeout.rs

@@ -14,60 +14,101 @@
 
 //! Timeout management for operations
 
-use rustfs_io_core::{TimeoutError, calculate_adaptive_timeout};
+use rustfs_io_core::{TimeoutConfig as CoreTimeoutConfig, TimeoutError, calculate_adaptive_timeout};
 use std::time::{Duration, Instant};
 use tokio_util::sync::CancellationToken;
 
-/// Timeout configuration
-#[derive(Debug, Clone)]
-pub struct TimeoutConfig {
+/// Facade policy for the concurrency-layer timeout manager.
+#[derive(Debug, Clone, Copy)]
+pub struct TimeoutManagerPolicy {
     /// Default timeout duration
     pub default_timeout: Duration,
     /// Maximum timeout duration
     pub max_timeout: Duration,
+    /// Minimum timeout floor (prevents dynamic calculation from going too low).
+    pub min_timeout: Duration,
     /// Enable dynamic timeout calculation
     pub enable_dynamic: bool,
 }
 
-impl Default for TimeoutConfig {
+impl Default for TimeoutManagerPolicy {
     fn default() -> Self {
         Self {
             default_timeout: Duration::from_secs(30),
             max_timeout: Duration::from_secs(300),
+            min_timeout: Duration::from_secs(5),
             enable_dynamic: true,
         }
     }
 }
 
+impl TimeoutManagerPolicy {
+    /// Convert the facade policy into the reusable io-core timeout configuration.
+    ///
+    /// This keeps the concurrency layer explicitly wired to the shared core
+    /// timeout primitives without changing the facade's public behavior.
+    pub fn to_core_config(&self) -> CoreTimeoutConfig {
+        CoreTimeoutConfig {
+            base_timeout: self.default_timeout,
+            timeout_per_mb: Duration::ZERO,
+            max_timeout: self.max_timeout,
+            min_timeout: self.min_timeout,
+            get_object_timeout: self.default_timeout,
+            put_object_timeout: self.max_timeout,
+            list_objects_timeout: self.default_timeout,
+            enable_dynamic_timeout: self.enable_dynamic,
+        }
+    }
+}
+
 /// Timeout manager
 pub struct TimeoutManager {
-    config: TimeoutConfig,
+    config: TimeoutManagerPolicy,
+    core_config: CoreTimeoutConfig,
 }
 
 impl TimeoutManager {
     /// Create a new timeout manager
     pub fn new(default_timeout: Duration, max_timeout: Duration, enable_dynamic: bool) -> Self {
-        Self {
-            config: TimeoutConfig {
-                default_timeout,
-                max_timeout,
-                enable_dynamic,
-            },
-        }
+        let min_timeout = default_timeout.min(max_timeout);
+        Self::from_policy(TimeoutManagerPolicy {
+            default_timeout,
+            max_timeout,
+            min_timeout,
+            enable_dynamic,
+        })
+    }
+
+    /// Create a new timeout manager from the facade policy type.
+    pub fn from_policy(config: TimeoutManagerPolicy) -> Self {
+        let config = TimeoutManagerPolicy {
+            // Guard clamp(min, max) from panic when callers provide an
+            // out-of-order policy (or very small max_timeout).
+            min_timeout: config.min_timeout.min(config.max_timeout),
+            ..config
+        };
+        let core_config = config.to_core_config();
+        Self { config, core_config }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &TimeoutConfig {
+    pub fn config(&self) -> &TimeoutManagerPolicy {
         &self.config
     }
 
+    /// Get the derived io-core timeout configuration.
+    pub fn core_config(&self) -> &CoreTimeoutConfig {
+        &self.core_config
+    }
+
     /// Calculate timeout for a given size
     pub fn calculate_timeout(&self, size: u64, _history: &[Duration]) -> Duration {
         if !self.config.enable_dynamic {
             return self.config.default_timeout;
         }
 
-        calculate_adaptive_timeout(self.config.default_timeout, None, 0, size).min(self.config.max_timeout)
+        calculate_adaptive_timeout(self.core_config.base_timeout, None, 0, size)
+            .clamp(self.core_config.min_timeout, self.core_config.max_timeout)
     }
 
     /// Wrap an operation with timeout control
@@ -87,7 +128,7 @@ impl TimeoutManager {
 
     /// Create a timeout guard for manual timeout control
     pub fn create_guard(&self, timeout: Option<Duration>) -> TimeoutGuard {
-        TimeoutGuard::new(timeout.unwrap_or(self.config.default_timeout))
+        TimeoutGuard::new(timeout.unwrap_or(self.core_config.base_timeout))
     }
 }
 
@@ -134,10 +175,41 @@ mod tests {
 
     #[test]
     fn test_timeout_config() {
-        let config = TimeoutConfig::default();
+        let config = TimeoutManagerPolicy::default();
         assert!(config.default_timeout < config.max_timeout);
     }
 
+    #[test]
+    fn test_timeout_policy_to_core_config() {
+        let policy = TimeoutManagerPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.base_timeout, policy.default_timeout);
+        assert_eq!(core.max_timeout, policy.max_timeout);
+        assert_eq!(core.min_timeout, policy.min_timeout);
+        assert_eq!(core.get_object_timeout, policy.default_timeout);
+        assert!(core.enable_dynamic_timeout);
+    }
+
+    #[test]
+    fn test_timeout_manager_new_sanitizes_min_timeout_with_small_max_timeout() {
+        let manager = TimeoutManager::new(Duration::from_secs(1), Duration::from_secs(1), true);
+        let timeout = manager.calculate_timeout(1024, &[]);
+        assert_eq!(timeout, Duration::from_secs(1));
+    }
+
+    #[test]
+    fn test_timeout_manager_from_policy_sanitizes_min_timeout() {
+        let manager = TimeoutManager::from_policy(TimeoutManagerPolicy {
+            default_timeout: Duration::from_secs(30),
+            max_timeout: Duration::from_secs(1),
+            min_timeout: Duration::from_secs(5),
+            enable_dynamic: true,
+        });
+
+        assert_eq!(manager.config().min_timeout, Duration::from_secs(1));
+        assert_eq!(manager.core_config().min_timeout, Duration::from_secs(1));
+    }
+
     #[tokio::test]
     async fn test_wrap_operation_success() {
         let manager = TimeoutManager::new(Duration::from_secs(5), Duration::from_secs(10), true);

crates/concurrency/src/workers.rs

@@ -12,10 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+//! Worker slot limiter used by long-running background workflows.
+
 use std::sync::Arc;
 use tokio::sync::{Mutex, Notify};
 use tracing::info;
 
+/// Cooperative worker-slot controller for async tasks.
 pub struct Workers {
     available: Mutex<usize>, // Available working slots
     notify: Notify,          // Used to notify waiting tasks
@@ -23,20 +26,20 @@ pub struct Workers {
 }
 
 impl Workers {
-    // Create a Workers object that allows up to n jobs to execute concurrently
-    pub fn new(n: usize) -> Result<Arc<Workers>, &'static str> {
+    /// Create a [`Workers`] object that allows up to `n` jobs to execute concurrently.
+    pub fn new(n: usize) -> Result<Arc<Self>, &'static str> {
         if n == 0 {
             return Err("n must be > 0");
         }
 
-        Ok(Arc::new(Workers {
+        Ok(Arc::new(Self {
             available: Mutex::new(n),
             notify: Notify::new(),
             limit: n,
         }))
     }
 
-    // Give a job a chance to be executed
+    /// Acquire a worker slot, waiting until one becomes available.
     pub async fn take(&self) {
         loop {
             let mut available = self.available.lock().await;
@@ -51,7 +54,7 @@ impl Workers {
         }
     }
 
-    // Release a job's slot
+    /// Release a worker slot.
     pub async fn give(&self) {
         let mut available = self.available.lock().await;
         info!("worker give, {}", *available);
@@ -59,7 +62,7 @@ impl Workers {
         self.notify.notify_one(); // Notify a waiting task
     }
 
-    // Wait for all concurrent jobs to complete
+    /// Wait until all worker slots are released.
     pub async fn wait(&self) {
         loop {
             {
@@ -74,6 +77,7 @@ impl Workers {
         info!("worker wait end");
     }
 
+    /// Return the current number of available worker slots.
     pub async fn available(&self) -> usize {
         *self.available.lock().await
     }

crates/config/src/constants/app.rs

@@ -155,6 +155,12 @@ pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
+/// Environment variable to explicitly allow public default root credentials.
+///
+/// This is intended for local development only. Production startup paths should
+/// provide non-default `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY` values.
+pub const ENV_RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS: &str = "RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS";
+
 /// Environment variable for server OBS endpoint.
 pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 
@@ -293,9 +299,9 @@ pub const DEFAULT_OBS_LOGS_EXPORT_ENABLED: bool = true;
 
 /// Default profiling export enabled
 /// It is used to enable or disable exporting profiles
-/// Default value: false
+/// Default value: true
 /// Environment variable: RUSTFS_OBS_PROFILING_EXPORT_ENABLED
-pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = false;
+pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = true;
 
 /// Default log local logging enabled for rustfs
 /// This is the default log local logging enabled for rustfs.

crates/config/src/constants/internode.rs

@@ -32,6 +32,10 @@ pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 3;
 pub const ENV_INTERNODE_RPC_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS";
 pub const DEFAULT_INTERNODE_RPC_TIMEOUT_SECS: u64 = 10;
 
+/// Environment variable for selecting the internode data-plane transport backend.
+pub const ENV_RUSTFS_INTERNODE_DATA_TRANSPORT: &str = "RUSTFS_INTERNODE_DATA_TRANSPORT";
+pub const DEFAULT_INTERNODE_DATA_TRANSPORT: &str = "tcp-http";
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -58,5 +62,7 @@ mod tests {
             "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS"
         );
         assert_eq!(ENV_INTERNODE_RPC_TIMEOUT_SECS, "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS");
+        assert_eq!(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT, "RUSTFS_INTERNODE_DATA_TRANSPORT");
+        assert_eq!(DEFAULT_INTERNODE_DATA_TRANSPORT, "tcp-http");
     }
 }

crates/config/src/constants/runtime.rs

@@ -59,6 +59,25 @@ pub const DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT: usize = 10;
 pub const DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE: f64 = 1.0; // 100% sampling
 // Note: S3 bucket/prefix have no default; absence means upload is disabled (modeled as Option<String>)
 
+/// Maximum transition workers used as a local fallback when runtime env is unset.
+pub const DEFAULT_TRANSITION_WORKERS_CAP: i64 = 16;
+/// Absolute upper bound for transition workers accepted from runtime env.
+pub const DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX: i64 = 32;
+/// Default capacity for the transition queue.
+pub const DEFAULT_TRANSITION_QUEUE_CAPACITY: usize = 1000;
+/// Default send timeout for transition queue enqueue attempts, in milliseconds.
+pub const DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS: usize = 100;
+/// Test-only fault injection env var that forces the immediate transition enqueue timeout path.
+pub const ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT: &str = "RUSTFS_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT";
+/// Runtime env var controlling the transition worker count.
+pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
+/// Runtime env var controlling the absolute maximum transition workers.
+pub const ENV_TRANSITION_WORKERS_ABSOLUTE_MAX: &str = "RUSTFS_ABSOLUTE_MAX_WORKERS";
+/// Runtime env var controlling the transition queue capacity.
+pub const ENV_TRANSITION_QUEUE_CAPACITY: &str = "RUSTFS_TRANSITION_QUEUE_CAPACITY";
+/// Runtime env var controlling the transition queue send timeout in milliseconds.
+pub const ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS: &str = "RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS";
+
 // Allocator reclaim configuration
 pub const ENV_ALLOCATOR_RECLAIM_ENABLED: &str = "RUSTFS_ALLOCATOR_RECLAIM_ENABLED";
 pub const ENV_ALLOCATOR_RECLAIM_INTERVAL_SECS: &str = "RUSTFS_ALLOCATOR_RECLAIM_INTERVAL_SECS";

crates/crypto/Cargo.toml

@@ -33,8 +33,11 @@ aes-gcm = { workspace = true, optional = true }
 argon2 = { workspace = true, optional = true }
 chacha20poly1305 = { workspace = true, optional = true }
 jsonwebtoken = { workspace = true }
+base64-simd = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }
 rand = { workspace = true, optional = true }
+rsa = { workspace = true, features = ["sha2"] }
+serde = { workspace = true, features = ["derive"] }
 sha2 = { workspace = true, optional = true }
 thiserror.workspace = true
 serde_json.workspace = true

crates/crypto/src/lib.rs

@@ -16,6 +16,7 @@
 mod encdec;
 mod error;
 mod jwt;
+pub mod license_token;
 
 pub use encdec::decrypt::decrypt_data;
 pub use encdec::encrypt::encrypt_data;
@@ -25,3 +26,4 @@ pub use encdec::stream_io::{decrypt_stream_io, encrypt_stream_io};
 pub use error::Error;
 pub use jwt::decode::decode as jwt_decode;
 pub use jwt::encode::encode as jwt_encode;
+pub use license_token::{Token, parse_license_with_public_key, parse_signed_license_token, sign_license_token};

crates/crypto/src/license_token.rs

@@ -115,12 +115,20 @@ mod tests {
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
 
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600, // 1 hour from now
         };
 
         let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
@@ -139,12 +147,20 @@ mod tests {
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
 
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600,
         };
 
         let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
@@ -159,11 +175,19 @@ mod tests {
         let mut rng = rand::rng();
         let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600,
         };
 
         let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
@@ -181,7 +205,7 @@ mod tests {
 
     #[test]
     fn test_source_does_not_embed_private_key() {
-        let source = include_str!("token.rs");
+        let source = include_str!("license_token.rs");
         let forbidden = ["BEGIN", "PRIVATE KEY"].join(" ");
 
         assert!(!source.contains(&forbidden));
@@ -192,7 +216,9 @@ mod tests {
         let mut rng = rand::rng();
         let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let invalid_token = "invalid_base64_token";
         let result = parse_signed_license_token(invalid_token, &public_key_pem);
@@ -204,7 +230,11 @@ mod tests {
     fn test_sign_license_token_with_invalid_signing_key() {
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600, // 1 hour from now
         };
 
         let invalid_key = "invalid_private_key";

crates/data-usage/Cargo.toml

@@ -13,24 +13,26 @@
 # limitations under the License.
 
 [package]
-name = "rustfs-workers"
+name = "rustfs-data-usage"
+version.workspace = true
 edition.workspace = true
 license.workspace = true
 repository.workspace = true
 rust-version.workspace = true
-version.workspace = true
 homepage.workspace = true
-description = "Workers for RustFS, providing background processing capabilities for tasks such as data synchronization, indexing, and more."
-keywords = ["workers", "tasks", "rustfs", "Minio"]
-categories = ["web-programming", "development-tools"]
-documentation = "https://docs.rs/rustfs-workers/latest/rustfs_workers/"
+description = "Shared data usage models and algorithms for RustFS"
+keywords = ["rustfs", "data-usage", "cache", "histogram"]
+categories = ["data-structures", "filesystem"]
 
 [lints]
 workspace = true
 
 [dependencies]
-tokio = { workspace = true, features = ["sync", "time", "macros"] }
-tracing.workspace = true
+serde = { workspace = true }
+path-clean = { workspace = true }
+rmp-serde = { workspace = true }
+async-trait = { workspace = true }
+rustfs-filemeta = { workspace = true }
 
 [lib]
 doctest = false

crates/data-usage/src/data_usage.rs

@@ -312,6 +312,12 @@ impl SizeHistogram {
         }
         res
     }
+
+    pub fn merge_from(&mut self, other: &Self) {
+        for (dst, src) in self.0.iter_mut().zip(other.0.iter()) {
+            *dst += src;
+        }
+    }
 }
 
 /// Versions histogram for version count distribution
@@ -361,6 +367,12 @@ impl VersionsHistogram {
         }
         res
     }
+
+    pub fn merge_from(&mut self, other: &Self) {
+        for (dst, src) in self.0.iter_mut().zip(other.0.iter()) {
+            *dst += src;
+        }
+    }
 }
 
 /// Replication statistics for a single target
@@ -419,6 +431,9 @@ pub struct DataUsageEntry {
     pub obj_versions: VersionsHistogram,
     pub replication_stats: Option<ReplicationAllStats>,
     pub compacted: bool,
+    /// Number of objects that failed to scan (e.g., IO errors)
+    #[serde(default)]
+    pub failed_objects: usize,
 }
 
 impl DataUsageEntry {
@@ -456,6 +471,7 @@ impl DataUsageEntry {
         self.versions += other.versions;
         self.delete_markers += other.delete_markers;
         self.size += other.size;
+        self.failed_objects += other.failed_objects;
 
         if let Some(o_rep) = &other.replication_stats {
             let s_rep = self.replication_stats.get_or_insert_with(ReplicationAllStats::default);
@@ -490,9 +506,11 @@ impl DataUsageEntry {
 #[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageCacheInfo {
     pub name: String,
-    pub next_cycle: u32,
+    pub next_cycle: u64,
     pub last_update: Option<SystemTime>,
     pub skip_healing: bool,
+    #[serde(default)]
+    pub failed_objects: HashMap<String, u64>,
 }
 
 /// Data usage cache

crates/data-usage/src/lib.rs

@@ -12,4 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-pub mod workers;
+pub mod data_usage;
+
+pub use data_usage::*;

crates/e2e_test/Cargo.toml

@@ -31,7 +31,7 @@ sftp = []
 [dependencies]
 rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-ecstore.workspace = true
-rustfs-common.workspace = true
+rustfs-data-usage.workspace = true
 rustfs-rio.workspace = true
 flatbuffers.workspace = true
 futures.workspace = true

crates/e2e_test/src/data_usage_test.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use aws_sdk_s3::primitives::ByteStream;
-use rustfs_common::data_usage::DataUsageInfo;
+use rustfs_data_usage::DataUsageInfo;
 use serial_test::serial;
 
 use crate::common::{RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};

crates/e2e_test/src/kms/README.md

@@ -244,9 +244,9 @@ Designed to run inside CI/CD pipelines:
 
 ## 📚 References
 
-- [KMS configuration guide](../../../../docs/kms/README.md)
-- [Dynamic configuration API](../../../../docs/kms/http-api.md)
-- [Troubleshooting](../../../../docs/kms/troubleshooting.md)
+- [KMS configuration types](../../../kms/src/config.rs)
+- [Dynamic configuration API handlers](../../../../rustfs/src/admin/handlers/kms_dynamic.rs)
+- [KMS management API handlers](../../../../rustfs/src/admin/handlers/kms_management.rs)
 
 ---
 

crates/e2e_test/src/protocols/sftp_compliance.rs

@@ -39,7 +39,7 @@
 use crate::protocols::sftp_compliance_tests::{
     cmptst_01, cmptst_02, cmptst_03, cmptst_04, cmptst_05, cmptst_06, cmptst_07, cmptst_08, cmptst_09, cmptst_10, cmptst_11,
     cmptst_12, cmptst_13, cmptst_14, cmptst_15, cmptst_16, cmptst_17, cmptst_18, cmptst_19, cmptst_20, cmptst_21, cmptst_22,
-    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, spawn_compliance_rustfs,
+    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, cmptst_34, spawn_compliance_rustfs,
 };
 #[cfg(target_os = "linux")]
 use crate::protocols::sftp_compliance_tests::{cmptst_24, cmptst_25, cmptst_26};
@@ -96,6 +96,15 @@ pub async fn test_sftp_compliance_suite() -> Result<()> {
         cmptst_13::run_implicit_dir_round_trip(&sftp).await?;
         cmptst_14::run_winscp_setstat_shape_on_handle(&sftp).await?;
 
+        // CMPTST-34 cross-checks the SFTP streaming-multipart write
+        // path against the S3 layer. The OPEN-time FileAttributes must
+        // reach the finalised object as x-amz-meta-* user metadata
+        // through the CreateMultipartUpload input field. The S3 client
+        // connects to the same rustfs process this suite already drives.
+        let s3 = build_test_s3_client(&format!("http://{COMPLIANCE_RW_S3_ADDRESS}"));
+        wait_for_s3_ready(&s3, 30).await?;
+        cmptst_34::run_open_attrs_round_trip_multipart(&sftp, &s3).await?;
+
         drop(sftp);
         session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
         info!("SFTP compliance suite passed");

crates/e2e_test/src/protocols/sftp_compliance_tests.rs

@@ -104,6 +104,11 @@
 //!   byte-exact with the production cache window.
 //! - CMPTST-33: read-cache disabled regression, 8 MiB download
 //!   byte-exact with RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES=0.
+//! - CMPTST-34: OPEN with non-default FileAttributes followed by a
+//!   payload that crosses the 5 MiB multipart boundary preserves the
+//!   client-supplied mtime and permissions through the streaming
+//!   CreateMultipartUpload path. HeadObject through aws-sdk-s3
+//!   confirms the metadata reached the finalised S3 object.
 
 use crate::common::rustfs_binary_path_with_features;
 use crate::protocols::sftp_helpers::{
@@ -3276,6 +3281,80 @@ pub(crate) mod cmptst_33 {
     }
 }
 
+// CMPTST-34: OPEN-time client attrs preservation across the streaming
+// multipart write path. The payload crosses the 5 MiB part-size
+// boundary so the driver transitions Buffering -> Streaming and
+// finalises via CompleteMultipartUpload. The OPEN-supplied mtime and
+// permissions must reach the resulting object as x-amz-meta-mtime and
+// x-amz-meta-mode. The S3 client connects to the same rustfs process
+// the shared-server suite already drives.
+pub(crate) mod cmptst_34 {
+    use super::*;
+
+    const COMPLIANCE_TEST_OUTPUT_ID: &str = "CMPTST-34";
+    const REQUESTED_MTIME: u32 = 1_715_000_010;
+    const REQUESTED_MODE: u32 = 0o600;
+
+    pub(crate) async fn run_open_attrs_round_trip_multipart(sftp: &SftpSession, s3: &S3Client) -> Result<()> {
+        info!("{COMPLIANCE_TEST_OUTPUT_ID}: OPEN with mtime + mode, multi-part payload, streaming path");
+        let bucket = "complopenattrsmpbucket";
+        let bucket_path = format!("/{bucket}");
+        sftp.create_dir(&bucket_path).await?;
+
+        let path = format!("/{bucket}/attr-mp.bin");
+        // 6 MiB exceeds the 5 MiB part-size boundary so the streaming
+        // path runs at least one full UploadPart before the CLOSE-time
+        // CompleteMultipartUpload finalises the object.
+        let payload = vec![0xA5u8; 6 * 1024 * 1024];
+
+        let client_attrs = FileAttributes {
+            mtime: Some(REQUESTED_MTIME),
+            atime: Some(REQUESTED_MTIME),
+            permissions: Some(REQUESTED_MODE),
+            ..FileAttributes::default()
+        };
+
+        let mut writer = sftp
+            .open_with_flags_and_attributes(&path, OpenFlags::CREATE | OpenFlags::TRUNCATE | OpenFlags::WRITE, client_attrs)
+            .await?;
+        writer.write_all(&payload).await?;
+        writer.flush().await?;
+        writer.shutdown().await?;
+
+        let head = s3
+            .head_object()
+            .bucket(bucket)
+            .key("attr-mp.bin")
+            .send()
+            .await
+            .map_err(|e| anyhow!("S3 HeadObject failed: {e:?}"))?;
+        let content_length = head.content_length().unwrap_or(0);
+        if content_length != payload.len() as i64 {
+            return Err(anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} unexpected size: got {content_length} bytes"));
+        }
+        let metadata = head
+            .metadata()
+            .ok_or_else(|| anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} HeadObject returned no metadata map"))?;
+        let mtime_value = metadata
+            .get("mtime")
+            .ok_or_else(|| anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} mtime key missing on the object"))?;
+        if mtime_value != &REQUESTED_MTIME.to_string() {
+            return Err(anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} mtime mismatch: got {mtime_value}"));
+        }
+        let mode_value = metadata
+            .get("mode")
+            .ok_or_else(|| anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} mode key missing on the object"))?;
+        if mode_value != &REQUESTED_MODE.to_string() {
+            return Err(anyhow!("{COMPLIANCE_TEST_OUTPUT_ID} mode mismatch: got {mode_value}"));
+        }
+
+        sftp.remove_file(&path).await?;
+        sftp.remove_dir(&bucket_path).await?;
+        info!("PASS {COMPLIANCE_TEST_OUTPUT_ID}: multipart upload preserved mtime + mode end to end");
+        Ok(())
+    }
+}
+
 // Shared parameters for CMPTST-32 (cache enabled) and CMPTST-33 (cache
 // disabled). Both cases seed the same fixture and download it
 // end-to-end, then assert byte-count and SHA256 against the

crates/e2e_test/src/replication_extension_test.rs

@@ -651,6 +651,22 @@ async fn list_service_accounts(
     Ok(response.json().await?)
 }
 
+async fn get_account_info(
+    env: &RustFSTestEnvironment,
+    signer_access_key: &str,
+    signer_secret_key: &str,
+) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
+    let url = format!("{}/rustfs/admin/v3/accountinfo", env.url);
+    let response = signed_request(http::Method::GET, &url, signer_access_key, signer_secret_key, None, None).await?;
+    if response.status() != StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        return Err(format!("account info failed: {status} {body}").into());
+    }
+
+    Ok(response.json().await?)
+}
+
 async fn wait_for_service_accounts(
     env: &RustFSTestEnvironment,
     signer_access_key: &str,
@@ -2379,6 +2395,57 @@ async fn test_site_replication_replicates_group_policy_backed_access_real_dual_n
     Ok(())
 }
 
+#[tokio::test]
+#[serial]
+async fn test_service_account_policy_from_accountinfo_round_trips_real_single_node() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let account_info = get_account_info(&env, &env.access_key, &env.secret_key).await?;
+    let policy_str = account_info
+        .get("policy")
+        .and_then(|value| value.as_str())
+        .ok_or("account info policy should be a JSON string")?;
+
+    let policy: serde_json::Value = serde_json::from_str(policy_str)?;
+    let statements = policy
+        .get("Statement")
+        .and_then(|value| value.as_array())
+        .ok_or("account info policy should include Statement array")?;
+
+    assert!(!statements.is_empty(), "account info policy Statement should not be empty: {policy}");
+
+    let req = AddServiceAccountReq {
+        policy: Some(policy),
+        target_user: None,
+        access_key: "svcacct-info-sample".to_string(),
+        secret_key: "svcacct-info-sample-secret-key-123456".to_string(),
+        name: Some("svcacct-info-sample".to_string()),
+        description: Some("service account created from accountinfo sample policy".to_string()),
+        expiration: None,
+        comment: None,
+    };
+
+    let created = add_service_account(&env, &env.access_key, &env.secret_key, &req).await?;
+    assert_eq!(created.0, "svcacct-info-sample");
+
+    let listed =
+        wait_for_service_accounts(&env, &env.access_key, &env.secret_key, Some(&env.access_key), &["svcacct-info-sample"])
+            .await?;
+    assert!(
+        listed
+            .accounts
+            .iter()
+            .any(|account| account.access_key == "svcacct-info-sample"),
+        "created service account should be listed for parent user: {:?}",
+        listed.accounts
+    );
+
+    Ok(())
+}
+
 #[tokio::test]
 #[serial]
 async fn test_site_replication_replicates_multiple_service_accounts_real_dual_node() -> Result<(), Box<dyn Error + Send + Sync>> {

crates/ecstore/Cargo.toml

@@ -45,7 +45,9 @@ rustfs-common.workspace = true
 rustfs-policy.workspace = true
 rustfs-protos.workspace = true
 rustfs-kms.workspace = true
-rustfs-s3-common = { workspace = true }
+rustfs-s3-types = { workspace = true }
+rustfs-data-usage.workspace = true
+rustfs-object-capacity.workspace = true
 async-trait.workspace = true
 bytes.workspace = true
 byteorder = { workspace = true }
@@ -103,7 +105,7 @@ memmap2 = { workspace = true }
 libc.workspace = true
 rustix = { workspace = true }
 rustfs-madmin.workspace = true
-rustfs-workers.workspace = true
+rustfs-concurrency.workspace = true
 reqwest = { workspace = true }
 aes-gcm.workspace = true
 aws-sdk-s3 = { workspace = true }
@@ -145,5 +147,9 @@ harness = false
 name = "comparison_benchmark"
 harness = false
 
+[[bench]]
+name = "rename_data_meta_benchmark"
+harness = false
+
 [lib]
 doctest = false

crates/ecstore/benches/rename_data_meta_benchmark.rs

@@ -0,0 +1,118 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main};
+use rustfs_filemeta::{ErasureAlgo, FileInfo, FileMeta};
+use std::hint::black_box;
+use std::time::Duration;
+use time::OffsetDateTime;
+use uuid::Uuid;
+
+const VERSION_COUNT_CASES: &[usize] = &[1, 8, 32, 64];
+const BENCH_BASE_TIME_UNIX_SECS: i64 = 1_700_000_000;
+
+fn make_file_info(version_id: Uuid, data_dir: Uuid, size: i64, mod_time: OffsetDateTime) -> FileInfo {
+    FileInfo {
+        version_id: Some(version_id),
+        data_dir: Some(data_dir),
+        size,
+        mod_time: Some(mod_time),
+        metadata: [("etag".to_string(), format!("etag-{version_id}"))].into_iter().collect(),
+        erasure: rustfs_filemeta::ErasureInfo {
+            algorithm: ErasureAlgo::ReedSolomon.to_string(),
+            data_blocks: 4,
+            parity_blocks: 2,
+            block_size: 1024 * 1024,
+            index: 1,
+            distribution: vec![1, 2, 3, 4, 5, 6],
+            ..Default::default()
+        },
+        ..Default::default()
+    }
+}
+
+fn build_meta_with_versions(version_count: usize) -> FileMeta {
+    let mut meta = FileMeta::new();
+    let base_time = OffsetDateTime::from_unix_timestamp(BENCH_BASE_TIME_UNIX_SECS).expect("valid bench base timestamp");
+    for i in 0..version_count {
+        let fi = make_file_info(Uuid::new_v4(), Uuid::new_v4(), 64 * 1024, base_time - Duration::from_secs(i as u64));
+        meta.add_version(fi).expect("seed add_version should succeed");
+    }
+    meta
+}
+
+fn bench_rename_data_meta_path(c: &mut Criterion) {
+    let mut group = c.benchmark_group("rename_data_meta");
+    group.sample_size(20);
+    group.measurement_time(Duration::from_secs(10));
+
+    for &version_count in VERSION_COUNT_CASES {
+        let seeded = build_meta_with_versions(version_count);
+        let dst_buf = seeded.marshal_msg().expect("marshal seeded meta");
+        let base_time = OffsetDateTime::from_unix_timestamp(BENCH_BASE_TIME_UNIX_SECS).expect("valid bench base timestamp");
+        let replace_version_id = seeded
+            .versions
+            .first()
+            .and_then(|v| v.header.version_id)
+            .unwrap_or(Uuid::nil());
+
+        group.bench_with_input(BenchmarkId::new("read_modify_write", version_count), &version_count, |b, _| {
+            b.iter(|| {
+                let mut xlmeta = FileMeta::load(black_box(&dst_buf)).expect("load dst meta");
+                let search_version_id = Some(replace_version_id);
+                let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(search_version_id);
+                if let Some(old_data_dir) = has_old_data_dir {
+                    let _ = xlmeta.data.remove_two(replace_version_id, old_data_dir);
+                }
+                let fi = make_file_info(replace_version_id, Uuid::new_v4(), 64 * 1024, base_time + Duration::from_millis(1));
+                xlmeta.add_version(fi).expect("add new version");
+                let out = xlmeta.marshal_msg().expect("marshal updated meta");
+                black_box(out);
+            });
+        });
+
+        let mut prepared = FileMeta::load(&dst_buf).expect("load prepared meta");
+        if let Some(old_data_dir) = prepared.find_unshared_data_dir_for_version(Some(replace_version_id)) {
+            let _ = prepared.data.remove_two(replace_version_id, old_data_dir);
+        }
+        group.bench_with_input(BenchmarkId::new("add_version_marshal_only", version_count), &version_count, |b, _| {
+            b.iter_batched(
+                || prepared.clone(),
+                |mut xlmeta| {
+                    let fi = make_file_info(replace_version_id, Uuid::new_v4(), 64 * 1024, base_time + Duration::from_millis(1));
+                    xlmeta.add_version(fi).expect("add new version");
+                    let out = xlmeta.marshal_msg().expect("marshal updated meta");
+                    black_box(out);
+                },
+                BatchSize::SmallInput,
+            );
+        });
+
+        group.bench_with_input(BenchmarkId::new("remove_two_only", version_count), &version_count, |b, _| {
+            b.iter(|| {
+                let mut xlmeta = FileMeta::load(black_box(&dst_buf)).expect("load dst meta");
+                let removed = if let Some(old_data_dir) = xlmeta.find_unshared_data_dir_for_version(Some(replace_version_id)) {
+                    xlmeta.data.remove_two(replace_version_id, old_data_dir).expect("remove two")
+                } else {
+                    false
+                };
+                black_box(removed);
+                black_box(xlmeta);
+            });
+        });
+    }
+}
+
+criterion_group!(benches, bench_rename_data_meta_path);
+criterion_main!(benches);

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -44,14 +44,19 @@ use async_channel::{Receiver as A_Receiver, Sender as A_Sender, bounded};
 use futures::Future;
 use http::HeaderMap;
 use lazy_static::lazy_static;
-use rustfs_common::data_usage::TierStats;
 use rustfs_common::heal_channel::rep_has_active_rules;
 use rustfs_common::metrics::{IlmAction, Metrics};
+use rustfs_config::{
+    DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
+    DEFAULT_TRANSITION_WORKERS_CAP, ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, ENV_TRANSITION_QUEUE_CAPACITY,
+    ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, ENV_TRANSITION_WORKERS, ENV_TRANSITION_WORKERS_ABSOLUTE_MAX,
+};
+use rustfs_data_usage::TierStats;
 use rustfs_filemeta::{
     FileInfo, FileInfoOpts, NULL_VERSION_ID, REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicationState, RestoreStatusOps,
     VersionPurgeStatusType, get_file_info, is_restored_object_on_disk,
 };
-use rustfs_s3_common::EventName;
+use rustfs_s3_types::EventName;
 use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::strings_has_prefix_fold};
 use s3s::dto::{
     BucketLifecycleConfiguration, DefaultRetention, ExpirationStatus, ReplicationConfiguration, RestoreRequest,
@@ -60,7 +65,7 @@ use s3s::dto::{
 use s3s::header::{X_AMZ_RESTORE, X_AMZ_SERVER_SIDE_ENCRYPTION};
 use sha2::{Digest, Sha256};
 use std::any::Any;
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
 use std::env;
 use std::pin::Pin;
 use std::sync::atomic::{AtomicI64, Ordering};
@@ -99,6 +104,57 @@ lazy_static! {
     pub static ref GLOBAL_TransitionState: Arc<TransitionState> = TransitionState::new();
 }
 
+fn resolve_transition_worker_count() -> (i64, i64, i64) {
+    let fallback = std::cmp::min(num_cpus::get() as i64, DEFAULT_TRANSITION_WORKERS_CAP);
+    let configured = env::var(ENV_TRANSITION_WORKERS)
+        .ok()
+        .and_then(|value| value.parse::<i64>().ok())
+        .filter(|value| *value > 0)
+        .unwrap_or(fallback);
+    let mut effective = configured;
+    let absolute_max = resolve_transition_workers_absolute_max();
+    effective = std::cmp::min(effective, absolute_max);
+    (configured, absolute_max, effective)
+}
+
+fn resolve_transition_workers_absolute_max() -> i64 {
+    let absolute_max = get_env_i64(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX);
+    if absolute_max > 0 {
+        absolute_max
+    } else {
+        DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX
+    }
+}
+
+fn resolve_transition_queue_capacity() -> usize {
+    get_env_usize(ENV_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_QUEUE_CAPACITY).max(1)
+}
+
+fn resolve_transition_queue_send_timeout() -> StdDuration {
+    StdDuration::from_millis(
+        get_env_usize(ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS, DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS).max(1) as u64,
+    )
+}
+
+fn is_immediate_transition_source(src: &LcEventSrc) -> bool {
+    matches!(
+        src,
+        LcEventSrc::S3PutObject | LcEventSrc::S3CopyObject | LcEventSrc::S3CompleteMultipartUpload
+    )
+}
+
+#[cfg(any(test, debug_assertions))]
+fn should_force_immediate_transition_enqueue_timeout() -> bool {
+    env::var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT)
+        .ok()
+        .is_some_and(|value| value == "1")
+}
+
+#[cfg(not(any(test, debug_assertions)))]
+fn should_force_immediate_transition_enqueue_timeout() -> bool {
+    false
+}
+
 pub struct LifecycleSys;
 
 impl LifecycleSys {
@@ -537,15 +593,34 @@ pub struct TransitionState {
     pub num_workers: AtomicI64,
     kill_tx: A_Sender<()>,
     kill_rx: A_Receiver<()>,
+    transition_queue_capacity: usize,
+    transition_queue_send_timeout: StdDuration,
     active_tasks: AtomicI64,
     missed_immediate_tasks: AtomicI64,
+    queue_full_tasks: AtomicI64,
+    queue_send_timeout_tasks: AtomicI64,
+    compensation_scheduled_tasks: AtomicI64,
+    compensation_running_tasks: AtomicI64,
+    compensation_buckets: Arc<Mutex<HashSet<String>>>,
     last_day_stats: Arc<Mutex<HashMap<String, LastDayTierStats>>>,
 }
 
+enum ImmediateEnqueueFailure {
+    ForcedTimeout,
+    QueueClosed { timeout_ms: Option<u64> },
+    QueueSendTimedOut { timeout_ms: u64 },
+}
+
 impl TransitionState {
     #[allow(clippy::new_ret_no_self)]
     pub fn new() -> Arc<Self> {
-        let (tx1, rx1) = bounded(1000);
+        Self::new_with_capacity(resolve_transition_queue_capacity())
+    }
+
+    fn new_with_capacity(capacity: usize) -> Arc<Self> {
+        let capacity = capacity.max(1);
+        let queue_send_timeout = resolve_transition_queue_send_timeout();
+        let (tx1, rx1) = bounded(capacity);
         let (tx2, rx2) = bounded(1);
         Arc::new(Self {
             transition_tx: tx1,
@@ -553,39 +628,191 @@ impl TransitionState {
             num_workers: AtomicI64::new(0),
             kill_tx: tx2,
             kill_rx: rx2,
+            transition_queue_capacity: capacity,
+            transition_queue_send_timeout: queue_send_timeout,
             active_tasks: AtomicI64::new(0),
             missed_immediate_tasks: AtomicI64::new(0),
+            queue_full_tasks: AtomicI64::new(0),
+            queue_send_timeout_tasks: AtomicI64::new(0),
+            compensation_scheduled_tasks: AtomicI64::new(0),
+            compensation_running_tasks: AtomicI64::new(0),
+            compensation_buckets: Arc::new(Mutex::new(HashSet::new())),
             last_day_stats: Arc::new(Mutex::new(HashMap::new())),
         })
     }
 
-    pub async fn queue_transition_task(&self, oi: &ObjectInfo, event: &lifecycle::Event, src: &LcEventSrc) {
+    fn schedule_bucket_compensation(self: &Arc<Self>, bucket: &str) -> bool {
+        let mut scheduled = self.compensation_buckets.lock().unwrap();
+        if !scheduled.insert(bucket.to_string()) {
+            return false;
+        }
+        Self::inc_counter(&self.compensation_scheduled_tasks);
+        let bucket = bucket.to_string();
+        let scheduled = Arc::clone(&self.compensation_buckets);
+        let state = Arc::clone(self);
+        tokio::spawn(async move {
+            Self::inc_counter(&state.compensation_running_tasks);
+            let Some(api) = crate::new_object_layer_fn() else {
+                scheduled.lock().unwrap().remove(&bucket);
+                Self::add_counter(&state.compensation_running_tasks, -1);
+                warn!(bucket = %bucket, "transition compensation skipped because object layer is unavailable");
+                return;
+            };
+
+            if let Err(err) = enqueue_transition_for_existing_objects(api, &bucket).await {
+                warn!(bucket = %bucket, error = ?err, "transition compensation backfill failed");
+            } else {
+                info!(bucket = %bucket, "transition compensation backfill completed");
+            }
+
+            scheduled.lock().unwrap().remove(&bucket);
+            Self::add_counter(&state.compensation_running_tasks, -1);
+        });
+        true
+    }
+
+    #[inline]
+    fn inc_counter(counter: &AtomicI64) {
+        Self::add_counter(counter, 1);
+    }
+
+    #[inline]
+    fn add_counter(counter: &AtomicI64, delta: i64) {
+        counter.fetch_add(delta, Ordering::Relaxed);
+    }
+
+    #[inline]
+    fn counter_value(counter: &AtomicI64) -> i64 {
+        counter.load(Ordering::Relaxed)
+    }
+
+    fn handle_immediate_enqueue_failure(self: &Arc<Self>, oi: &ObjectInfo, src: &LcEventSrc, failure: ImmediateEnqueueFailure) {
+        Self::inc_counter(&self.missed_immediate_tasks);
+        let scheduled = self.schedule_bucket_compensation(&oi.bucket);
+        match failure {
+            ImmediateEnqueueFailure::ForcedTimeout => {
+                Self::inc_counter(&self.queue_send_timeout_tasks);
+                warn!(
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    source = ?src,
+                    compensation_scheduled = scheduled,
+                    "transition enqueue forced into timeout path for test fault injection"
+                );
+            }
+            ImmediateEnqueueFailure::QueueClosed { timeout_ms } => match timeout_ms {
+                Some(timeout_ms) => {
+                    warn!(
+                        bucket = %oi.bucket,
+                        object = %oi.name,
+                        source = ?src,
+                        timeout_ms,
+                        compensation_scheduled = scheduled,
+                        "transition enqueue failed because the queue is closed"
+                    );
+                }
+                None => {
+                    warn!(
+                        bucket = %oi.bucket,
+                        object = %oi.name,
+                        source = ?src,
+                        compensation_scheduled = scheduled,
+                        "transition enqueue failed because the queue is closed"
+                    );
+                }
+            },
+            ImmediateEnqueueFailure::QueueSendTimedOut { timeout_ms } => {
+                Self::inc_counter(&self.queue_send_timeout_tasks);
+                warn!(
+                    bucket = %oi.bucket,
+                    object = %oi.name,
+                    source = ?src,
+                    timeout_ms,
+                    compensation_scheduled = scheduled,
+                    "transition enqueue timed out under backpressure"
+                );
+            }
+        }
+    }
+
+    pub async fn queue_transition_task(self: &Arc<Self>, oi: &ObjectInfo, event: &lifecycle::Event, src: &LcEventSrc) {
+        if is_immediate_transition_source(src) && should_force_immediate_transition_enqueue_timeout() {
+            self.handle_immediate_enqueue_failure(oi, src, ImmediateEnqueueFailure::ForcedTimeout);
+            return;
+        }
+
         let task = TransitionTask {
             obj_info: oi.clone(),
             src: src.clone(),
             event: event.clone(),
         };
-        select! {
-            //_ -> t.ctx.Done() => (),
-            _ = self.transition_tx.send(Some(task)) => (),
-            else => {
-                match src {
-                    LcEventSrc::S3PutObject | LcEventSrc::S3CopyObject | LcEventSrc::S3CompleteMultipartUpload => {
-                        self.missed_immediate_tasks.fetch_add(1, Ordering::SeqCst);
+        if is_immediate_transition_source(src) {
+            match self.transition_tx.try_send(Some(task)) {
+                Ok(()) => {}
+                Err(async_channel::TrySendError::Full(task)) => {
+                    Self::inc_counter(&self.queue_full_tasks);
+                    let send_timeout = self.transition_queue_send_timeout;
+                    match tokio::time::timeout(send_timeout, self.transition_tx.send(task)).await {
+                        Ok(Ok(())) => {}
+                        Ok(Err(_)) => {
+                            self.handle_immediate_enqueue_failure(
+                                oi,
+                                src,
+                                ImmediateEnqueueFailure::QueueClosed {
+                                    timeout_ms: Some(send_timeout.as_millis() as u64),
+                                },
+                            );
+                        }
+                        Err(_) => {
+                            self.handle_immediate_enqueue_failure(
+                                oi,
+                                src,
+                                ImmediateEnqueueFailure::QueueSendTimedOut {
+                                    timeout_ms: send_timeout.as_millis() as u64,
+                                },
+                            );
+                        }
                     }
-                    _ => ()
                 }
-            },
+                Err(async_channel::TrySendError::Closed(_task)) => {
+                    self.handle_immediate_enqueue_failure(oi, src, ImmediateEnqueueFailure::QueueClosed { timeout_ms: None });
+                }
+            }
+            return;
+        }
+
+        if let Err(err) = self.transition_tx.try_send(Some(task)) {
+            match err {
+                async_channel::TrySendError::Full(_) => {
+                    debug!(
+                        bucket = %oi.bucket,
+                        object = %oi.name,
+                        source = ?src,
+                        "transition queue is full; deferring to scanner/backfill"
+                    );
+                }
+                async_channel::TrySendError::Closed(_) => {
+                    warn!(
+                        bucket = %oi.bucket,
+                        object = %oi.name,
+                        source = ?src,
+                        "transition enqueue failed because the queue is closed"
+                    );
+                }
+            }
         }
     }
 
     pub async fn init(api: Arc<ECStore>) {
-        let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
-        let mut n = max_workers;
-        let tw = 8; //globalILMConfig.getTransitionWorkers();
-        if tw > 0 {
-            n = tw;
-        }
+        let (configured, absolute_max, n) = resolve_transition_worker_count();
+        info!(
+            configured_transition_workers = configured,
+            absolute_max_workers = absolute_max,
+            effective_transition_workers = n,
+            transition_queue_capacity = GLOBAL_TransitionState.transition_queue_capacity,
+            transition_queue_send_timeout_ms = GLOBAL_TransitionState.transition_queue_send_timeout.as_millis() as u64,
+            "transition worker count resolved"
+        );
 
         //let mut transition_state = GLOBAL_TransitionState.write().await;
         //self.objAPI = objAPI
@@ -599,11 +826,27 @@ impl TransitionState {
     }
 
     pub fn active_tasks(&self) -> i64 {
-        self.active_tasks.load(Ordering::SeqCst)
+        Self::counter_value(&self.active_tasks)
     }
 
     pub fn missed_immediate_tasks(&self) -> i64 {
-        self.missed_immediate_tasks.load(Ordering::SeqCst)
+        Self::counter_value(&self.missed_immediate_tasks)
+    }
+
+    pub fn queue_full_tasks(&self) -> i64 {
+        Self::counter_value(&self.queue_full_tasks)
+    }
+
+    pub fn queue_send_timeout_tasks(&self) -> i64 {
+        Self::counter_value(&self.queue_send_timeout_tasks)
+    }
+
+    pub fn compensation_scheduled_tasks(&self) -> i64 {
+        Self::counter_value(&self.compensation_scheduled_tasks)
+    }
+
+    pub fn compensation_running_tasks(&self) -> i64 {
+        Self::counter_value(&self.compensation_running_tasks)
     }
 
     pub async fn worker(api: Arc<ECStore>) {
@@ -626,7 +869,7 @@ impl TransitionState {
                     if task.as_any().is::<TransitionTask>() {
                         let task = task.as_any().downcast_ref::<TransitionTask>().expect("TransitionTask downcast failed");
 
-                        GLOBAL_TransitionState.active_tasks.fetch_add(1, Ordering::SeqCst);
+                        TransitionState::inc_counter(&GLOBAL_TransitionState.active_tasks);
 
                         let obj_info_for_event = ObjectInfo {
                             bucket: task.obj_info.bucket.clone(),
@@ -671,7 +914,7 @@ impl TransitionState {
                                 ..Default::default()
                             });
                         }
-                        GLOBAL_TransitionState.active_tasks.fetch_add(-1, Ordering::SeqCst);
+                        TransitionState::add_counter(&GLOBAL_TransitionState.active_tasks, -1);
                     }
                 }
                 else => ()
@@ -702,15 +945,17 @@ impl TransitionState {
 
     pub async fn update_workers_inner(api: Arc<ECStore>, n: i64) {
         let mut n = n;
+        let requested = n;
         if n == 0 {
-            let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
-            n = max_workers;
+            let (_, _, effective) = resolve_transition_worker_count();
+            n = effective;
         }
         // Allow environment override of maximum workers
-        let absolute_max = get_env_i64("RUSTFS_ABSOLUTE_MAX_WORKERS", 32);
+        let absolute_max = resolve_transition_workers_absolute_max();
         n = std::cmp::min(n, absolute_max);
 
-        let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        let previous_num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        let mut num_workers = previous_num_workers;
         while num_workers < n {
             let clone_api = api.clone();
             tokio::spawn(async move {
@@ -727,6 +972,15 @@ impl TransitionState {
             num_workers -= 1;
             GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
         }
+
+        info!(
+            requested_transition_workers = requested,
+            effective_transition_workers = n,
+            absolute_max_workers = absolute_max,
+            previous_transition_workers = previous_num_workers,
+            current_transition_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst),
+            "transition workers updated"
+        );
     }
 }
 
@@ -2004,11 +2258,13 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc,
 #[cfg(test)]
 mod tests {
     use super::{
-        DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, StaleMultipartUploadCandidate, cleanup_empty_multipart_sha_dirs_on_local_disks,
-        cleanup_stale_multipart_uploads_once_at, lifecycle_deleted_object, lifecycle_rule_has_date_expiration,
-        lifecycle_version_purge_state_from_completed_targets, mark_delete_opts_skip_decommissioned_on_remote_success,
-        merge_stale_multipart_candidate, replication_state_for_delete, should_defer_date_expiry_for_recent_config_update,
-        should_reuse_lifecycle_delete_replication_state,
+        DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
+        DEFAULT_TRANSITION_WORKERS_CAP, GLOBAL_TransitionState, StaleMultipartUploadCandidate, TransitionState,
+        cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at, lifecycle_deleted_object,
+        lifecycle_rule_has_date_expiration, lifecycle_version_purge_state_from_completed_targets,
+        mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate, replication_state_for_delete,
+        resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
+        should_defer_date_expiry_for_recent_config_update, should_reuse_lifecycle_delete_replication_state,
     };
     use crate::bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
     use crate::bucket::metadata_sys;
@@ -2021,12 +2277,16 @@ mod tests {
     use crate::store_api::{
         BucketOperations, BucketOptions, MakeBucketOptions, MultipartOperations, ObjectInfo, ObjectOptions, PutObjReader,
     };
+    use futures::FutureExt;
+    use rustfs_config::ENV_TRANSITION_WORKERS_ABSOLUTE_MAX;
     use rustfs_filemeta::{ReplicateDecision, VersionPurgeStatusType};
     use s3s::dto::{BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, Timestamp};
     use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::collections::HashMap;
+    use std::env;
     use std::path::PathBuf;
+    use std::sync::atomic::Ordering;
     use std::sync::{Arc, OnceLock};
     use std::time::Duration as StdDuration;
     use time::OffsetDateTime;
@@ -2043,6 +2303,216 @@ mod tests {
         assert!(opts.skip_decommissioned);
     }
 
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    fn with_transition_worker_env<F>(transition: Option<&str>, absolute: Option<&str>, test_fn: F)
+    where
+        F: FnOnce(),
+    {
+        let original_transition = env::var_os("RUSTFS_MAX_TRANSITION_WORKERS");
+        let original_absolute = env::var_os(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+
+        match transition {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_MAX_TRANSITION_WORKERS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_MAX_TRANSITION_WORKERS");
+            },
+        }
+        match absolute {
+            Some(value) => unsafe {
+                env::set_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            },
+        }
+
+        let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(test_fn));
+
+        match original_transition {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_MAX_TRANSITION_WORKERS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_MAX_TRANSITION_WORKERS");
+            },
+        }
+        match original_absolute {
+            Some(value) => unsafe {
+                env::set_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    async fn with_transition_worker_env_async<F, Fut>(transition: Option<&str>, absolute: Option<&str>, test_fn: F)
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = ()>,
+    {
+        let original_transition = env::var_os("RUSTFS_MAX_TRANSITION_WORKERS");
+        let original_absolute = env::var_os(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+
+        match transition {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_MAX_TRANSITION_WORKERS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_MAX_TRANSITION_WORKERS");
+            },
+        }
+        match absolute {
+            Some(value) => unsafe {
+                env::set_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            },
+        }
+
+        let result = std::panic::AssertUnwindSafe(test_fn()).catch_unwind().await;
+
+        match original_transition {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_MAX_TRANSITION_WORKERS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_MAX_TRANSITION_WORKERS");
+            },
+        }
+        match original_absolute {
+            Some(value) => unsafe {
+                env::set_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, value);
+            },
+            None => unsafe {
+                env::remove_var(ENV_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    fn with_transition_queue_env<F>(capacity: Option<&str>, timeout_ms: Option<&str>, test_fn: F)
+    where
+        F: FnOnce(),
+    {
+        let original_capacity = env::var_os("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+        let original_timeout = env::var_os("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+
+        match capacity {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_CAPACITY", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+            },
+        }
+        match timeout_ms {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+            },
+        }
+
+        let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(test_fn));
+
+        match original_capacity {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_CAPACITY", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+            },
+        }
+        match original_timeout {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
+    // SAFETY: this helper is only used from `#[serial]` tests and those tests run under a
+    // single-thread runtime (`worker_threads = 1`), so no concurrent reader/writer can access
+    // process environment while `env::set_var`/`env::remove_var` is active.
+    #[allow(unsafe_code)]
+    async fn with_transition_queue_env_async<F, Fut>(capacity: Option<&str>, timeout_ms: Option<&str>, test_fn: F)
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = ()>,
+    {
+        let original_capacity = env::var_os("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+        let original_timeout = env::var_os("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+
+        match capacity {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_CAPACITY", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+            },
+        }
+        match timeout_ms {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+            },
+        }
+
+        let result = std::panic::AssertUnwindSafe(test_fn()).catch_unwind().await;
+
+        match original_capacity {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_CAPACITY", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_CAPACITY");
+            },
+        }
+        match original_timeout {
+            Some(value) => unsafe {
+                env::set_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS", value);
+            },
+            None => unsafe {
+                env::remove_var("RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS");
+            },
+        }
+
+        if let Err(e) = result {
+            std::panic::resume_unwind(e);
+        }
+    }
+
     #[test]
     fn lifecycle_rule_has_date_expiration_detects_enabled_date_rule() {
         let lc = BucketLifecycleConfiguration {
@@ -2068,6 +2538,134 @@ mod tests {
         assert!(!lifecycle_rule_has_date_expiration(&lc, "missing-rule"));
     }
 
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_uses_fallback_when_env_missing() {
+        with_transition_worker_env(None, None, || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            let fallback = std::cmp::min(num_cpus::get() as i64, DEFAULT_TRANSITION_WORKERS_CAP);
+            assert_eq!(configured, fallback);
+            assert_eq!(absolute_max, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            assert_eq!(effective, fallback);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_honors_positive_env_value() {
+        with_transition_worker_env(Some("4"), Some("32"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            assert_eq!(configured, 4);
+            assert_eq!(absolute_max, 32);
+            assert_eq!(effective, 4);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_clamps_to_absolute_max() {
+        with_transition_worker_env(Some("64"), Some("16"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            assert_eq!(configured, 64);
+            assert_eq!(absolute_max, 16);
+            assert_eq!(effective, 16);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_ignores_non_positive_absolute_max() {
+        with_transition_worker_env(Some("4"), Some("0"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            assert_eq!(configured, 4);
+            assert_eq!(absolute_max, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            assert_eq!(effective, 4);
+        });
+
+        with_transition_worker_env(Some("4"), Some("-1"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            assert_eq!(configured, 4);
+            assert_eq!(absolute_max, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX);
+            assert_eq!(effective, 4);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_worker_count_falls_back_for_zero_value() {
+        with_transition_worker_env(Some("0"), Some("32"), || {
+            let (configured, absolute_max, effective) = resolve_transition_worker_count();
+
+            let fallback = std::cmp::min(num_cpus::get() as i64, DEFAULT_TRANSITION_WORKERS_CAP);
+            assert_eq!(configured, fallback);
+            assert_eq!(absolute_max, 32);
+            assert_eq!(effective, fallback);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_queue_capacity_uses_default_when_env_missing() {
+        with_transition_queue_env(None, None, || {
+            assert_eq!(resolve_transition_queue_capacity(), DEFAULT_TRANSITION_QUEUE_CAPACITY);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_queue_capacity_honors_positive_env_value() {
+        with_transition_queue_env(Some("128"), None, || {
+            assert_eq!(resolve_transition_queue_capacity(), 128);
+        });
+    }
+
+    #[test]
+    #[serial]
+    fn resolve_transition_queue_send_timeout_honors_positive_env_value() {
+        with_transition_queue_env(None, Some("250"), || {
+            assert_eq!(resolve_transition_queue_send_timeout(), StdDuration::from_millis(250));
+        });
+    }
+
+    #[tokio::test(flavor = "current_thread")]
+    async fn schedule_bucket_compensation_deduplicates_same_bucket() {
+        let state = TransitionState::new_with_capacity(1);
+
+        let first = state.schedule_bucket_compensation("bucket-a");
+        let second = state.schedule_bucket_compensation("bucket-a");
+
+        assert!(first);
+        assert!(!second);
+        assert_eq!(state.compensation_scheduled_tasks(), 1);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn transition_state_init_honors_runtime_configured_worker_count() {
+        let (_paths, ecstore) = setup_test_env().await;
+        let original_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        with_transition_worker_env_async(Some("3"), Some("8"), || async {
+            TransitionState::update_workers(ecstore.clone(), 0).await;
+            assert_eq!(GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst), 3);
+        })
+        .await;
+
+        let current_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
+        if original_workers > 0 {
+            TransitionState::update_workers(ecstore, original_workers).await;
+        } else {
+            for _ in 0..current_workers {
+                let _ = GLOBAL_TransitionState.kill_tx.send(()).await;
+                GLOBAL_TransitionState.num_workers.fetch_add(-1, Ordering::SeqCst);
+            }
+        }
+    }
+
     #[test]
     fn should_defer_date_expiry_for_recent_config_update_respects_grace_window() {
         let now = OffsetDateTime::now_utc();

crates/ecstore/src/bucket/lifecycle/tier_last_day_stats.rs

@@ -18,7 +18,7 @@
 #![allow(unused_must_use)]
 #![allow(clippy::all)]
 
-use rustfs_common::data_usage::TierStats;
+use rustfs_data_usage::TierStats;
 use sha2::Sha256;
 use std::collections::HashMap;
 use std::ops::Sub;

crates/ecstore/src/bucket/metadata.rs

@@ -724,7 +724,7 @@ impl BucketMetadata {
             return Err(Error::other("errServerNotInitialized"));
         };
 
-        self.parse_all_configs(store.clone())?;
+        self.parse_all_configs()?;
 
         let mut buf: Vec<u8> = vec![0; 4];
 
@@ -752,7 +752,7 @@ impl BucketMetadata {
         Ok(())
     }
 
-    fn parse_all_configs(&mut self, _api: Arc<ECStore>) -> Result<()> {
+    fn parse_all_configs(&mut self) -> Result<()> {
         if let Err(e) = self.parse_policy_config() {
             tracing::warn!(bucket = %self.name, config = "policy", error = %e, "parse_all_configs: failed to parse");
         }
@@ -881,11 +881,9 @@ pub async fn load_bucket_metadata_parse(api: Arc<ECStore>, bucket: &str, parse:
     bm.default_timestamps();
 
     if parse {
-        bm.parse_all_configs(api)?;
+        bm.parse_all_configs()?;
     }
 
-    // TODO: parse_all_configs
-
     Ok(bm)
 }
 
@@ -939,6 +937,23 @@ mod test {
         assert_eq!(bm.name, new.name);
     }
 
+    #[test]
+    fn parse_all_configs_parses_stored_configs_without_store_dependency() {
+        let mut bm = BucketMetadata::new("test-bucket");
+        bm.policy_config_json = br#"{"Version":"2012-10-17","Statement":[]}"#.to_vec();
+        bm.bucket_targets_config_json =
+            br#"{"targets":[{"endpoint":"s3.amazonaws.com","targetbucket":"target-bucket","arn":"arn:aws:s3:::target-bucket"}]}"#
+                .to_vec();
+
+        bm.parse_all_configs().unwrap();
+
+        assert!(bm.policy_config.is_some());
+        let bucket_targets = bm.bucket_target_config.unwrap();
+        assert_eq!(bucket_targets.targets.len(), 1);
+        assert_eq!(bucket_targets.targets[0].endpoint, "s3.amazonaws.com");
+        assert_eq!(bucket_targets.targets[0].target_bucket, "target-bucket");
+    }
+
     #[tokio::test]
     async fn marshal_msg_complete_example() {
         // Create a complete BucketMetadata with various configurations

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -56,7 +56,7 @@ use rustfs_filemeta::{
     ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision, VersionPurgeStatusType,
     get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
 };
-use rustfs_s3_common::EventName;
+use rustfs_s3_types::EventName;
 use rustfs_utils::http::{
     AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_TAGGING, AMZ_TAGGING_DIRECTIVE, CONTENT_ENCODING, HeaderExt as _,
     SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP,

crates/ecstore/src/cache_value/metacache_set.rs

@@ -19,9 +19,10 @@ use futures::future::join_all;
 use metrics::counter;
 use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
 use std::{
+    collections::VecDeque,
     future::Future,
     pin::Pin,
-    sync::{Arc, Mutex},
+    sync::{Arc, OnceLock},
     time::Duration,
 };
 use tokio::io::AsyncRead;
@@ -112,9 +113,9 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
 
     let mut jobs: Vec<tokio::task::JoinHandle<std::result::Result<(), DiskError>>> = Vec::new();
     let mut readers = Vec::with_capacity(opts.disks.len());
-    let fds = opts.fallback_disks.iter().flatten().cloned().collect::<Vec<_>>();
+    let fds = opts.fallback_disks.iter().flatten().cloned().collect::<VecDeque<_>>();
     let max_disk_failures = opts.disks.len().saturating_sub(opts.min_disks);
-    let producer_errs = Arc::new(Mutex::new(vec![None; opts.disks.len()]));
+    let producer_errs: Arc<[OnceLock<DiskError>]> = (0..opts.disks.len()).map(|_| OnceLock::new()).collect::<Vec<_>>().into();
 
     let cancel_rx = CancellationToken::new();
 
@@ -137,14 +138,14 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                         return Ok(());
                     }
                     TestReaderBehavior::ProducerError(err) => {
-                        producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                        record_producer_error(&producer_errs_clone, disk_idx, &err);
                         return Err(err);
                     }
                     TestReaderBehavior::PartialThenTimeout(entries) => {
                         let mut wr = wr;
                         let mut out = rustfs_filemeta::MetacacheWriter::new(&mut wr);
                         let err = DiskError::Timeout;
-                        producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                        record_producer_error(&producer_errs_clone, disk_idx, &err);
                         let _ = out.write(&entries).await;
                         drop(out);
                         return Err(err);
@@ -187,8 +188,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
 
             while need_fallback {
                 let mut disk_op = None;
-                while !fds_clone.is_empty() {
-                    let disk = fds_clone.remove(0);
+                while let Some(disk) = fds_clone.pop_front() {
                     if disk.is_online().await {
                         disk_op = Some(disk);
                         break;
@@ -198,7 +198,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                 let Some(disk) = disk_op else {
                     warn!("list_path_raw: fallback disk is none");
                     let err = last_err.unwrap_or(DiskError::DiskNotFound);
-                    producer_errs_clone.lock().expect("producer error mutex poisoned")[disk_idx] = Some(err.clone());
+                    record_producer_error(&producer_errs_clone, disk_idx, &err);
                     return Err(err);
                 };
 
@@ -281,7 +281,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                             // info!("read entry disk: {}, name: {}", i, entry.name);
                             entry
                         } else {
-                            if let Some(err) = producer_errs.lock().expect("producer error mutex poisoned")[i].clone() {
+                            if let Some(err) = producer_error(&producer_errs, i) {
                                 has_err += 1;
                                 errs[i] = Some(err);
                                 continue;
@@ -293,7 +293,7 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
                         }
                     }
                     PeekOutcome::Error(err) => {
-                        if let Some(err) = producer_errs.lock().expect("producer error mutex poisoned")[i].clone() {
+                        if let Some(err) = producer_error(&producer_errs, i) {
                             has_err += 1;
                             errs[i] = Some(err);
                             continue;
@@ -510,10 +510,21 @@ pub async fn list_path_raw(rx: CancellationToken, opts: ListPathRawOptions) -> d
     Ok(())
 }
 
+#[inline]
+fn record_producer_error(producer_errs: &[OnceLock<DiskError>], idx: usize, err: &DiskError) {
+    let _ = producer_errs[idx].set(err.clone());
+}
+
+#[inline]
+fn producer_error(producer_errs: &[OnceLock<DiskError>], idx: usize) -> Option<DiskError> {
+    producer_errs[idx].get().cloned()
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
     use rustfs_filemeta::MetacacheWriter;
+    use std::sync::Mutex;
 
     #[tokio::test]
     async fn list_path_raw_empty_disks_returns_read_quorum() {

crates/ecstore/src/data_usage.rs

@@ -27,7 +27,7 @@ pub use local_snapshot::{
     data_usage_dir, data_usage_state_dir, ensure_data_usage_layout, read_snapshot as read_local_snapshot, snapshot_file_name,
     snapshot_object_path, snapshot_path, write_snapshot as write_local_snapshot,
 };
-use rustfs_common::data_usage::{
+use rustfs_data_usage::{
     BucketTargetUsageInfo, BucketUsageInfo, DataUsageCache, DataUsageEntry, DataUsageInfo, DiskUsageStatus, SizeSummary,
 };
 use rustfs_io_metrics::record_system_path_failure;
@@ -686,7 +686,7 @@ pub async fn save_data_usage_cache(cache: &DataUsageCache, name: &str) -> crate:
 #[cfg(test)]
 mod tests {
     use super::*;
-    use rustfs_common::data_usage::BucketUsageInfo;
+    use rustfs_data_usage::BucketUsageInfo;
 
     fn aggregate_for_test(
         inputs: Vec<(DiskUsageStatus, Result<Option<LocalUsageSnapshot>, Error>)>,

crates/ecstore/src/disk/disk_store.rs

@@ -44,6 +44,12 @@ use uuid::Uuid;
 const DISK_HEALTH_OK: u32 = 0;
 const DISK_HEALTH_FAULTY: u32 = 1;
 
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum TimeoutHealthAction {
+    MarkFailure,
+    IgnoreFailure,
+}
+
 pub const ENV_RUSTFS_DRIVE_ACTIVE_MONITORING: &str = "RUSTFS_DRIVE_ACTIVE_MONITORING";
 pub const DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING: bool = true;
 pub const SKIP_IF_SUCCESS_BEFORE: Duration = Duration::from_secs(5);
@@ -551,8 +557,6 @@ impl LocalDiskWrapper {
 
     /// Monitor disk writability periodically
     async fn monitor_disk_writable(disk: Arc<LocalDisk>, health: Arc<DiskHealthTracker>, cancel_token: CancellationToken) {
-        // TODO: config interval
-
         let mut interval = time::interval(get_drive_active_check_interval());
 
         loop {
@@ -804,6 +808,21 @@ impl LocalDiskWrapper {
         operation: F,
         timeout_duration: Duration,
     ) -> Result<T>
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = Result<T>>,
+    {
+        self.track_disk_health_with_op_and_timeout_action(op, operation, timeout_duration, TimeoutHealthAction::MarkFailure)
+            .await
+    }
+
+    async fn track_disk_health_with_op_and_timeout_action<T, F, Fut>(
+        &self,
+        op: &'static str,
+        operation: F,
+        timeout_duration: Duration,
+        timeout_health_action: TimeoutHealthAction,
+    ) -> Result<T>
     where
         F: FnOnce() -> Fut,
         Fut: std::future::Future<Output = Result<T>>,
@@ -848,7 +867,9 @@ impl LocalDiskWrapper {
             Err(_) => {
                 // Timeout occurred, mark disk as potentially faulty and decrement waiting counter
                 self.health.decrement_waiting();
-                if self.health.mark_failure(&self.endpoint(), "operation_timeout") {
+                if timeout_health_action == TimeoutHealthAction::MarkFailure
+                    && self.health.mark_failure(&self.endpoint(), "operation_timeout")
+                {
                     self.spawn_recovery_monitor_if_needed();
                 }
                 counter!(
@@ -996,8 +1017,13 @@ impl DiskAPI for LocalDiskWrapper {
     }
 
     async fn walk_dir<W: tokio::io::AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
-        self.track_disk_health_with_op("walk_dir", || async { self.disk.walk_dir(opts, wr).await }, get_drive_walkdir_timeout())
-            .await
+        self.track_disk_health_with_op_and_timeout_action(
+            "walk_dir",
+            || async { self.disk.walk_dir(opts, wr).await },
+            get_drive_walkdir_timeout(),
+            TimeoutHealthAction::IgnoreFailure,
+        )
+        .await
     }
 
     async fn delete_version(
@@ -1203,6 +1229,28 @@ mod tests {
     use super::*;
     use crate::disk::endpoint::Endpoint;
     use crate::disk::health_state::RuntimeDriveHealthState;
+    use std::{
+        io,
+        pin::Pin,
+        task::{Context, Poll},
+    };
+    use tokio::io::AsyncWrite;
+
+    struct PendingWriter;
+
+    impl AsyncWrite for PendingWriter {
+        fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<io::Result<usize>> {
+            Poll::Pending
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
 
     #[test]
     fn drive_metadata_timeout_uses_default_when_unset() {
@@ -1323,6 +1371,97 @@ mod tests {
         assert!(health.offline_duration().is_none());
     }
 
+    #[tokio::test]
+    async fn ignored_timeout_does_not_mark_drive_failure() {
+        let dir = tempfile::tempdir().expect("temp dir should be created");
+        let endpoint =
+            Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+        let wrapper = LocalDiskWrapper::new(disk, false);
+
+        let result = wrapper
+            .track_disk_health_with_op_and_timeout_action(
+                "walk_dir",
+                || async {
+                    tokio::time::sleep(Duration::from_millis(20)).await;
+                    Ok(())
+                },
+                Duration::from_millis(1),
+                TimeoutHealthAction::IgnoreFailure,
+            )
+            .await;
+
+        assert_eq!(result.expect_err("operation should time out"), DiskError::Timeout);
+        assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+        assert!(!wrapper.health.is_faulty());
+    }
+
+    #[tokio::test]
+    async fn walk_dir_writer_backpressure_timeout_does_not_mark_drive_failure() {
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_WALKDIR_TIMEOUT_SECS, Some("1"))], async {
+            let dir = tempfile::tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+            let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+            let wrapper = LocalDiskWrapper::new(disk, false);
+            let bucket = "test-bucket";
+            let object = "test-object";
+
+            wrapper.make_volume(bucket).await.expect("bucket should be created");
+
+            let mut file_info = FileInfo::new(&format!("{bucket}/{object}"), 1, 0);
+            file_info.volume = bucket.to_string();
+            file_info.name = object.to_string();
+            file_info.mod_time = Some(::time::OffsetDateTime::now_utc());
+            file_info.erasure.index = 1;
+
+            wrapper
+                .write_metadata("", bucket, object, file_info)
+                .await
+                .expect("object metadata should be written");
+
+            let mut writer = PendingWriter;
+            let result = wrapper
+                .walk_dir(
+                    WalkDirOptions {
+                        bucket: bucket.to_string(),
+                        recursive: true,
+                        ..Default::default()
+                    },
+                    &mut writer,
+                )
+                .await;
+
+            assert_eq!(result.expect_err("walk_dir should time out"), DiskError::Timeout);
+            assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Online);
+            assert!(!wrapper.health.is_faulty());
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    async fn default_timeout_marks_drive_failure() {
+        let dir = tempfile::tempdir().expect("temp dir should be created");
+        let endpoint =
+            Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
+        let wrapper = LocalDiskWrapper::new(disk, false);
+
+        let result = wrapper
+            .track_disk_health_with_op(
+                "read_metadata",
+                || async {
+                    tokio::time::sleep(Duration::from_millis(20)).await;
+                    Ok(())
+                },
+                Duration::from_millis(1),
+            )
+            .await;
+
+        assert_eq!(result.expect_err("operation should time out"), DiskError::Timeout);
+        assert_eq!(wrapper.runtime_state(), RuntimeDriveHealthState::Suspect);
+    }
+
     #[test]
     fn reset_for_store_init_retry_clears_faulty_and_back_online() {
         let endpoint = Endpoint::try_from("/tmp/reset-store-init-retry").expect("endpoint should parse");

crates/ecstore/src/disk/endpoint.rs

@@ -82,17 +82,22 @@ impl TryFrom<&str> for Endpoint {
                 #[cfg(not(windows))]
                 let path = Path::new(&path).absolutize()?;
 
-                // On windows having a preceding SlashSeparator will cause problems, if the
-                // command line already has C:/<export-folder/ in it. Final resulting
-                // path on windows might become C:/C:/ this will cause problems
-                // of starting rustfs server properly in distributed mode on windows.
-                // As a special case make sure to trim the separator.
                 #[cfg(windows)]
-                let path = Path::new(&path[1..]).absolutize()?;
+                let path = if has_leading_slash_windows_drive(&path) {
+                    // Url::path() exposes file-like Windows paths as `/C:/...`.
+                    // Strip only that synthetic leading slash; plain URL paths
+                    // such as `/export1` must stay URL paths, not become
+                    // relative paths under the current drive.
+                    Path::new(&path[1..]).absolutize()?.to_string_lossy().into_owned()
+                } else {
+                    path
+                };
+                #[cfg(windows)]
+                let path = Path::new(&path);
 
                 debug!("endpoint try_from: path={}", path.display());
 
-                if path.parent().is_none() || Path::new("").eq(&path) {
+                if path.parent().is_none() || path.as_os_str().is_empty() {
                     return Err(Error::other("empty or root path is not supported in URL endpoint"));
                 }
 
@@ -217,6 +222,12 @@ impl Endpoint {
     }
 }
 
+#[cfg(windows)]
+fn has_leading_slash_windows_drive(path: &str) -> bool {
+    let bytes = path.as_bytes();
+    bytes.len() >= 4 && bytes[0] == b'/' && bytes[1].is_ascii_alphabetic() && bytes[2] == b':' && bytes[3] == b'/'
+}
+
 /// parse a file path into a URL.
 fn url_parse_from_file_path(value: &str) -> Result<Url> {
     // Only check if the arg is an ip address and ask for scheme since its absent.
@@ -242,6 +253,14 @@ fn url_parse_from_file_path(value: &str) -> Result<Url> {
 mod test {
     use super::*;
 
+    fn expected_file_path(path: &str) -> String {
+        Path::new(path).absolutize().unwrap().to_string_lossy().replace('\\', "/")
+    }
+
+    fn expected_file_url(path: &str) -> Url {
+        url_parse_from_file_path(path).unwrap()
+    }
+
     #[test]
     fn test_new_endpoint() {
         #[derive(Default)]
@@ -255,7 +274,7 @@ mod test {
         let u2 = Url::parse("https://example.org/path").unwrap();
         let u4 = Url::parse("http://192.168.253.200/path").unwrap();
         let u6 = Url::parse("http://server:/path").unwrap();
-        let root_slash_foo = Url::from_file_path("/foo").unwrap();
+        let root_slash_foo = expected_file_url("/foo");
 
         let test_cases = [
             TestCase {
@@ -416,7 +435,7 @@ mod test {
         // Test file path display
         let file_endpoint = Endpoint::try_from("/tmp/data").unwrap();
         let display_str = format!("{file_endpoint}");
-        assert_eq!(display_str, "/tmp/data");
+        assert_eq!(display_str, expected_file_path("/tmp/data"));
 
         // Test URL display
         let url_endpoint = Endpoint::try_from("http://example.com:9000/path").unwrap();
@@ -479,12 +498,25 @@ mod test {
     #[test]
     fn test_endpoint_get_file_path() {
         let file_endpoint = Endpoint::try_from("/tmp/data").unwrap();
-        assert_eq!(file_endpoint.get_file_path(), "/tmp/data");
+        assert_eq!(file_endpoint.get_file_path(), expected_file_path("/tmp/data"));
 
         let url_endpoint = Endpoint::try_from("http://example.com:9000/path/to/data").unwrap();
         assert_eq!(url_endpoint.get_file_path(), "/path/to/data");
     }
 
+    #[cfg(windows)]
+    #[test]
+    fn test_windows_url_drive_path_requires_separator_after_colon() {
+        let drive_path_endpoint = Endpoint::try_from("http://host/C:/data").unwrap();
+        assert_eq!(drive_path_endpoint.get_type(), EndpointType::Url);
+        assert!(has_leading_slash_windows_drive(Url::parse("http://host/C:/data").unwrap().path()));
+
+        let url_path_endpoint = Endpoint::try_from("http://host/C:foo").unwrap();
+        assert_eq!(url_path_endpoint.get_type(), EndpointType::Url);
+        assert!(!has_leading_slash_windows_drive(Url::parse("http://host/C:foo").unwrap().path()));
+        assert_eq!(url_path_endpoint.get_file_path(), "/C:foo");
+    }
+
     #[test]
     fn test_endpoint_clone_and_equality() {
         let endpoint1 = Endpoint::try_from("/tmp/data").unwrap();
@@ -503,7 +535,7 @@ mod test {
         // Test with complex paths
         let complex_path = "/var/lib/rustfs/data/bucket1";
         let endpoint = Endpoint::try_from(complex_path).unwrap();
-        assert_eq!(endpoint.get_file_path(), complex_path);
+        assert_eq!(endpoint.get_file_path(), expected_file_path(complex_path));
         assert!(endpoint.is_local);
         assert_eq!(endpoint.get_type(), EndpointType::Path);
     }
@@ -512,7 +544,7 @@ mod test {
     fn test_endpoint_with_spaces_in_path() {
         let path_with_spaces = "/Users/test/Library/Application Support/rustfs/data";
         let endpoint = Endpoint::try_from(path_with_spaces).unwrap();
-        assert_eq!(endpoint.get_file_path(), path_with_spaces);
+        assert_eq!(endpoint.get_file_path(), expected_file_path(path_with_spaces));
         assert!(endpoint.is_local);
         assert_eq!(endpoint.get_type(), EndpointType::Path);
     }
@@ -532,7 +564,7 @@ mod test {
         // Verify that get_file_path() decodes the percent-encoded path correctly
         assert_eq!(
             endpoint.get_file_path(),
-            "/Users/test/Library/Application Support/rustfs/data",
+            expected_file_path("/Users/test/Library/Application Support/rustfs/data"),
             "get_file_path() should decode percent-encoded spaces"
         );
     }
@@ -544,7 +576,7 @@ mod test {
         let endpoint = Endpoint::try_from(path_with_special).unwrap();
 
         // get_file_path() should return the original path with decoded characters
-        assert_eq!(endpoint.get_file_path(), path_with_special);
+        assert_eq!(endpoint.get_file_path(), expected_file_path(path_with_special));
     }
 
     #[test]

crates/ecstore/src/disk/fs.rs

@@ -545,7 +545,7 @@ mod tests {
 
         // Create two different files
         tokio::fs::write(&file1_path, b"content1").await.unwrap();
-        tokio::fs::write(&file2_path, b"content2").await.unwrap();
+        tokio::fs::write(&file2_path, b"different content").await.unwrap();
 
         // Get metadata
         let metadata1 = tokio::fs::metadata(&file1_path).await.unwrap();

crates/ecstore/src/disk/local.rs

@@ -1179,14 +1179,7 @@ impl LocalDisk {
                 f.write_all(buf).await.map_err(to_file_error)?;
             }
             InternalBuf::Owned(buf) => {
-                // Reduce one copy by using the owned buffer directly.
-                // It may be more efficient for larger writes.
-                let mut f = f.into_std().await;
-                let task = tokio::task::spawn_blocking(move || {
-                    use std::io::Write as _;
-                    f.write_all(buf.as_ref()).map_err(to_file_error)
-                });
-                task.await??;
+                f.write_all(buf.as_ref()).await.map_err(to_file_error)?;
             }
         }
 
@@ -1199,7 +1192,9 @@ impl LocalDisk {
             skip_parent = self.root.as_path();
         }
 
-        if let Some(parent) = path.as_ref().parent() {
+        if let Some(parent) = path.as_ref().parent()
+            && parent != skip_parent
+        {
             os::make_dir_all(parent, skip_parent).await?;
         }
 
@@ -1208,6 +1203,11 @@ impl LocalDisk {
         Ok(f)
     }
 
+    async fn open_file_read_only(&self, path: impl AsRef<Path>) -> Result<File> {
+        let f = super::fs::open_file(path.as_ref(), O_RDONLY).await.map_err(to_file_error)?;
+        Ok(f)
+    }
+
     #[allow(dead_code)]
     fn get_metrics(&self) -> DiskMetrics {
         DiskMetrics::default()
@@ -1234,6 +1234,7 @@ impl LocalDisk {
     }
 
     #[async_recursion::async_recursion]
+    #[allow(clippy::too_many_arguments)]
     async fn scan_dir<W>(
         &self,
         mut current: String,
@@ -1242,6 +1243,7 @@ impl LocalDisk {
         out: &mut MetacacheWriter<W>,
         objs_returned: &mut i32,
         skip_current_dir_object: bool,
+        multipart_dir_to_skip: Option<HashSet<String>>,
     ) -> Result<()>
     where
         W: AsyncWrite + Unpin + Send,
@@ -1298,6 +1300,14 @@ impl LocalDisk {
             if opts.limit > 0 && *objs_returned >= opts.limit {
                 return Ok(());
             }
+            // check multipart dir
+            if skip_current_dir_object
+                && let Some(ref dir_to_skip) = multipart_dir_to_skip
+                && dir_to_skip.contains(entry.trim_end_matches(SLASH_SEPARATOR))
+            {
+                *item = "".to_owned();
+                continue;
+            }
             // check prefix
             if !prefix.is_empty() && !entry.starts_with(prefix.as_str()) {
                 *item = "".to_owned();
@@ -1367,15 +1377,25 @@ impl LocalDisk {
             }
         }
 
-        let mut dir_stack: Vec<(String, bool)> = Vec::with_capacity(5);
+        let mut dir_stack: Vec<(String, bool, Option<HashSet<String>>)> = Vec::with_capacity(5);
         // Explicit directory markers and real directories can resolve to the same logical path.
-        let schedule_dir = |dir_stack: &mut Vec<(String, bool)>, dir_name: String, skip_object: bool| {
-            if let Some((last_dir_name, existing_skip_object)) = dir_stack.last_mut()
+        let schedule_dir = |dir_stack: &mut Vec<(String, bool, Option<HashSet<String>>)>,
+                            dir_name: String,
+                            skip_object: bool,
+                            dir_to_skip: Option<HashSet<String>>| {
+            if let Some((last_dir_name, existing_skip_object, existing_dir_to_skip)) = dir_stack.last_mut()
                 && *last_dir_name == dir_name
             {
                 *existing_skip_object |= skip_object;
+                if let Some(existing_dir_to_skip) = existing_dir_to_skip {
+                    if let Some(new_dir_to_skip) = &dir_to_skip {
+                        existing_dir_to_skip.extend(new_dir_to_skip.iter().cloned());
+                    }
+                } else {
+                    *existing_dir_to_skip = dir_to_skip;
+                }
             } else {
-                dir_stack.push((dir_name, skip_object));
+                dir_stack.push((dir_name, skip_object, dir_to_skip));
             }
         };
         prefix = "".to_owned();
@@ -1391,9 +1411,10 @@ impl LocalDisk {
 
             let name = path_join_buf(&[current.as_str(), entry.as_str()]);
 
-            while let Some((pop, skip_object)) = dir_stack.last().cloned()
-                && pop < name
+            while let Some((last_name, _, _)) = dir_stack.last()
+                && *last_name < name
             {
+                let (pop, skip_object, dir_to_skip) = dir_stack.pop().unwrap();
                 out.write_obj(&MetaCacheEntry {
                     name: pop.clone(),
                     ..Default::default()
@@ -1401,11 +1422,11 @@ impl LocalDisk {
                 .await?;
 
                 if opts.recursive
-                    && let Err(er) = Box::pin(self.scan_dir(pop, prefix.clone(), opts, out, objs_returned, skip_object)).await
+                    && let Err(er) =
+                        Box::pin(self.scan_dir(pop, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await
                 {
                     error!("scan_dir err {:?}", er);
                 }
-                dir_stack.pop();
             }
 
             let mut meta = MetaCacheEntry {
@@ -1442,11 +1463,24 @@ impl LocalDisk {
                     // }
 
                     if opts.recursive {
+                        let mut dir_to_skip = HashSet::new();
+                        if let Ok(file_meta) = FileMeta::load(&res)
+                            && let Ok(data_dirs) = file_meta.get_data_dirs()
+                        {
+                            for data_dir in data_dirs.iter().flatten() {
+                                dir_to_skip.insert(data_dir.to_string());
+                            }
+                        }
                         let mut dir_name = meta.name.clone();
                         if !dir_name.ends_with(SLASH_SEPARATOR) {
                             dir_name.push_str(SLASH_SEPARATOR);
                         }
-                        schedule_dir(&mut dir_stack, dir_name, true);
+                        schedule_dir(
+                            &mut dir_stack,
+                            dir_name,
+                            true,
+                            if dir_to_skip.is_empty() { None } else { Some(dir_to_skip) },
+                        );
                     }
                 }
                 Err(err) => {
@@ -1455,7 +1489,7 @@ impl LocalDisk {
                         // If dirObject, but no metadata (which is unexpected) we skip it.
                         if !is_dir_obj && !is_empty_dir(self.get_object_path(&opts.bucket, &meta.name)?).await {
                             meta.name.push_str(SLASH_SEPARATOR);
-                            schedule_dir(&mut dir_stack, meta.name, false);
+                            schedule_dir(&mut dir_stack, meta.name, false, None);
                         }
                     }
 
@@ -1464,7 +1498,7 @@ impl LocalDisk {
             };
         }
 
-        while let Some((dir, skip_object)) = dir_stack.pop() {
+        while let Some((dir, skip_object, dir_to_skip)) = dir_stack.pop() {
             if opts.limit > 0 && *objs_returned >= opts.limit {
                 return Ok(());
             }
@@ -1476,7 +1510,8 @@ impl LocalDisk {
             .await?;
 
             if opts.recursive
-                && let Err(er) = Box::pin(self.scan_dir(dir, prefix.clone(), opts, out, objs_returned, skip_object)).await
+                && let Err(er) =
+                    Box::pin(self.scan_dir(dir, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await
             {
                 warn!("scan_dir err {:?}", &er);
             }
@@ -2107,7 +2142,7 @@ impl DiskAPI for LocalDisk {
         let file_path = self.get_object_path(volume, path)?;
         check_path_length(file_path.to_string_lossy().as_ref())?;
 
-        let f = self.open_file(file_path, O_RDONLY, volume_dir).await?;
+        let f = self.open_file_read_only(file_path).await?;
 
         Ok(Box::new(f))
     }
@@ -2124,7 +2159,7 @@ impl DiskAPI for LocalDisk {
         let file_path = self.get_object_path(volume, path)?;
         check_path_length(file_path.to_string_lossy().as_ref())?;
 
-        let mut f = self.open_file(file_path, O_RDONLY, volume_dir).await?;
+        let mut f = self.open_file_read_only(file_path).await?;
 
         let meta = f.metadata().await?;
         let end_offset = offset.checked_add(length).ok_or(DiskError::FileCorrupt)?;
@@ -2153,9 +2188,6 @@ impl DiskAPI for LocalDisk {
     #[allow(unsafe_code)]
     #[tracing::instrument(level = "debug", skip(self))]
     async fn read_file_zero_copy(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<Bytes> {
-        use std::time::Instant;
-
-        let start = Instant::now();
         let volume_dir = self.get_bucket_path(volume)?;
         if !skip_access_checks(volume) {
             access(&volume_dir)
@@ -2188,6 +2220,9 @@ impl DiskAPI for LocalDisk {
         #[cfg(unix)]
         {
             use memmap2::MmapOptions;
+            use std::time::Instant;
+
+            let start = Instant::now();
             let file_path_clone = file_path.clone();
 
             let should_reclaim_after_read = should_reclaim_file_cache_after_read(length);
@@ -2272,8 +2307,7 @@ impl DiskAPI for LocalDisk {
                 f.seek(SeekFrom::Start(offset as u64)).await?;
             }
 
-            let mut buffer = Vec::with_capacity(length);
-            buffer.resize(length, 0);
+            let mut buffer = vec![0; length];
             f.read_exact(&mut buffer).await?;
 
             Ok(Bytes::from(buffer))
@@ -2329,6 +2363,7 @@ impl DiskAPI for LocalDisk {
         let mut objs_returned = 0;
 
         let mut skip_current_dir_object = false;
+        let mut multipart_dir_to_skip: HashSet<String> = HashSet::new();
         if opts.base_dir.ends_with(SLASH_SEPARATOR) {
             if let Ok(data) = self
                 .read_metadata(
@@ -2352,10 +2387,23 @@ impl DiskAPI for LocalDisk {
                 let fpath =
                     self.get_object_path(&opts.bucket, path_join_buf(&[opts.base_dir.as_str(), STORAGE_FORMAT_FILE]).as_str())?;
 
-                if let Ok(meta) = tokio::fs::metadata(fpath).await
+                if let Ok(meta) = tokio::fs::metadata(&fpath).await
                     && meta.is_file()
                 {
                     skip_current_dir_object = true;
+                    if let Ok(meta_bytes) = self
+                        .read_metadata(
+                            opts.bucket.as_str(),
+                            path_join_buf(&[opts.base_dir.as_str(), STORAGE_FORMAT_FILE]).as_str(),
+                        )
+                        .await
+                        && let Ok(file_meta) = FileMeta::load(&meta_bytes)
+                        && let Ok(data_dirs) = file_meta.get_data_dirs()
+                    {
+                        for data_dir in data_dirs.iter().flatten() {
+                            multipart_dir_to_skip.insert(data_dir.to_string());
+                        }
+                    }
                 }
             }
         }
@@ -2367,6 +2415,11 @@ impl DiskAPI for LocalDisk {
             &mut out,
             &mut objs_returned,
             skip_current_dir_object,
+            if multipart_dir_to_skip.is_empty() {
+                None
+            } else {
+                Some(multipart_dir_to_skip)
+            },
         )
         .await?;
 
@@ -2465,32 +2518,41 @@ impl DiskAPI for LocalDisk {
 
         // TODO: Healing
 
-        let search_version_id = fi.version_id.or(Some(Uuid::nil()));
+        let version_id = fi.version_id.unwrap_or_default();
+        let search_version_id = Some(version_id);
+        let no_inline = fi.data.is_none() && fi.size > 0;
 
         // Check if there's an existing version with the same version_id that has a data_dir to clean up
-        let has_old_data_dir = {
-            xlmeta.find_version(search_version_id).ok().and_then(|(_, ver)| {
-                // shard_count == 0 means no other version shares this data_dir
-                ver.get_data_dir()
-                    .filter(|&data_dir| xlmeta.shard_data_dir_count(&search_version_id, &Some(data_dir)) == 0)
-            })
-        };
+        // Reuse one metadata scan to find the version data_dir and determine whether it is shared.
+        let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(search_version_id);
         if let Some(old_data_dir) = has_old_data_dir.as_ref() {
-            let _ = xlmeta.data.remove(vec![search_version_id.unwrap_or_default(), *old_data_dir]);
+            let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
         }
 
-        xlmeta.add_version(fi.clone())?;
+        xlmeta.add_version(fi)?;
 
         if xlmeta.versions.len() <= 10 {
             // TODO: Sign
         }
 
-        let new_dst_buf = xlmeta.marshal_msg()?;
-
-        self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf.into())
-            .await?;
         if let Some((src_data_path, dst_data_path)) = has_data_dir_path.as_ref() {
-            let no_inline = fi.data.is_none() && fi.size > 0;
+            let src_file_parent = src_file_path.parent().unwrap_or(src_volume_dir.as_path());
+            let meta_skip_parent = if no_inline {
+                src_file_parent
+            } else {
+                src_volume_dir.as_path()
+            };
+            let new_dst_buf = xlmeta.marshal_msg()?;
+
+            self.write_all_private(
+                src_volume,
+                format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(),
+                new_dst_buf.into(),
+                true,
+                meta_skip_parent,
+            )
+            .await?;
+
             if no_inline && let Err(err) = rename_all(&src_data_path, &dst_data_path, &skip_parent).await {
                 let _ = self.delete_file(&dst_volume_dir, dst_data_path, false, false).await;
                 info!(
@@ -2499,6 +2561,10 @@ impl DiskAPI for LocalDisk {
                 );
                 return Err(err);
             }
+        } else {
+            let new_dst_buf = xlmeta.marshal_msg()?;
+            self.write_all(src_volume, format!("{}/{}", &src_path, STORAGE_FORMAT_FILE).as_str(), new_dst_buf.into())
+                .await?;
         }
 
         if let Some(old_data_dir) = has_old_data_dir {
@@ -2985,7 +3051,6 @@ impl DiskAPI for LocalDisk {
     #[tracing::instrument(skip(self))]
     async fn disk_info(&self, _: &DiskInfoOptions) -> Result<DiskInfo> {
         let mut info = Cache::get(self.disk_info_cache.clone()).await?;
-        // TODO: nr_requests, rotational
         info.nr_requests = self.nrrequests;
         info.rotational = self.rotational;
         info.mount_path = self.path().to_str().unwrap().to_string();
@@ -3152,7 +3217,7 @@ mod test {
         };
         let mut objs_returned = 0;
 
-        disk.scan_dir("".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false)
+        disk.scan_dir("".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false, None)
             .await
             .unwrap();
         out.close().await.unwrap();
@@ -3207,7 +3272,7 @@ mod test {
         };
         let mut objs_returned = 0;
 
-        disk.scan_dir("marker/".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false)
+        disk.scan_dir("marker/".to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false, None)
             .await
             .unwrap();
         out.close().await.unwrap();
@@ -3267,7 +3332,7 @@ mod test {
             };
             let mut objs_returned = 0;
 
-            disk.scan_dir(base_dir.to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false)
+            disk.scan_dir(base_dir.to_string(), "".to_string(), &opts, &mut out, &mut objs_returned, false, None)
                 .await
                 .unwrap();
             out.close().await.unwrap();
@@ -3328,6 +3393,155 @@ mod test {
         assert_eq!(double_count as usize, double_names.len());
     }
 
+    #[tokio::test]
+    async fn test_walk_dir_ignore_multipart_dirs() {
+        use rustfs_filemeta::MetacacheReader;
+        use tempfile::tempdir;
+
+        const UUID_MULTIPART_1: &str = "8b262d24-fcf9-473d-a4cd-f9b27f24f60e";
+        const UUID_MULTIPART_2: &str = "fbf3183c-63be-45cc-b3bf-424ddb7f95f8";
+        const UUID_OBJ: &str = "db8b9b74-9016-4f9e-83e9-82a772947d28";
+        const VER_ID_1: &str = "c683f9f8-c0a1-4bc5-8a67-0faafa839a1a";
+        const VER_ID_2: &str = "a4b84f6e-c8ba-461b-8f9d-43feb0893efb";
+        const VER_ID_3: &str = "892c9ae7-2bb3-44ee-9a71-bc7ddf08d765";
+        const BASE_DIR: &str = "dir1/obj/";
+        const MULTIPART_DIR: &str = "multipart-file";
+        const DIR_IN_MULTIPART_DIR: &str = "dir-in-multipart";
+        const EMPTY_STR: &str = "";
+
+        let parse_uuid = |s: &str| Uuid::parse_str(s).unwrap();
+        let create_file_info = |version_id: &str, data_dir: &str| FileInfo {
+            version_id: Some(parse_uuid(version_id)),
+            data_dir: Some(parse_uuid(data_dir)),
+            mod_time: Some(OffsetDateTime::now_utc()),
+            ..Default::default()
+        };
+
+        let dir = tempdir().unwrap();
+        let obj_base = dir.path().join("test-bucket").join(BASE_DIR);
+        let multipart_base = obj_base.join(MULTIPART_DIR);
+        let dir_in_multipart_base = multipart_base.join(DIR_IN_MULTIPART_DIR);
+
+        fs::create_dir_all(&multipart_base).await.unwrap();
+        for uuid in &[UUID_MULTIPART_1, UUID_MULTIPART_2] {
+            fs::create_dir_all(multipart_base.join(uuid)).await.unwrap();
+            fs::write(multipart_base.join(uuid).join("part.1"), b"part").await.unwrap();
+        }
+        fs::create_dir_all(obj_base.join(UUID_OBJ)).await.unwrap();
+        fs::write(obj_base.join(UUID_OBJ).join("part.1"), b"part").await.unwrap();
+
+        fs::create_dir_all(&dir_in_multipart_base).await.unwrap();
+        fs::write(dir_in_multipart_base.join(STORAGE_FORMAT_FILE), b"meta")
+            .await
+            .unwrap();
+
+        let mut fm = FileMeta::default();
+        fm.add_version(create_file_info(VER_ID_1, UUID_MULTIPART_1)).unwrap();
+        fm.add_version(create_file_info(VER_ID_2, UUID_MULTIPART_2)).unwrap();
+        fs::write(multipart_base.join(STORAGE_FORMAT_FILE), fm.marshal_msg().unwrap())
+            .await
+            .unwrap();
+
+        let mut fm = FileMeta::default();
+        fm.add_version(create_file_info(VER_ID_3, UUID_OBJ)).unwrap();
+        fs::write(obj_base.join(STORAGE_FORMAT_FILE), fm.marshal_msg().unwrap())
+            .await
+            .unwrap();
+
+        let endpoint = Endpoint::try_from(dir.path().to_str().unwrap()).unwrap();
+        let disk = LocalDisk::new(&endpoint, false).await.unwrap();
+
+        let (reader, mut writer) = tokio::io::duplex(4096);
+        disk.walk_dir(
+            WalkDirOptions {
+                bucket: "test-bucket".to_string(),
+                base_dir: BASE_DIR.to_string(),
+                recursive: true,
+                filter_prefix: Some(EMPTY_STR.to_string()),
+                ..Default::default()
+            },
+            &mut writer,
+        )
+        .await
+        .unwrap();
+        MetacacheWriter::new(&mut writer).close().await.unwrap();
+
+        let mut reader = MetacacheReader::new(reader);
+        let entries = reader.read_all().await.unwrap();
+        let names: Vec<String> = entries.into_iter().map(|entry| entry.name).collect();
+
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}", BASE_DIR, MULTIPART_DIR))
+                .count(),
+            1
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/", BASE_DIR, MULTIPART_DIR))
+                .count(),
+            1
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}", BASE_DIR, MULTIPART_DIR, DIR_IN_MULTIPART_DIR))
+                .count(),
+            1
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}/", BASE_DIR, MULTIPART_DIR, DIR_IN_MULTIPART_DIR))
+                .count(),
+            1
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}", BASE_DIR, MULTIPART_DIR, UUID_MULTIPART_1))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}", BASE_DIR, MULTIPART_DIR, UUID_MULTIPART_2))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}", BASE_DIR, UUID_OBJ))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}/", BASE_DIR, MULTIPART_DIR, UUID_MULTIPART_1))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/{}/", BASE_DIR, MULTIPART_DIR, UUID_MULTIPART_2))
+                .count(),
+            0
+        );
+        assert_eq!(
+            names
+                .iter()
+                .filter(|name| *name == &format!("{}{}/", BASE_DIR, UUID_OBJ))
+                .count(),
+            0
+        );
+    }
+
     #[tokio::test]
     async fn test_make_volume() {
         let p = "./testv0";
@@ -3497,13 +3711,15 @@ mod test {
         let disk_info = disk.disk_info(&disk_info_opts).await.unwrap();
 
         // Basic checks on disk info
-        // Note: On macOS and some other Unix systems, fs_type may be empty
+        // Note: On macOS, Windows, and some other systems, fs_type may be empty
         // because statvfs does not provide filesystem type information.
         // This is a platform limitation, not a bug.
-        #[cfg(not(target_os = "macos"))]
+        #[cfg(not(any(target_os = "macos", windows)))]
         assert!(!disk_info.fs_type.is_empty(), "fs_type should not be empty on this platform");
         assert!(disk_info.total > 0);
         assert!(disk_info.free <= disk_info.total);
+        assert_eq!(disk_info.nr_requests, disk.nrrequests);
+        assert_eq!(disk_info.rotational, disk.rotational);
         assert!(!disk_info.mount_path.is_empty());
         assert!(!disk_info.endpoint.is_empty());
 

crates/ecstore/src/disk/mod.rs

@@ -37,6 +37,7 @@ use crate::disk::disk_store::LocalDiskWrapper;
 use crate::disk::health_state::RuntimeDriveHealthState;
 use crate::disk::local::ScanGuard;
 use crate::rpc::RemoteDisk;
+use crate::rpc::build_internode_data_transport_from_env;
 use bytes::Bytes;
 use endpoint::Endpoint;
 use error::DiskError;
@@ -476,7 +477,8 @@ pub async fn new_disk(ep: &Endpoint, opt: &DiskOption) -> Result<DiskStore> {
         let s = LocalDisk::new(ep, opt.cleanup).await?;
         Ok(Arc::new(Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(s), opt.health_check)))))
     } else {
-        let remote_disk = RemoteDisk::new(ep, opt).await?;
+        let data_transport = build_internode_data_transport_from_env();
+        let remote_disk = RemoteDisk::new(ep, opt, data_transport?).await?;
         Ok(Arc::new(Disk::Remote(Box::new(remote_disk))))
     }
 }
@@ -789,7 +791,6 @@ mod tests {
     use super::*;
     use endpoint::Endpoint;
     use local::LocalDisk;
-    use std::path::PathBuf;
     use tokio::fs;
     use uuid::Uuid;
 
@@ -1094,7 +1095,7 @@ mod tests {
         assert!(disk.is_ok());
 
         let disk = disk.unwrap();
-        assert_eq!(disk.path(), PathBuf::from(test_dir).canonicalize().unwrap());
+        assert_eq!(disk.path(), rustfs_utils::canonicalize(test_dir).unwrap());
         assert!(disk.is_local());
         // Note: is_online() might return false for local disks without proper initialization
         // This is expected behavior for test environments
@@ -1155,6 +1156,7 @@ mod tests {
                 cleanup: false,
                 health_check: false,
             },
+            Arc::new(crate::rpc::TcpHttpInternodeDataTransport),
         )
         .await
         .unwrap();

crates/ecstore/src/disk/os.rs

@@ -215,24 +215,29 @@ pub async fn os_mkdir_all(dir_path: impl AsRef<Path>, base_dir: impl AsRef<Path>
         return Ok(());
     }
 
-    if let Some(parent) = dir_path.as_ref().parent() {
-        // Without recursion support, fall back to create_dir_all
-        if let Err(e) = super::fs::make_dir_all(&parent).await {
-            if e.kind() == io::ErrorKind::AlreadyExists {
-                return Ok(());
-            }
-
-            return Err(e);
-        }
-        // Box::pin(os_mkdir_all(&parent, &base_dir)).await?;
-    }
-
     if let Err(e) = super::fs::mkdir(dir_path.as_ref()).await {
         if e.kind() == io::ErrorKind::AlreadyExists {
             return Ok(());
         }
 
-        return Err(e);
+        if e.kind() != io::ErrorKind::NotFound {
+            return Err(e);
+        }
+
+        if let Some(parent) = dir_path.as_ref().parent() {
+            // Fall back to creating the missing parent chain only when the direct mkdir proves it is required.
+            if let Err(parent_err) = super::fs::make_dir_all(parent).await
+                && parent_err.kind() != io::ErrorKind::AlreadyExists
+            {
+                return Err(parent_err);
+            }
+        }
+
+        if let Err(retry_err) = super::fs::mkdir(dir_path.as_ref()).await
+            && retry_err.kind() != io::ErrorKind::AlreadyExists
+        {
+            return Err(retry_err);
+        }
     }
 
     Ok(())

crates/ecstore/src/endpoints.rs

@@ -749,6 +749,7 @@ fn validate_local_cross_device_mounts(local_paths: &[String]) -> Result<()> {
 
 #[cfg(test)]
 mod test {
+    use path_absolutize::Absolutize;
     use rustfs_utils::must_get_local_ips;
 
     use super::*;
@@ -1452,9 +1453,10 @@ mod test {
     }
 
     fn must_file_path(s: impl AsRef<Path>) -> url::Url {
-        let url = url::Url::from_file_path(s.as_ref());
+        let path = s.as_ref().absolutize().expect("absolute test path");
+        let url = url::Url::from_file_path(&path);
 
-        assert!(url.is_ok(), "failed to convert path to URL: {}", s.as_ref().display());
+        assert!(url.is_ok(), "failed to convert path to URL: {}", path.display());
 
         url.unwrap()
     }

crates/ecstore/src/event/name.rs

@@ -13,6 +13,6 @@
 // limitations under the License.
 
 //! Compatibility re-export for the legacy `rustfs_ecstore::event::name::EventName` path.
-//! The canonical event definition now lives in `rustfs_s3_common::EventName`.
+//! The canonical event definition now lives in `rustfs_s3_types::EventName`.
 
-pub use rustfs_s3_common::EventName;
+pub use rustfs_s3_types::EventName;

crates/ecstore/src/metrics_realtime.rs

@@ -14,11 +14,12 @@
 
 use crate::{admin_server_info::get_local_server_property, new_object_layer_fn, store_api::StorageAPI};
 use chrono::Utc;
-use rustfs_common::{
-    GLOBAL_LOCAL_NODE_NAME, GLOBAL_RUSTFS_ADDR, heal_channel::DriveState, internode_metrics::global_internode_metrics,
-    metrics::global_metrics,
+use rustfs_common::{GLOBAL_LOCAL_NODE_NAME, GLOBAL_RUSTFS_ADDR, heal_channel::DriveState, metrics::global_metrics};
+use rustfs_io_metrics::internode_metrics::global_internode_metrics;
+use rustfs_madmin::metrics::{
+    DiskIOStats, DiskMetric, LastMinute as MadminLastMinute, NetDevLine, NetMetrics, RPCMetrics, RealtimeMetrics,
+    ScannerMetrics as MadminScannerMetrics, TimedAction as MadminTimedAction,
 };
-use rustfs_madmin::metrics::{DiskIOStats, DiskMetric, NetDevLine, NetMetrics, RPCMetrics, RealtimeMetrics};
 use rustfs_utils::os::get_drive_stats;
 use serde::{Deserialize, Serialize};
 use std::collections::{HashMap, HashSet};
@@ -56,6 +57,51 @@ impl MetricType {
     }
 }
 
+fn to_madmin_scanner_metrics(metrics: rustfs_common::metrics::ScannerMetricsReport) -> MadminScannerMetrics {
+    MadminScannerMetrics {
+        collected_at: metrics.collected_at,
+        current_cycle: metrics.current_cycle,
+        current_started: metrics.current_started,
+        cycles_completed_at: metrics.cycles_completed_at,
+        ongoing_buckets: metrics.ongoing_buckets,
+        life_time_ops: metrics.life_time_ops,
+        life_time_ilm: metrics.life_time_ilm,
+        last_minute: MadminLastMinute {
+            actions: metrics
+                .last_minute
+                .actions
+                .into_iter()
+                .map(|(key, value)| {
+                    (
+                        key,
+                        MadminTimedAction {
+                            count: value.count,
+                            acc_time: value.acc_time,
+                            bytes: value.bytes,
+                        },
+                    )
+                })
+                .collect(),
+            ilm: metrics
+                .last_minute
+                .ilm
+                .into_iter()
+                .map(|(key, value)| {
+                    (
+                        key,
+                        MadminTimedAction {
+                            count: value.count,
+                            acc_time: value.acc_time,
+                            bytes: value.bytes,
+                        },
+                    )
+                })
+                .collect(),
+        },
+        active_paths: metrics.active_paths,
+    }
+}
+
 impl MetricType {
     fn contains(&self, x: &MetricType) -> bool {
         (self.0 & x.0) == x.0
@@ -114,7 +160,7 @@ pub async fn collect_local_metrics(types: MetricType, opts: &CollectMetricsOpts)
         if let Some(init_time) = rustfs_common::get_global_init_time().await {
             metrics.current_started = init_time;
         }
-        real_time_metrics.aggregated.scanner = Some(metrics);
+        real_time_metrics.aggregated.scanner = Some(to_madmin_scanner_metrics(metrics));
     }
 
     // if types.contains(&MetricType::OS) {}
@@ -252,7 +298,7 @@ async fn collect_local_disks_metrics(disks: &HashSet<String>) -> HashMap<String,
 #[cfg(test)]
 mod test {
     use super::*;
-    use rustfs_common::internode_metrics::global_internode_metrics;
+    use rustfs_io_metrics::internode_metrics::global_internode_metrics;
     use std::time::Duration;
 
     #[test]

crates/ecstore/src/notification_sys.rs

@@ -872,10 +872,7 @@ mod tests {
         let result = call_peer_with_timeout(
             Duration::from_millis(5),
             "peer-3",
-            || async {
-                tokio::time::sleep(Duration::from_millis(25)).await;
-                Ok::<_, Error>(build_props("slow"))
-            },
+            std::future::pending::<Result<ServerProperties>>,
             || build_props("fallback"),
         )
         .await;

crates/ecstore/src/pools.rs

@@ -51,9 +51,9 @@ use rmp_serde::Deserializer;
 use rmp_serde::Serializer;
 use rustfs_common::defer;
 use rustfs_common::heal_channel::HealOpts;
+use rustfs_concurrency::workers::Workers;
 use rustfs_filemeta::{FileInfoVersions, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
 use rustfs_utils::path::{encode_dir_object, path_join, path_to_bucket_object, path_to_bucket_object_with_base_path};
-use rustfs_workers::workers::Workers;
 use s3s::dto::{BucketLifecycleConfiguration, DefaultRetention, ReplicationConfiguration};
 use serde::{Deserialize, Serialize};
 use std::collections::{HashMap, HashSet};

crates/ecstore/src/rebalance.rs

@@ -1169,6 +1169,37 @@ fn resolve_rebalance_worker_result(
     }
 }
 
+type RebalanceEntryTask = tokio::task::JoinHandle<Result<()>>;
+
+async fn wait_rebalance_entry_tasks(set_idx: usize, tasks: Arc<tokio::sync::Mutex<Vec<RebalanceEntryTask>>>) -> Result<()> {
+    let tasks = {
+        let mut tasks = tasks.lock().await;
+        std::mem::take(&mut *tasks)
+    };
+
+    let mut first_error = None;
+    for task in tasks {
+        match task.await {
+            Ok(Ok(())) => {}
+            Ok(Err(err)) => {
+                error!("rebalance entry task failed for set {}: {}", set_idx, err);
+                if first_error.is_none() {
+                    first_error = Some(err);
+                }
+            }
+            Err(err) => {
+                let err = Error::other(format!("rebalance entry task join error for set {set_idx}: {err}"));
+                error!("{}", err);
+                if first_error.is_none() {
+                    first_error = Some(err);
+                }
+            }
+        }
+    }
+
+    if let Some(err) = first_error { Err(err) } else { Ok(()) }
+}
+
 fn resolve_rebalance_save_task_result(
     pool_idx: usize,
     save_task_result: std::result::Result<Result<()>, tokio::task::JoinError>,
@@ -1691,26 +1722,28 @@ impl ECStore {
 
         let mut jobs = Vec::new();
         let entry_error = Arc::new(tokio::sync::Mutex::new(None::<Error>));
+        let entry_workers = Arc::new(tokio::sync::Semaphore::new(pool.disk_set.len().max(1)));
 
-        // let wk = Workers::new(pool.disk_set.len() * 2).map_err(Error::other)?;
-        // wk.clone().take().await;
         for (set_idx, set) in pool.disk_set.iter().enumerate() {
+            let entry_tasks = Arc::new(tokio::sync::Mutex::new(Vec::<RebalanceEntryTask>::new()));
             let rebalance_entry: ListCallback = Arc::new({
                 let this = Arc::clone(self);
                 let bucket = bucket.clone();
                 let entry_error = entry_error.clone();
                 let callback_rx = rx.clone();
-                // let wk = wk.clone();
                 let set = set.clone();
                 let bucket_configs = bucket_configs.clone();
+                let entry_tasks = entry_tasks.clone();
+                let entry_workers = entry_workers.clone();
                 move |entry: MetaCacheEntry| {
                     let this = this.clone();
                     let bucket = bucket.clone();
                     let entry_error = entry_error.clone();
                     let callback_rx = callback_rx.clone();
-                    // let wk = wk.clone();
                     let set = set.clone();
                     let bucket_configs = bucket_configs.clone();
+                    let entry_tasks = entry_tasks.clone();
+                    let entry_workers = entry_workers.clone();
                     Box::pin(async move {
                         if callback_rx.is_cancelled() {
                             return;
@@ -1719,20 +1752,38 @@ impl ECStore {
                             return;
                         }
 
-                        info!("rebalance_entry: rebalance_entry spawn start");
-                        // wk.take().await;
-                        // tokio::spawn(async move {
-                        info!("rebalance_entry: rebalance_entry spawn start2");
-                        if let Err(err) = this.rebalance_entry(bucket, pool_index, entry, set, bucket_configs).await {
-                            error!("rebalance_entry: rebalance entry failed: {err}");
-                            let mut first_err = entry_error.lock().await;
-                            if first_err.is_none() {
-                                *first_err = Some(err);
-                                callback_rx.cancel();
-                            }
+                        let permit = tokio::select! {
+                            _ = callback_rx.cancelled() => return,
+                            permit = entry_workers.clone().acquire_owned() => match permit {
+                                Ok(permit) => permit,
+                                Err(err) => {
+                                    error!("rebalance_entry: worker semaphore closed: {err}");
+                                    return;
+                                }
+                            },
+                        };
+
+                        if entry_error.lock().await.is_some() {
+                            return;
                         }
-                        info!("rebalance_entry: rebalance_entry spawn done");
-                        // });
+
+                        let task = tokio::spawn(async move {
+                            let _permit = permit;
+                            info!("rebalance_entry: rebalance entry task start");
+                            let result = this.rebalance_entry(bucket, pool_index, entry, set, bucket_configs).await;
+                            if let Err(err) = &result {
+                                error!("rebalance_entry: rebalance entry failed: {err}");
+                                let mut first_err = entry_error.lock().await;
+                                if first_err.is_none() {
+                                    *first_err = Some(err.clone());
+                                    callback_rx.cancel();
+                                }
+                            }
+                            info!("rebalance_entry: rebalance entry task done");
+                            result
+                        });
+
+                        entry_tasks.lock().await.push(task);
                     })
                 }
             });
@@ -1740,23 +1791,23 @@ impl ECStore {
             let set = set.clone();
             let rx = rx.clone();
             let bucket = bucket.clone();
-            // let wk = wk.clone();
+            let entry_tasks = entry_tasks.clone();
 
             let job = tokio::spawn(async move {
-                let result = set.list_objects_to_rebalance(rx, bucket, rebalance_entry).await;
+                let list_result = set.list_objects_to_rebalance(rx, bucket, rebalance_entry).await;
+                let entry_result = wait_rebalance_entry_tasks(set_idx, entry_tasks).await;
+                let result = list_result.and(entry_result);
                 if let Err(err) = &result {
                     error!("Rebalance worker {} error: {}", set_idx, err);
                 } else {
                     info!("Rebalance worker {} done", set_idx);
                 }
-                // wk.clone().give().await;
                 result
             });
 
             jobs.push((set_idx, job));
         }
 
-        // wk.wait().await;
         let mut worker_error: Option<Error> = None;
         for (set_idx, job) in jobs {
             if let Err(err) = resolve_rebalance_worker_result(set_idx, job.await)
@@ -2680,6 +2731,29 @@ mod rebalance_unit_tests {
         assert!(err.to_string().contains("rebalance worker 7 task join error"));
     }
 
+    #[tokio::test]
+    async fn test_wait_rebalance_entry_tasks_returns_ok_for_successful_tasks() {
+        let tasks = Arc::new(tokio::sync::Mutex::new(vec![tokio::spawn(async { Ok(()) })]));
+
+        super::wait_rebalance_entry_tasks(1, tasks)
+            .await
+            .expect("successful entry tasks should pass");
+    }
+
+    #[tokio::test]
+    async fn test_wait_rebalance_entry_tasks_returns_first_task_error() {
+        let tasks = Arc::new(tokio::sync::Mutex::new(vec![
+            tokio::spawn(async { Ok(()) }),
+            tokio::spawn(async { Err(Error::other("entry failed")) }),
+        ]));
+
+        let err = super::wait_rebalance_entry_tasks(1, tasks)
+            .await
+            .expect_err("entry task failure should be returned");
+
+        assert!(err.to_string().contains("entry failed"));
+    }
+
     #[test]
     fn test_resolve_rebalance_save_task_result_passthrough() {
         assert!(resolve_rebalance_save_task_result(0, Ok(Ok(()))).is_ok());

crates/ecstore/src/rpc/internode_data_transport.rs

@@ -0,0 +1,321 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::disk::error::{Error, Result};
+use crate::disk::{FileReader, FileWriter};
+use crate::rpc::build_auth_headers;
+use async_trait::async_trait;
+use http::{HeaderMap, HeaderValue, Method, header::CONTENT_TYPE};
+use rustfs_config::{DEFAULT_INTERNODE_DATA_TRANSPORT, ENV_RUSTFS_INTERNODE_DATA_TRANSPORT};
+use rustfs_rio::{HttpReader, HttpWriter};
+use std::sync::{Arc, OnceLock};
+use std::time::Duration;
+
+pub const INTERNODE_DATA_TRANSPORT_TCP: &str = "tcp";
+static INTERNODE_DATA_TRANSPORT: OnceLock<std::result::Result<Arc<dyn InternodeDataTransport>, String>> = OnceLock::new();
+
+const READ_FILE_STREAM_PATH: &str = "/rustfs/rpc/read_file_stream";
+const PUT_FILE_STREAM_PATH: &str = "/rustfs/rpc/put_file_stream";
+const WALK_DIR_PATH: &str = "/rustfs/rpc/walk_dir";
+const CONTENT_TYPE_JSON: &str = "application/json";
+
+fn unsupported_transport_message(transport: &str) -> String {
+    format!(
+        "invalid {ENV_RUSTFS_INTERNODE_DATA_TRANSPORT}={transport:?}; supported values: {DEFAULT_INTERNODE_DATA_TRANSPORT}, {INTERNODE_DATA_TRANSPORT_TCP}"
+    )
+}
+
+#[derive(Debug, Clone, Copy, Eq, PartialEq)]
+pub struct InternodeDataTransportCapabilities {
+    pub stream_read: bool,
+    pub stream_write: bool,
+    pub walk_dir: bool,
+    pub registered_memory: bool,
+    pub scatter_gather: bool,
+    pub zero_copy_receive: bool,
+}
+
+impl InternodeDataTransportCapabilities {
+    pub const fn tcp_http() -> Self {
+        Self {
+            stream_read: true,
+            stream_write: true,
+            walk_dir: true,
+            registered_memory: false,
+            scatter_gather: false,
+            zero_copy_receive: false,
+        }
+    }
+}
+
+#[derive(Debug, Clone)]
+pub struct ReadStreamRequest {
+    pub endpoint: String,
+    pub disk: String,
+    pub volume: String,
+    pub path: String,
+    pub offset: usize,
+    pub length: usize,
+}
+
+#[derive(Debug, Clone)]
+pub struct WriteStreamRequest {
+    pub endpoint: String,
+    pub disk: String,
+    pub volume: String,
+    pub path: String,
+    pub append: bool,
+    pub size: i64,
+}
+
+#[derive(Debug, Clone)]
+pub struct WalkDirStreamRequest {
+    pub endpoint: String,
+    pub disk: String,
+    pub body: Vec<u8>,
+    pub stall_timeout: Option<Duration>,
+}
+
+#[async_trait]
+pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
+    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
+    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
+    async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
+    fn name(&self) -> &'static str;
+    fn capabilities(&self) -> InternodeDataTransportCapabilities;
+}
+
+#[derive(Debug, Default)]
+pub struct TcpHttpInternodeDataTransport;
+
+#[async_trait]
+impl InternodeDataTransport for TcpHttpInternodeDataTransport {
+    async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader> {
+        let url = build_read_file_stream_url(&request);
+        let mut headers = json_headers();
+        build_auth_headers(&url, &Method::GET, &mut headers)?;
+        Ok(Box::new(HttpReader::new(url, Method::GET, headers, None).await?))
+    }
+
+    async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter> {
+        let url = build_put_file_stream_url(&request);
+        let mut headers = json_headers();
+        build_auth_headers(&url, &Method::PUT, &mut headers)?;
+        Ok(Box::new(HttpWriter::new(url, Method::PUT, headers).await?))
+    }
+
+    async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader> {
+        let url = build_walk_dir_url(&request);
+        let mut headers = json_headers();
+        build_auth_headers(&url, &Method::GET, &mut headers)?;
+        Ok(Box::new(
+            HttpReader::new_with_stall_timeout(url, Method::GET, headers, Some(request.body), request.stall_timeout).await?,
+        ))
+    }
+
+    fn name(&self) -> &'static str {
+        DEFAULT_INTERNODE_DATA_TRANSPORT
+    }
+
+    fn capabilities(&self) -> InternodeDataTransportCapabilities {
+        InternodeDataTransportCapabilities::tcp_http()
+    }
+}
+
+fn build_read_file_stream_url(request: &ReadStreamRequest) -> String {
+    format!(
+        "{}{}?disk={}&volume={}&path={}&offset={}&length={}",
+        request.endpoint,
+        READ_FILE_STREAM_PATH,
+        urlencoding::encode(&request.disk),
+        urlencoding::encode(&request.volume),
+        urlencoding::encode(&request.path),
+        request.offset,
+        request.length
+    )
+}
+
+fn build_put_file_stream_url(request: &WriteStreamRequest) -> String {
+    format!(
+        "{}{}?disk={}&volume={}&path={}&append={}&size={}",
+        request.endpoint,
+        PUT_FILE_STREAM_PATH,
+        urlencoding::encode(&request.disk),
+        urlencoding::encode(&request.volume),
+        urlencoding::encode(&request.path),
+        request.append,
+        request.size
+    )
+}
+
+fn build_walk_dir_url(request: &WalkDirStreamRequest) -> String {
+    format!("{}{}?disk={}", request.endpoint, WALK_DIR_PATH, urlencoding::encode(&request.disk))
+}
+
+fn json_headers() -> HeaderMap {
+    let mut headers = HeaderMap::new();
+    headers.insert(CONTENT_TYPE, HeaderValue::from_static(CONTENT_TYPE_JSON));
+    headers
+}
+
+fn build_internode_data_transport_result(
+    configured_transport: Option<&str>,
+) -> std::result::Result<Arc<dyn InternodeDataTransport>, String> {
+    match configured_transport.map(str::trim).filter(|transport| !transport.is_empty()) {
+        None => Ok(Arc::new(TcpHttpInternodeDataTransport)),
+        Some(transport)
+            if transport.eq_ignore_ascii_case(DEFAULT_INTERNODE_DATA_TRANSPORT)
+                || transport.eq_ignore_ascii_case(INTERNODE_DATA_TRANSPORT_TCP) =>
+        {
+            Ok(Arc::new(TcpHttpInternodeDataTransport))
+        }
+        Some(transport) => Err(unsupported_transport_message(transport)),
+    }
+}
+
+pub fn build_internode_data_transport(configured_transport: Option<&str>) -> Result<Arc<dyn InternodeDataTransport>> {
+    build_internode_data_transport_result(configured_transport).map_err(Error::other)
+}
+
+pub fn build_internode_data_transport_from_env() -> Result<Arc<dyn InternodeDataTransport>> {
+    let configured_transport = std::env::var(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT).ok();
+    #[cfg(test)]
+    {
+        build_internode_data_transport(configured_transport.as_deref())
+    }
+
+    #[cfg(not(test))]
+    INTERNODE_DATA_TRANSPORT
+        .get_or_init(|| build_internode_data_transport_result(configured_transport.as_deref()))
+        .as_ref()
+        .map(Arc::clone)
+        .map_err(|err| Error::other(err.clone()))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn tcp_http_capabilities_are_behavior_preserving() {
+        let transport = TcpHttpInternodeDataTransport;
+
+        assert_eq!(transport.name(), DEFAULT_INTERNODE_DATA_TRANSPORT);
+        assert_eq!(
+            transport.capabilities(),
+            InternodeDataTransportCapabilities {
+                stream_read: true,
+                stream_write: true,
+                walk_dir: true,
+                registered_memory: false,
+                scatter_gather: false,
+                zero_copy_receive: false,
+            }
+        );
+    }
+
+    #[test]
+    fn read_file_stream_url_encodes_query_values() {
+        let url = build_read_file_stream_url(&ReadStreamRequest {
+            endpoint: "http://node1:9000".to_string(),
+            disk: "http://node1:9000/data/rustfs0".to_string(),
+            volume: ".rustfs.sys".to_string(),
+            path: "pool.bin/../part.1".to_string(),
+            offset: 7,
+            length: 11,
+        });
+
+        assert_eq!(
+            url,
+            "http://node1:9000/rustfs/rpc/read_file_stream?disk=http%3A%2F%2Fnode1%3A9000%2Fdata%2Frustfs0&volume=.rustfs.sys&path=pool.bin%2F..%2Fpart.1&offset=7&length=11"
+        );
+    }
+
+    #[test]
+    fn put_file_stream_url_encodes_query_values() {
+        let url = build_put_file_stream_url(&WriteStreamRequest {
+            endpoint: "http://node1:9000".to_string(),
+            disk: "http://node1:9000/data/rustfs0".to_string(),
+            volume: "bucket".to_string(),
+            path: "object/part.1".to_string(),
+            append: false,
+            size: 4096,
+        });
+
+        assert_eq!(
+            url,
+            "http://node1:9000/rustfs/rpc/put_file_stream?disk=http%3A%2F%2Fnode1%3A9000%2Fdata%2Frustfs0&volume=bucket&path=object%2Fpart.1&append=false&size=4096"
+        );
+    }
+
+    #[test]
+    fn walk_dir_url_encodes_disk_ref() {
+        let url = build_walk_dir_url(&WalkDirStreamRequest {
+            endpoint: "http://node1:9000".to_string(),
+            disk: "http://node1:9000/data/rustfs0".to_string(),
+            body: Vec::new(),
+            stall_timeout: None,
+        });
+
+        assert_eq!(
+            url,
+            "http://node1:9000/rustfs/rpc/walk_dir?disk=http%3A%2F%2Fnode1%3A9000%2Fdata%2Frustfs0"
+        );
+    }
+
+    #[test]
+    fn transport_config_defaults_to_tcp_http() {
+        let transport = build_internode_data_transport(None).unwrap();
+
+        assert_eq!(transport.name(), DEFAULT_INTERNODE_DATA_TRANSPORT);
+    }
+
+    #[test]
+    fn transport_config_blank_value_falls_back_to_default() {
+        let transport = build_internode_data_transport(Some("   ")).unwrap();
+
+        assert_eq!(transport.name(), DEFAULT_INTERNODE_DATA_TRANSPORT);
+    }
+
+    #[test]
+    fn transport_config_accepts_tcp_aliases() {
+        for configured in [
+            DEFAULT_INTERNODE_DATA_TRANSPORT,
+            INTERNODE_DATA_TRANSPORT_TCP,
+            "TCP-HTTP",
+            "TCP",
+        ] {
+            let transport = build_internode_data_transport(Some(configured)).unwrap();
+
+            assert_eq!(transport.name(), DEFAULT_INTERNODE_DATA_TRANSPORT);
+        }
+    }
+
+    #[test]
+    fn transport_config_rejects_unknown_backend() {
+        let err = build_internode_data_transport(Some("rdma")).expect_err("unknown backend should fail closed");
+
+        assert!(err.to_string().contains(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT));
+        assert!(err.to_string().contains("rdma"));
+    }
+
+    #[test]
+    fn cached_transport_config_error_uses_raw_message() {
+        let err = build_internode_data_transport_result(Some("rdma")).expect_err("unknown backend should fail closed");
+
+        assert!(!err.starts_with("io error "));
+        assert!(err.contains(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT));
+        assert!(err.contains("rdma"));
+    }
+}

crates/ecstore/src/rpc/mod.rs

@@ -15,6 +15,7 @@
 mod client;
 mod context_propagation;
 mod http_auth;
+mod internode_data_transport;
 mod peer_rest_client;
 mod peer_s3_client;
 mod remote_disk;
@@ -24,6 +25,10 @@ pub use client::{
     TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
 };
 pub use http_auth::{TONIC_RPC_PREFIX, build_auth_headers, gen_signature_headers, verify_rpc_signature};
+pub use internode_data_transport::{
+    InternodeDataTransport, InternodeDataTransportCapabilities, ReadStreamRequest, TcpHttpInternodeDataTransport,
+    WalkDirStreamRequest, WriteStreamRequest, build_internode_data_transport, build_internode_data_transport_from_env,
+};
 pub use peer_rest_client::PeerRestClient;
 pub use peer_s3_client::{LocalPeerS3Client, PeerS3Client, RemotePeerS3Client, S3PeerSys};
 pub use remote_disk::RemoteDisk;

crates/ecstore/src/rpc/remote_disk.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use crate::disk::error::{Error, Result};
 use crate::disk::{
     CheckPartsResp, DeleteOptions, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader,
     FileWriter, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
@@ -27,16 +28,15 @@ use crate::disk::{disk_store::DiskHealthTracker, error::DiskError, local::ScanGu
 use crate::rpc::client::{
     TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error, node_service_time_out_client,
 };
+use crate::rpc::internode_data_transport::{InternodeDataTransport, ReadStreamRequest, WalkDirStreamRequest, WriteStreamRequest};
 use crate::set_disk::DEFAULT_READ_BUFFER_SIZE;
-use crate::{
-    disk::error::{Error, Result},
-    rpc::build_auth_headers,
-};
 use bytes::Bytes;
 use futures::lock::Mutex;
-use http::{HeaderMap, HeaderValue, Method, header::CONTENT_TYPE};
 use metrics::counter;
 use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
+use rustfs_io_metrics::internode_metrics::{
+    INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_WRITE_ALL, global_internode_metrics,
+};
 use rustfs_protos::evict_failed_connection;
 use rustfs_protos::proto_gen::node_service::RenamePartRequest;
 use rustfs_protos::proto_gen::node_service::{
@@ -46,7 +46,6 @@ use rustfs_protos::proto_gen::node_service::{
     RenameFileRequest, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
     node_service_client::NodeServiceClient,
 };
-use rustfs_rio::{HttpReader, HttpWriter};
 use serde::{Serialize, de::DeserializeOwned};
 use std::{
     io::Cursor,
@@ -68,6 +67,12 @@ use tonic::{Request, service::interceptor::InterceptedService, transport::Channe
 use tracing::{debug, info, warn};
 use uuid::Uuid;
 
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum FailureHealthAction {
+    MarkFailure,
+    IgnoreFailure,
+}
+
 async fn copy_stream_with_buffer<R, W>(reader: &mut R, writer: &mut W, buffer_size: usize) -> io::Result<u64>
 where
     R: AsyncRead + Unpin,
@@ -100,10 +105,11 @@ pub struct RemoteDisk {
     health: Arc<DiskHealthTracker>,
     /// Cancellation token for monitoring tasks
     cancel_token: CancellationToken,
+    data_transport: Arc<dyn InternodeDataTransport>,
 }
 
 impl RemoteDisk {
-    pub async fn new(ep: &Endpoint, opt: &DiskOption) -> Result<Self> {
+    pub(crate) async fn new(ep: &Endpoint, opt: &DiskOption, data_transport: Arc<dyn InternodeDataTransport>) -> Result<Self> {
         let addr = if let Some(port) = ep.url.port() {
             format!("{}://{}:{}", ep.url.scheme(), ep.url.host_str().unwrap(), port)
         } else {
@@ -121,6 +127,7 @@ impl RemoteDisk {
             health_check: opt.health_check && env_health_check,
             health: Arc::new(DiskHealthTracker::new()),
             cancel_token: CancellationToken::new(),
+            data_transport,
         };
         record_drive_runtime_state(ep, RuntimeDriveHealthState::Online);
 
@@ -320,6 +327,21 @@ impl RemoteDisk {
         operation: F,
         timeout_duration: Duration,
     ) -> Result<T>
+    where
+        F: FnOnce() -> Fut,
+        Fut: std::future::Future<Output = Result<T>>,
+    {
+        self.execute_with_timeout_for_op_and_health_action(op, operation, timeout_duration, FailureHealthAction::MarkFailure)
+            .await
+    }
+
+    async fn execute_with_timeout_for_op_and_health_action<T, F, Fut>(
+        &self,
+        op: &'static str,
+        operation: F,
+        timeout_duration: Duration,
+        failure_health_action: FailureHealthAction,
+    ) -> Result<T>
     where
         F: FnOnce() -> Fut,
         Fut: std::future::Future<Output = Result<T>>,
@@ -338,6 +360,17 @@ impl RemoteDisk {
         self.health.last_started.store(now, std::sync::atomic::Ordering::Relaxed);
         self.health.increment_waiting();
 
+        if timeout_duration == Duration::ZERO {
+            let operation_result = operation().await;
+            if operation_result.is_ok() {
+                self.health.log_success();
+            }
+            self.health.decrement_waiting();
+            self.handle_network_like_error(op, timeout_duration, &operation_result, failure_health_action)
+                .await;
+            return operation_result;
+        }
+
         // Execute operation with timeout
         let result = time::timeout(timeout_duration, operation()).await;
 
@@ -348,24 +381,8 @@ impl RemoteDisk {
                     self.health.log_success();
                 }
                 self.health.decrement_waiting();
-                if let Err(err) = &operation_result
-                    && is_network_like_disk_error(err)
-                {
-                    counter!(
-                        "rustfs_drive_op_network_error_total",
-                        "endpoint" => self.endpoint.to_string(),
-                        "op" => op.to_string()
-                    )
-                    .increment(1);
-                    warn!(
-                        endpoint = %self.endpoint,
-                        addr = %self.addr,
-                        op,
-                        timeout_ms = timeout_duration.as_millis(),
-                        "Remote disk operation returned a network-like error"
-                    );
-                    self.mark_faulty_and_evict("operation_network_error").await;
-                }
+                self.handle_network_like_error(op, timeout_duration, &operation_result, failure_health_action)
+                    .await;
                 operation_result
             }
             Err(_) => {
@@ -377,7 +394,9 @@ impl RemoteDisk {
                     "op" => op.to_string()
                 )
                 .increment(1);
-                self.mark_faulty_and_evict("operation_timeout").await;
+                if failure_health_action == FailureHealthAction::MarkFailure {
+                    self.mark_faulty_and_evict("operation_timeout").await;
+                }
                 warn!(
                     endpoint = %self.endpoint,
                     addr = %self.addr,
@@ -390,6 +409,35 @@ impl RemoteDisk {
         }
     }
 
+    async fn handle_network_like_error<T>(
+        &self,
+        op: &'static str,
+        timeout_duration: Duration,
+        operation_result: &Result<T>,
+        failure_health_action: FailureHealthAction,
+    ) {
+        if let Err(err) = operation_result
+            && is_network_like_disk_error(err)
+        {
+            counter!(
+                "rustfs_drive_op_network_error_total",
+                "endpoint" => self.endpoint.to_string(),
+                "op" => op.to_string()
+            )
+            .increment(1);
+            warn!(
+                endpoint = %self.endpoint,
+                addr = %self.addr,
+                op,
+                timeout_ms = timeout_duration.as_millis(),
+                "Remote disk operation returned a network-like error"
+            );
+            if failure_health_action == FailureHealthAction::MarkFailure {
+                self.mark_faulty_and_evict("operation_network_error").await;
+            }
+        }
+    }
+
     async fn mark_faulty_and_evict(&self, reason: &'static str) {
         if self.health.mark_offline(&self.endpoint, reason) {
             self.spawn_recovery_monitor_if_needed();
@@ -1146,32 +1194,27 @@ impl DiskAPI for RemoteDisk {
     async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
         info!("walk_dir {}", self.endpoint.to_string());
 
-        self.execute_with_timeout(
+        self.execute_with_timeout_for_op_and_health_action(
+            "walk_dir",
             || async {
                 let disk = self.disk_ref().await;
-
-                let url = format!("{}/rustfs/rpc/walk_dir?disk={}", self.endpoint.grid_host(), urlencoding::encode(&disk),);
-
                 let opts = serde_json::to_vec(&opts)?;
-
-                let mut headers = HeaderMap::new();
-                headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
-                build_auth_headers(&url, &Method::GET, &mut headers)?;
-
-                let mut reader = HttpReader::new_with_stall_timeout(
-                    url,
-                    Method::GET,
-                    headers,
-                    Some(opts),
-                    Some(get_drive_walkdir_stall_timeout()),
-                )
-                .await?;
+                let mut reader = self
+                    .data_transport
+                    .open_walk_dir(WalkDirStreamRequest {
+                        endpoint: self.endpoint.grid_host(),
+                        disk,
+                        body: opts,
+                        stall_timeout: Some(get_drive_walkdir_stall_timeout()),
+                    })
+                    .await?;
 
                 copy_stream_with_buffer(&mut reader, wr, DEFAULT_READ_BUFFER_SIZE).await?;
 
                 Ok(())
             },
             get_drive_walkdir_timeout(),
+            FailureHealthAction::IgnoreFailure,
         )
         .await
     }
@@ -1196,21 +1239,16 @@ impl DiskAPI for RemoteDisk {
             return Err(DiskError::FaultyDisk);
         }
         let disk = self.disk_ref().await;
-
-        let url = format!(
-            "{}/rustfs/rpc/read_file_stream?disk={}&volume={}&path={}&offset={}&length={}",
-            self.endpoint.grid_host(),
-            urlencoding::encode(&disk),
-            urlencoding::encode(volume),
-            urlencoding::encode(path),
-            offset,
-            length
-        );
-
-        let mut headers = HeaderMap::new();
-        headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
-        build_auth_headers(&url, &Method::GET, &mut headers)?;
-        Ok(Box::new(HttpReader::new(url, Method::GET, headers, None).await?))
+        self.data_transport
+            .open_read(ReadStreamRequest {
+                endpoint: self.endpoint.grid_host(),
+                disk,
+                volume: volume.to_string(),
+                path: path.to_string(),
+                offset,
+                length,
+            })
+            .await
     }
 
     /// Zero-copy read for remote disks falls back to efficient network read.
@@ -1239,21 +1277,16 @@ impl DiskAPI for RemoteDisk {
             return Err(DiskError::FaultyDisk);
         }
         let disk = self.disk_ref().await;
-
-        let url = format!(
-            "{}/rustfs/rpc/put_file_stream?disk={}&volume={}&path={}&append={}&size={}",
-            self.endpoint.grid_host(),
-            urlencoding::encode(&disk),
-            urlencoding::encode(volume),
-            urlencoding::encode(path),
-            true,
-            0
-        );
-
-        let mut headers = HeaderMap::new();
-        headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
-        build_auth_headers(&url, &Method::PUT, &mut headers)?;
-        Ok(Box::new(HttpWriter::new(url, Method::PUT, headers).await?))
+        self.data_transport
+            .open_write(WriteStreamRequest {
+                endpoint: self.endpoint.grid_host(),
+                disk,
+                volume: volume.to_string(),
+                path: path.to_string(),
+                append: true,
+                size: 0,
+            })
+            .await
     }
 
     #[tracing::instrument(level = "debug", skip(self))]
@@ -1270,21 +1303,16 @@ impl DiskAPI for RemoteDisk {
             return Err(DiskError::FaultyDisk);
         }
         let disk = self.disk_ref().await;
-
-        let url = format!(
-            "{}/rustfs/rpc/put_file_stream?disk={}&volume={}&path={}&append={}&size={}",
-            self.endpoint.grid_host(),
-            urlencoding::encode(&disk),
-            urlencoding::encode(volume),
-            urlencoding::encode(path),
-            false,
-            file_size
-        );
-
-        let mut headers = HeaderMap::new();
-        headers.insert(CONTENT_TYPE, HeaderValue::from_static("application/json"));
-        build_auth_headers(&url, &Method::PUT, &mut headers)?;
-        Ok(Box::new(HttpWriter::new(url, Method::PUT, headers).await?))
+        self.data_transport
+            .open_write(WriteStreamRequest {
+                endpoint: self.endpoint.grid_host(),
+                disk,
+                volume: volume.to_string(),
+                path: path.to_string(),
+                append: false,
+                size: file_size,
+            })
+            .await
     }
 
     #[tracing::instrument(level = "debug", skip(self))]
@@ -1527,11 +1555,12 @@ impl DiskAPI for RemoteDisk {
 
         self.execute_with_timeout(
             || async {
+                let data_len = data.len();
                 let disk = self.disk_ref().await;
-                let mut client = self
-                    .get_client()
-                    .await
-                    .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
+                let mut client = self.get_client().await.map_err(|err| {
+                    global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
+                    Error::other(format!("can not get client, err: {err}"))
+                })?;
                 let request = Request::new(WriteAllRequest {
                     disk,
                     volume: volume.to_string(),
@@ -1539,9 +1568,19 @@ impl DiskAPI for RemoteDisk {
                     data,
                 });
 
-                let response = client.write_all(request).await?.into_inner();
+                global_internode_metrics().record_outgoing_request_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
+                let response = match client.write_all(request).await {
+                    Ok(response) => response.into_inner(),
+                    Err(err) => {
+                        global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
+                        return Err(err.into());
+                    }
+                };
+
+                global_internode_metrics().record_sent_bytes_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL, data_len);
 
                 if !response.success {
+                    global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
                     return Err(response.error.unwrap_or_default().into());
                 }
 
@@ -1559,22 +1598,32 @@ impl DiskAPI for RemoteDisk {
         self.execute_with_timeout(
             || async {
                 let disk = self.disk_ref().await;
-                let mut client = self
-                    .get_client()
-                    .await
-                    .map_err(|err| Error::other(format!("can not get client, err: {err}")))?;
+                let mut client = self.get_client().await.map_err(|err| {
+                    global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
+                    Error::other(format!("can not get client, err: {err}"))
+                })?;
                 let request = Request::new(ReadAllRequest {
                     disk,
                     volume: volume.to_string(),
                     path: path.to_string(),
                 });
 
-                let response = client.read_all(request).await?.into_inner();
+                global_internode_metrics().record_outgoing_request_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
+                let response = match client.read_all(request).await {
+                    Ok(response) => response.into_inner(),
+                    Err(err) => {
+                        global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
+                        return Err(err.into());
+                    }
+                };
 
                 if !response.success {
+                    global_internode_metrics().record_error_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
                     return Err(response.error.unwrap_or_default().into());
                 }
 
+                global_internode_metrics()
+                    .record_recv_bytes_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL, response.data.len());
                 Ok(response.data)
             },
             get_max_timeout_duration(),
@@ -1622,6 +1671,7 @@ impl DiskAPI for RemoteDisk {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::rpc::TcpHttpInternodeDataTransport;
     use rustfs_common::GLOBAL_CONN_MAP;
     use std::sync::Once;
     use tokio::io::duplex;
@@ -1659,7 +1709,9 @@ mod tests {
             health_check: false,
         };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
 
         assert!(!remote_disk.is_local());
         assert_eq!(remote_disk.endpoint.url, url);
@@ -1685,7 +1737,9 @@ mod tests {
             health_check: false,
         };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
 
         // Test basic properties
         assert!(!remote_disk.is_local());
@@ -1717,7 +1771,9 @@ mod tests {
             health_check: false,
         };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
         let path = remote_disk.path();
 
         // Remote disk path should be based on the URL path
@@ -1743,7 +1799,9 @@ mod tests {
             health_check: false,
         };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
         assert!(remote_disk.is_online().await);
 
         drop(listener);
@@ -1774,7 +1832,9 @@ mod tests {
             health_check: true,
         };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
         remote_disk.enable_health_check();
 
         // wait for health check connect timeout
@@ -1817,7 +1877,9 @@ mod tests {
             health_check: false,
         };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
 
         // Initially, disk ID should be None
         let initial_id = remote_disk.get_disk_id().await.unwrap();
@@ -1852,7 +1914,9 @@ mod tests {
             health_check: false,
         };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
         assert_eq!(remote_disk.disk_ref().await, endpoint.to_string());
 
         let disk_id = Uuid::new_v4();
@@ -1885,7 +1949,9 @@ mod tests {
                 health_check: false,
             };
 
-            let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+            let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+                .await
+                .unwrap();
 
             assert!(!remote_disk.is_local());
             assert_eq!(remote_disk.host_name(), expected_hostname);
@@ -1911,7 +1977,9 @@ mod tests {
             health_check: false,
         };
 
-        let remote_disk = RemoteDisk::new(&valid_endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&valid_endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
         let location = remote_disk.get_disk_location();
         assert!(location.valid());
         assert_eq!(location.pool_idx, Some(0));
@@ -1927,7 +1995,9 @@ mod tests {
             disk_idx: -1,
         };
 
-        let remote_disk_invalid = RemoteDisk::new(&invalid_endpoint, &disk_option).await.unwrap();
+        let remote_disk_invalid = RemoteDisk::new(&invalid_endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
         let invalid_location = remote_disk_invalid.get_disk_location();
         assert!(!invalid_location.valid());
         assert_eq!(invalid_location.pool_idx, None);
@@ -1951,7 +2021,9 @@ mod tests {
             health_check: false,
         };
 
-        let remote_disk = RemoteDisk::new(&endpoint, &disk_option).await.unwrap();
+        let remote_disk = RemoteDisk::new(&endpoint, &disk_option, Arc::new(TcpHttpInternodeDataTransport))
+            .await
+            .unwrap();
 
         // Test close operation (should succeed)
         let result = remote_disk.close().await;
@@ -1975,6 +2047,7 @@ mod tests {
                 cleanup: false,
                 health_check: false,
             },
+            Arc::new(TcpHttpInternodeDataTransport),
         )
         .await
         .unwrap();
@@ -1994,6 +2067,84 @@ mod tests {
         assert!(!remote_disk.is_online().await, "remote disk should be marked faulty after timeout");
     }
 
+    #[tokio::test]
+    async fn test_execute_with_timeout_can_ignore_remote_timeout_failure() {
+        let url = url::Url::parse("http://remote-timeout-ignored:9000").unwrap();
+        let endpoint = Endpoint {
+            url,
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+
+        let remote_disk = RemoteDisk::new(
+            &endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+            Arc::new(TcpHttpInternodeDataTransport),
+        )
+        .await
+        .unwrap();
+
+        let err = remote_disk
+            .execute_with_timeout_for_op_and_health_action(
+                "walk_dir",
+                || async {
+                    tokio::time::sleep(Duration::from_millis(50)).await;
+                    Ok::<(), Error>(())
+                },
+                Duration::from_millis(10),
+                FailureHealthAction::IgnoreFailure,
+            )
+            .await
+            .expect_err("timeout should fail");
+
+        assert!(err.to_string().contains("timeout"));
+        assert!(remote_disk.is_online().await, "ignored timeout should not mark remote disk faulty");
+    }
+
+    #[tokio::test]
+    async fn test_execute_with_timeout_zero_duration_waits_for_operation() {
+        let url = url::Url::parse("http://remote-no-timeout:9000").unwrap();
+        let endpoint = Endpoint {
+            url,
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+
+        let remote_disk = RemoteDisk::new(
+            &endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+            Arc::new(TcpHttpInternodeDataTransport),
+        )
+        .await
+        .unwrap();
+
+        remote_disk
+            .execute_with_timeout(
+                || async {
+                    tokio::time::sleep(Duration::from_millis(10)).await;
+                    Ok::<(), Error>(())
+                },
+                Duration::ZERO,
+            )
+            .await
+            .expect("zero duration should disable the operation timeout");
+
+        assert!(
+            remote_disk.is_online().await,
+            "successful no-timeout operation should keep remote disk online"
+        );
+    }
+
     #[tokio::test]
     async fn test_execute_with_timeout_evicts_cached_connection() {
         let addr = "http://127.0.0.1:59991".to_string();
@@ -2012,6 +2163,7 @@ mod tests {
                 cleanup: false,
                 health_check: false,
             },
+            Arc::new(TcpHttpInternodeDataTransport),
         )
         .await
         .unwrap();
@@ -2055,6 +2207,7 @@ mod tests {
                 cleanup: false,
                 health_check: false,
             },
+            Arc::new(TcpHttpInternodeDataTransport),
         )
         .await
         .unwrap();
@@ -2102,6 +2255,7 @@ mod tests {
                 cleanup: false,
                 health_check: false,
             },
+            Arc::new(TcpHttpInternodeDataTransport),
         )
         .await
         .unwrap();
@@ -2136,6 +2290,59 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn test_execute_with_timeout_can_ignore_network_like_error() {
+        let addr = "http://127.0.0.1:59995".to_string();
+        let url = url::Url::parse(&format!("{addr}/data")).unwrap();
+        let endpoint = Endpoint {
+            url,
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+
+        let remote_disk = RemoteDisk::new(
+            &endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+            Arc::new(TcpHttpInternodeDataTransport),
+        )
+        .await
+        .unwrap();
+
+        let channel = TonicEndpoint::from_shared(addr.clone()).unwrap().connect_lazy();
+        GLOBAL_CONN_MAP.write().await.insert(addr.clone(), channel);
+
+        let err = remote_disk
+            .execute_with_timeout_for_op_and_health_action(
+                "walk_dir",
+                || async { Err::<(), Error>(DiskError::Io(std::io::Error::new(std::io::ErrorKind::TimedOut, "stall timeout"))) },
+                Duration::from_secs(1),
+                FailureHealthAction::IgnoreFailure,
+            )
+            .await
+            .expect_err("timeout-like operation error should fail");
+
+        assert_eq!(
+            match &err {
+                DiskError::Io(io_err) => io_err.kind(),
+                other => panic!("expected io timeout error, got {other:?}"),
+            },
+            std::io::ErrorKind::TimedOut
+        );
+        assert!(
+            remote_disk.is_online().await,
+            "ignored network-like error should not mark remote disk faulty"
+        );
+        assert!(
+            GLOBAL_CONN_MAP.read().await.contains_key(&addr),
+            "ignored network-like error should not evict cached connection"
+        );
+    }
+
     #[tokio::test]
     async fn test_execute_with_timeout_keeps_remote_disk_online_for_business_error() {
         let addr = "http://127.0.0.1:59994".to_string();
@@ -2154,6 +2361,7 @@ mod tests {
                 cleanup: false,
                 health_check: false,
             },
+            Arc::new(TcpHttpInternodeDataTransport),
         )
         .await
         .unwrap();

crates/ecstore/src/set_disk.rs

@@ -68,7 +68,6 @@ use http::HeaderMap;
 use md5::{Digest as Md5Digest, Md5};
 use rand::{Rng, seq::SliceRandom};
 use regex::Regex;
-use rustfs_common::capacity_scope::{CapacityScope, CapacityScopeDisk, record_capacity_scope, record_global_dirty_scope};
 use rustfs_common::heal_channel::{DriveState, HealChannelPriority, HealItemType, HealOpts, HealScanMode, send_heal_disk};
 use rustfs_config::MI_B;
 use rustfs_filemeta::{
@@ -80,8 +79,11 @@ use rustfs_lock::fast_lock::types::LockResult;
 use rustfs_lock::local_lock::LocalLock;
 use rustfs_lock::{FastLockGuard, LockManager, NamespaceLock, NamespaceLockGuard, NamespaceLockWrapper, ObjectKey};
 use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem};
+use rustfs_object_capacity::capacity_scope::{
+    CapacityScope, CapacityScopeDisk, record_capacity_scope, record_global_dirty_scope,
+};
 use rustfs_rio::{EtagResolvable, HashReader, HashReaderMut, TryGetIndex as _};
-use rustfs_s3_common::EventName;
+use rustfs_s3_types::EventName;
 use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
 use rustfs_utils::http::headers::AMZ_STORAGE_CLASS;
 use rustfs_utils::http::headers::{
@@ -97,7 +99,6 @@ use rustfs_utils::{
     crypto::hex,
     path::{SLASH_SEPARATOR, encode_dir_object, has_suffix, path_join_buf},
 };
-use rustfs_workers::workers::Workers;
 use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_RESTORE};
 use sha2::{Digest, Sha256};
 use std::hash::Hash;

crates/ecstore/src/store_api/readers.rs

@@ -1231,6 +1231,57 @@ mod tests {
         .await;
     }
 
+    #[tokio::test]
+    async fn test_get_object_reader_decrypts_managed_sse_range_on_plaintext_semantics() {
+        async_with_vars([("__RUSTFS_SSE_SIMPLE_CMK", Some(BASE64_STANDARD.encode([0u8; 32])))], async {
+            let plaintext = b"0123456789abcdefghijklmnopqrstuvwxyz".to_vec();
+            let data_key = [0x23; 32];
+            let base_nonce = [0x13; 12];
+            let encrypted_dek = encrypt_managed_dek_for_test(data_key, [0u8; 32]);
+
+            let mut encrypted = Vec::new();
+            rustfs_rio::EncryptReader::new(Cursor::new(plaintext.clone()), data_key, base_nonce)
+                .read_to_end(&mut encrypted)
+                .await
+                .expect("encrypt managed ranged object");
+
+            let object_info = ObjectInfo {
+                size: encrypted.len() as i64,
+                user_defined: HashMap::from([
+                    ("x-amz-server-side-encryption".to_string(), "AES256".to_string()),
+                    ("x-rustfs-encryption-key".to_string(), BASE64_STANDARD.encode(encrypted_dek.as_bytes())),
+                    ("x-rustfs-encryption-iv".to_string(), BASE64_STANDARD.encode(base_nonce)),
+                    ("x-rustfs-encryption-original-size".to_string(), plaintext.len().to_string()),
+                ]),
+                ..Default::default()
+            };
+            let range = HTTPRangeSpec {
+                is_suffix_length: false,
+                start: 5,
+                end: 11,
+            };
+
+            let (mut reader, offset, length) = GetObjectReader::new(
+                Box::new(Cursor::new(encrypted.clone())),
+                Some(range),
+                &object_info,
+                &ObjectOptions::default(),
+                &HeaderMap::new(),
+            )
+            .await
+            .expect("managed encrypted range reads should decrypt inside ecstore");
+
+            let mut actual = Vec::new();
+            reader.read_to_end(&mut actual).await.expect("read managed ranged plaintext");
+
+            assert_eq!(offset, 0);
+            assert_eq!(length, encrypted.len() as i64);
+            assert_eq!(reader.object_info.size, 7);
+            assert_eq!(actual, b"56789ab");
+        })
+        .await;
+    }
+
     #[tokio::test]
     async fn test_get_object_reader_uses_local_managed_fallback_without_env() {
         async_with_vars(

crates/ecstore/src/store_list_objects.rs

@@ -587,11 +587,14 @@ impl ECStore {
         let store = self.clone();
         let opts = o.clone();
         let cancel_rx1 = cancel.clone();
+        let cancel_rx1_for_err = cancel_rx1.clone();
         let err_tx1 = err_tx.clone();
         let job1 = tokio::spawn(async move {
             let mut opts = opts;
             opts.stop_disk_at_limit = true;
-            if let Err(err) = store.list_merged(cancel_rx1, opts, sender).await {
+            if let Err(err) = store.list_merged(cancel_rx1, opts, sender).await
+                && !cancel_rx1_for_err.is_cancelled()
+            {
                 error!("list_merged err {:?}", err);
                 let _ = err_tx1.send(Arc::new(err));
             }
@@ -1011,15 +1014,11 @@ impl ECStore {
 }
 
 async fn gather_results(
-    _rx: CancellationToken,
+    rx: CancellationToken,
     opts: ListPathOptions,
     recv: Receiver<MetaCacheEntry>,
     results_tx: Sender<MetaCacheEntriesSortedResult>,
 ) -> Result<()> {
-    let mut returned = false;
-
-    let mut sender = Some(results_tx);
-
     let mut recv = recv;
     let mut entries = Vec::new();
     while let Some(mut entry) = recv.recv().await {
@@ -1029,10 +1028,6 @@ async fn gather_results(
             entry.name = entry.name.replace("\\", "/");
         }
 
-        if returned {
-            continue;
-        }
-
         // TODO: rx.recv()
 
         // TODO: isLatestDeletemarker
@@ -1065,9 +1060,13 @@ async fn gather_results(
 
         // TODO: Lifecycle
 
+        entries.push(Some(entry));
+
         if opts.limit > 0 && entries.len() >= opts.limit as usize {
-            if let Some(tx) = sender {
-                tx.send(MetaCacheEntriesSortedResult {
+            rx.cancel();
+
+            results_tx
+                .send(MetaCacheEntriesSortedResult {
                     entries: Some(MetaCacheEntriesSorted {
                         o: MetaCacheEntries(entries.clone()),
                         ..Default::default()
@@ -1076,20 +1075,13 @@ async fn gather_results(
                 })
                 .await
                 .map_err(Error::other)?;
-
-                returned = true;
-                sender = None;
-            }
-            continue;
+            return Ok(());
         }
-
-        entries.push(Some(entry));
-        // entries.push(entry);
     }
 
     // finish not full, return eof
-    if let Some(tx) = sender {
-        tx.send(MetaCacheEntriesSortedResult {
+    results_tx
+        .send(MetaCacheEntriesSortedResult {
             entries: Some(MetaCacheEntriesSorted {
                 o: MetaCacheEntries(entries.clone()),
                 ..Default::default()
@@ -1098,7 +1090,6 @@ async fn gather_results(
         })
         .await
         .map_err(Error::other)?;
-    }
 
     Ok(())
 }
@@ -1337,6 +1328,8 @@ impl SetDisks {
 
         let tx1 = sender.clone();
         let tx2 = sender.clone();
+        let cancel_for_send1 = rx.clone();
+        let cancel_for_send2 = rx.clone();
 
         list_path_raw(
             rx,
@@ -1353,8 +1346,11 @@ impl SetDisks {
                 agreed: Some(Box::new(move |entry: MetaCacheEntry| {
                     Box::pin({
                         let value = tx1.clone();
+                        let cancel_token = cancel_for_send1.clone();
                         async move {
-                            if let Err(err) = value.send(entry).await {
+                            if let Err(err) = value.send(entry).await
+                                && !cancel_token.is_cancelled()
+                            {
                                 error!("list_path send fail {:?}", err);
                             }
                         }
@@ -1364,9 +1360,11 @@ impl SetDisks {
                     Box::pin({
                         let value = tx2.clone();
                         let resolver = resolver.clone();
+                        let cancel_token = cancel_for_send2.clone();
                         async move {
                             if let Some(entry) = entries.resolve(resolver)
                                 && let Err(err) = value.send(entry).await
+                                && !cancel_token.is_cancelled()
                             {
                                 error!("list_path send fail {:?}", err);
                             }
@@ -1443,10 +1441,54 @@ fn calc_common_counter(infos: &[DiskInfo], read_quorum: usize) -> u64 {
 
 #[cfg(test)]
 mod test {
-    use super::{ListPathOptions, MAX_OBJECT_LIST, max_keys_plus_one, walk_result_from_set_errors};
+    use super::{ListPathOptions, MAX_OBJECT_LIST, gather_results, max_keys_plus_one, walk_result_from_set_errors};
     use crate::error::StorageError;
+    use rustfs_filemeta::MetaCacheEntry;
+    use std::time::Duration;
+    use tokio::sync::mpsc;
+    use tokio::time::timeout;
+    use tokio_util::sync::CancellationToken;
     use uuid::Uuid;
 
+    fn test_meta_entry(name: &str) -> MetaCacheEntry {
+        MetaCacheEntry {
+            name: name.to_owned(),
+            ..Default::default()
+        }
+    }
+
+    #[tokio::test]
+    async fn gather_results_returns_after_limit_without_waiting_for_input_close() {
+        let (entry_tx, entry_rx) = mpsc::channel(4);
+        let (result_tx, mut result_rx) = mpsc::channel(1);
+
+        entry_tx.send(test_meta_entry("obj-a")).await.unwrap();
+
+        let handle = tokio::spawn(gather_results(
+            CancellationToken::new(),
+            ListPathOptions {
+                bucket: "bucket".to_owned(),
+                limit: 1,
+                incl_deleted: true,
+                ..Default::default()
+            },
+            entry_rx,
+            result_tx,
+        ));
+
+        let result = timeout(Duration::from_secs(1), result_rx.recv())
+            .await
+            .expect("limited result should be sent promptly")
+            .expect("limited result should be present");
+        assert_eq!(result.entries.unwrap().entries().len(), 1);
+
+        timeout(Duration::from_secs(1), handle)
+            .await
+            .expect("gather_results should finish after sending a limited result")
+            .expect("gather_results task should not panic")
+            .expect("gather_results should succeed");
+    }
+
     #[test]
     fn test_max_keys_plus_one_caps_before_lookahead() {
         assert_eq!(max_keys_plus_one(999, true), 1000);

crates/filemeta/src/filemeta.rs

@@ -282,6 +282,11 @@ impl FileMeta {
             fi.version_id = Some(Uuid::nil());
         }
 
+        if fi.data.is_none() && self.data.after_version().is_empty() {
+            let version = FileMetaVersion::from(fi);
+            return self.add_version_filemata(version);
+        }
+
         let version_key = data_key_for_version(fi.version_id);
         let mut next_data = self.data.clone();
 
@@ -317,46 +322,30 @@ impl FileMeta {
         }
 
         let vid = version.get_version_id();
-
-        // Match existing version for replace; null version: None and Some(nil) are equivalent
-        let matches = |h: &Option<Uuid>| {
-            let v_null = vid.is_none() || vid == Some(Uuid::nil());
-            let h_null = h.is_none() || *h == Some(Uuid::nil());
-            (v_null && h_null) || (vid == *h)
+        let vid_is_null = vid.is_none() || vid == Some(Uuid::nil());
+        let existing_idx = if vid_is_null {
+            self.versions
+                .iter()
+                .position(|v| v.header.version_id.is_none() || v.header.version_id == Some(Uuid::nil()))
+        } else {
+            self.versions.iter().position(|v| v.header.version_id == vid)
         };
 
-        if let Some(fidx) = self.versions.iter().position(|v| matches(&v.header.version_id)) {
+        if let Some(fidx) = existing_idx {
             return self.set_idx(fidx, version);
         }
 
-        // append placeholder to find insert position
-        let placeholder = FileMetaShallowVersion {
-            header: FileMetaVersionHeader {
-                mod_time: None, // None sorts before any real mod_time
-                ..Default::default()
-            },
-            meta: Vec::new(),
-        };
-        self.versions.push(placeholder);
-
         let mod_time = version.get_mod_time();
         let new_shallow = FileMetaShallowVersion::try_from(version)?;
-
-        for (idx, exist) in self.versions.iter().enumerate() {
-            let ex_mt = exist.header.mod_time;
-            let insert_here = match (ex_mt, mod_time) {
-                (None, _) => true, // placeholder: always insert before
-                (Some(em), Some(nm)) => em <= nm,
-                (Some(_), None) => false,
-            };
-            if insert_here {
-                self.versions.insert(idx, new_shallow);
-                self.versions.pop(); // remove placeholder
-                return Ok(());
-            }
-        }
-        self.versions.pop(); // remove placeholder on fallback
-        Err(Error::other("add_version failed"))
+        let insert_pos = match mod_time {
+            Some(nm) => self.versions.partition_point(|exist| match exist.header.mod_time {
+                Some(em) => em > nm,
+                None => false,
+            }),
+            None => self.versions.partition_point(|exist| exist.header.mod_time.is_some()),
+        };
+        self.versions.insert(insert_pos, new_shallow);
+        Ok(())
 
         // if !ver.valid() {
         //     return Err(Error::other("attempted to add invalid version"));

crates/filemeta/src/filemeta/inline_data.rs

@@ -13,8 +13,48 @@
 // limitations under the License.
 
 use super::*;
+use std::collections::HashSet;
 
 impl FileMeta {
+    pub fn find_unshared_data_dir_for_version(&self, version_id: Option<Uuid>) -> Option<Uuid> {
+        let vid = version_id.unwrap_or_default();
+        let mut target_data_dir = None;
+        let mut target_selected = false;
+        let mut other_data_dirs = HashSet::new();
+
+        for version in self
+            .versions
+            .iter()
+            .filter(|v| v.header.version_type == VersionType::Object && v.header.uses_data_dir())
+        {
+            let is_target_version = version.header.version_id.unwrap_or_default() == vid;
+            if is_target_version {
+                if target_selected {
+                    continue;
+                }
+
+                target_selected = true;
+                target_data_dir = FileMetaVersion::decode_data_dir_from_meta(&version.meta).unwrap_or_default();
+                if let Some(dir) = target_data_dir
+                    && other_data_dirs.contains(&dir)
+                {
+                    return None;
+                }
+                continue;
+            }
+
+            let dir = FileMetaVersion::decode_data_dir_from_meta(&version.meta).unwrap_or_default();
+            if let Some(dir) = dir {
+                if target_data_dir == Some(dir) {
+                    return None;
+                }
+                other_data_dirs.insert(dir);
+            }
+        }
+
+        target_data_dir
+    }
+
     pub fn shard_data_dir_count(&self, vid: &Option<Uuid>, data_dir: &Option<Uuid>) -> usize {
         let vid = vid.unwrap_or_default();
         self.versions
@@ -58,3 +98,70 @@ impl FileMeta {
             .count()
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use s3s::header::X_AMZ_RESTORE;
+    use time::format_description::well_known::Rfc3339;
+    use time::{Duration, OffsetDateTime};
+
+    fn make_file_info(version_id: Uuid, data_dir: Uuid) -> FileInfo {
+        let restore_header = format!(
+            "ongoing-request=\"false\", expiry-date=\"{}\"",
+            (OffsetDateTime::now_utc() + Duration::days(1))
+                .format(&Rfc3339)
+                .expect("format restore expiry"),
+        );
+        FileInfo {
+            version_id: Some(version_id),
+            data_dir: Some(data_dir),
+            size: 64 * 1024,
+            mod_time: Some(OffsetDateTime::now_utc()),
+            metadata: [
+                ("etag".to_string(), format!("etag-{version_id}")),
+                (X_AMZ_RESTORE.as_str().to_string(), restore_header),
+            ]
+            .into_iter()
+            .collect(),
+            erasure: ErasureInfo {
+                algorithm: ErasureAlgo::ReedSolomon.to_string(),
+                data_blocks: 4,
+                parity_blocks: 2,
+                block_size: 1024 * 1024,
+                index: 1,
+                distribution: vec![1, 2, 3, 4, 5, 6],
+                ..Default::default()
+            },
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn find_unshared_data_dir_for_version_returns_data_dir_when_unique() {
+        let target_version = Uuid::new_v4();
+        let target_data_dir = Uuid::new_v4();
+        let mut meta = FileMeta::new();
+        meta.add_version(make_file_info(target_version, target_data_dir))
+            .expect("seed target version");
+        meta.add_version(make_file_info(Uuid::new_v4(), Uuid::new_v4()))
+            .expect("seed non-shared version");
+
+        let got = meta.find_unshared_data_dir_for_version(Some(target_version));
+        assert_eq!(got, Some(target_data_dir));
+    }
+
+    #[test]
+    fn find_unshared_data_dir_for_version_returns_none_when_shared() {
+        let target_version = Uuid::new_v4();
+        let shared_data_dir = Uuid::new_v4();
+        let mut meta = FileMeta::new();
+        meta.add_version(make_file_info(target_version, shared_data_dir))
+            .expect("seed target version");
+        meta.add_version(make_file_info(Uuid::new_v4(), shared_data_dir))
+            .expect("seed shared version");
+
+        let got = meta.find_unshared_data_dir_for_version(Some(target_version));
+        assert_eq!(got, None);
+    }
+}

crates/filemeta/src/filemeta/version.rs

@@ -312,6 +312,77 @@ pub struct FileMetaVersion {
 }
 
 impl FileMetaVersion {
+    fn decode_data_dir_from_v2_object(buf: &[u8]) -> Result<Option<Uuid>> {
+        let mut cur = std::io::Cursor::new(buf);
+        let mut fields = rmp::decode::read_map_len(&mut cur)?;
+        let mut version_type = VersionType::Invalid;
+
+        while fields > 0 {
+            fields -= 1;
+
+            let key_len = rmp::decode::read_str_len(&mut cur)? as usize;
+            let mut key_buf = vec![0u8; key_len];
+            cur.read_exact(&mut key_buf)?;
+            let key = String::from_utf8(key_buf)?;
+
+            match key.as_str() {
+                "Type" => {
+                    let v: i64 = rmp::decode::read_int(&mut cur)?;
+                    version_type = VersionType::from_u8(v as u8);
+                }
+                "V2Obj" => {
+                    if version_type != VersionType::Object {
+                        skip_msgp_value(&mut cur)?;
+                        continue;
+                    }
+
+                    let mut first = [0u8; 1];
+                    cur.read_exact(&mut first)?;
+                    if first[0] == 0xc0 {
+                        return Ok(None);
+                    }
+
+                    let mut prepend = PrependByteReader {
+                        byte: Some(first[0]),
+                        inner: &mut cur,
+                    };
+                    let mut obj_fields = rmp::decode::read_map_len(&mut prepend)?;
+                    let mut data_dir: Option<Uuid> = None;
+
+                    while obj_fields > 0 {
+                        obj_fields -= 1;
+
+                        let obj_key_len = rmp::decode::read_str_len(&mut prepend)? as usize;
+                        let mut obj_key_buf = vec![0u8; obj_key_len];
+                        prepend.read_exact(&mut obj_key_buf)?;
+                        let obj_key = String::from_utf8(obj_key_buf)?;
+
+                        if obj_key == "DDir" {
+                            let bin_len = rmp::decode::read_bin_len(&mut prepend)? as usize;
+                            if bin_len != 16 {
+                                return Err(Error::other(format!("DDir must be 16 bytes, got {bin_len}")));
+                            }
+                            let mut raw = [0u8; 16];
+                            prepend.read_exact(&mut raw)?;
+                            let id = Uuid::from_bytes(raw);
+                            data_dir = if id.is_nil() { None } else { Some(id) };
+                            break;
+                        }
+
+                        skip_msgp_value(&mut prepend)?;
+                    }
+
+                    return Ok(data_dir);
+                }
+                _ => {
+                    skip_msgp_value(&mut cur)?;
+                }
+            }
+        }
+
+        Ok(None)
+    }
+
     pub fn valid(&self) -> bool {
         if !self.version_type.valid() {
             return false;
@@ -367,6 +438,9 @@ impl FileMetaVersion {
 
     // decode_data_dir_from_meta reads data_dir from meta TODO: directly parse only data_dir from meta buf, msg.skip
     pub fn decode_data_dir_from_meta(buf: &[u8]) -> Result<Option<Uuid>> {
+        if let Ok(data_dir) = Self::decode_data_dir_from_v2_object(buf) {
+            return Ok(data_dir);
+        }
         Ok(Self::try_from(buf)?.get_data_dir())
     }
 
@@ -3213,4 +3287,31 @@ mod tests {
         assert_eq!(fi.mod_time, Some(sample_mod_time()));
         assert_eq!(fi.metadata.get("x-rustfs-test").map(String::as_str), Some("gone"));
     }
+
+    #[test]
+    fn decode_data_dir_from_meta_extracts_v2_object_fast_path() {
+        let data_dir = Uuid::new_v4();
+        let version = FileMetaVersion {
+            version_type: VersionType::Object,
+            object: Some(MetaObject {
+                version_id: Some(Uuid::new_v4()),
+                data_dir: Some(data_dir),
+                erasure_algorithm: ErasureAlgo::ReedSolomon,
+                erasure_m: 2,
+                erasure_n: 4,
+                erasure_block_size: 1024 * 1024,
+                erasure_index: 1,
+                erasure_dist: vec![1, 2, 3, 4, 5, 6],
+                bitrot_checksum_algo: ChecksumAlgo::HighwayHash,
+                size: 64 * 1024,
+                mod_time: Some(OffsetDateTime::now_utc()),
+                ..Default::default()
+            }),
+            ..Default::default()
+        };
+
+        let encoded = version.marshal_msg().expect("marshal");
+        let decoded = FileMetaVersion::decode_data_dir_from_meta(&encoded).expect("decode data_dir");
+        assert_eq!(decoded, Some(data_dir));
+    }
 }

crates/filemeta/src/filemeta_inline.rs

@@ -23,6 +23,141 @@ pub struct InlineData(Vec<u8>);
 const INLINE_DATA_VER: u8 = 1;
 
 impl InlineData {
+    fn contains_key_by<F>(&self, mut should_remove: F) -> Result<bool>
+    where
+        F: FnMut(&[u8]) -> bool,
+    {
+        let buf = self.after_version();
+        if buf.is_empty() {
+            return Ok(false);
+        }
+
+        let mut scan_cur = Cursor::new(buf);
+        let mut scan_fields_len = rmp::decode::read_map_len(&mut scan_cur)? as usize;
+
+        while scan_fields_len > 0 {
+            scan_fields_len -= 1;
+
+            let str_len = rmp::decode::read_str_len(&mut scan_cur)? as usize;
+            let key_start = scan_cur.position() as usize;
+            let key_end = key_start + str_len;
+            scan_cur.set_position(key_end as u64);
+
+            let bin_len = rmp::decode::read_bin_len(&mut scan_cur)? as usize;
+            let value_start = scan_cur.position() as usize;
+            let value_end = value_start + bin_len;
+            scan_cur.set_position(value_end as u64);
+
+            if should_remove(&buf[key_start..key_end]) {
+                return Ok(true);
+            }
+        }
+
+        Ok(false)
+    }
+
+    fn remove_keys_by<F>(&mut self, mut should_remove: F) -> Result<bool>
+    where
+        F: FnMut(&[u8]) -> bool,
+    {
+        let buf = self.after_version();
+        if buf.is_empty() {
+            return Ok(false);
+        }
+
+        let mut cur = Cursor::new(buf);
+        let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize;
+        let mut keys = Vec::with_capacity(fields_len);
+        let mut values = Vec::with_capacity(fields_len);
+        let mut found = false;
+
+        while fields_len > 0 {
+            fields_len -= 1;
+
+            let str_len = rmp::decode::read_str_len(&mut cur)? as usize;
+            let mut field_buf = vec![0u8; str_len];
+            cur.read_exact(&mut field_buf)?;
+
+            let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize;
+            let start = cur.position() as usize;
+            let end = start + bin_len;
+            cur.set_position(end as u64);
+
+            if should_remove(field_buf.as_slice()) {
+                found = true;
+                continue;
+            }
+
+            keys.push(String::from_utf8(field_buf)?);
+            values.push(buf[start..end].to_vec());
+        }
+
+        if !found {
+            return Ok(false);
+        }
+
+        if keys.is_empty() {
+            self.0 = Vec::new();
+            return Ok(true);
+        }
+
+        self.serialize(keys, values)?;
+        Ok(true)
+    }
+
+    fn remove_two_keys_by_bytes(&mut self, first_key: &[u8], second_key: &[u8]) -> Result<bool> {
+        let buf = self.after_version();
+        if buf.is_empty() {
+            return Ok(false);
+        }
+
+        let same = first_key == second_key;
+        let mut cur = Cursor::new(buf);
+        let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize;
+        let mut keys = Vec::with_capacity(fields_len + 1);
+        let mut values = Vec::with_capacity(fields_len + 1);
+        let mut found = false;
+
+        while fields_len > 0 {
+            fields_len -= 1;
+
+            let str_len = rmp::decode::read_str_len(&mut cur)? as usize;
+            let mut field_buf = vec![0u8; str_len];
+            cur.read_exact(&mut field_buf)?;
+
+            let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize;
+            let start = cur.position() as usize;
+            let end = start + bin_len;
+            cur.set_position(end as u64);
+
+            let should_remove = if same {
+                field_buf.as_slice() == first_key
+            } else {
+                field_buf.as_slice() == first_key || field_buf.as_slice() == second_key
+            };
+
+            if should_remove {
+                found = true;
+                continue;
+            }
+
+            keys.push(String::from_utf8(field_buf)?);
+            values.push(buf[start..end].to_vec());
+        }
+
+        if !found {
+            return Ok(false);
+        }
+
+        if keys.is_empty() {
+            self.0 = Vec::new();
+            return Ok(true);
+        }
+
+        self.serialize(keys, values)?;
+        Ok(true)
+    }
+
     pub fn new() -> Self {
         Self(Vec::new())
     }
@@ -183,115 +318,29 @@ impl InlineData {
     }
 
     pub fn remove_key(&mut self, key: &str) -> Result<bool> {
-        let buf = self.after_version();
-        if buf.is_empty() {
+        let key_bytes = key.as_bytes();
+        if !self.contains_key_by(|candidate| candidate == key_bytes)? {
             return Ok(false);
         }
-
-        let mut cur = Cursor::new(buf);
-
-        let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize;
-        let mut keys = Vec::with_capacity(fields_len);
-        let mut values = Vec::with_capacity(fields_len);
-        let mut found = false;
-
-        while fields_len > 0 {
-            fields_len -= 1;
-
-            let str_len = rmp::decode::read_str_len(&mut cur)?;
-
-            let mut field_buff = vec![0u8; str_len as usize];
-
-            cur.read_exact(&mut field_buff)?;
-
-            let find_key = String::from_utf8(field_buff)?;
-
-            let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize;
-            let start = cur.position() as usize;
-            let end = start + bin_len;
-            cur.set_position(end as u64);
-
-            if find_key == key {
-                found = true;
-                continue;
-            }
-
-            keys.push(find_key);
-            values.push(buf[start..end].to_vec());
-        }
-
-        if !found {
-            return Ok(false);
-        }
-
-        if keys.is_empty() {
-            self.0 = Vec::new();
-            return Ok(true);
-        }
-
-        self.serialize(keys, values)?;
-        Ok(true)
+        self.remove_keys_by(|candidate| candidate == key_bytes)
     }
 
     pub fn remove(&mut self, remove_keys: Vec<Uuid>) -> Result<bool> {
-        let buf = self.after_version();
-        if buf.is_empty() {
-            return Ok(false);
-        }
-        let mut cur = Cursor::new(buf);
-
-        let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize;
-        let mut keys = Vec::with_capacity(fields_len + 1);
-        let mut values = Vec::with_capacity(fields_len + 1);
-
-        let remove_key = |found_key: &str| {
-            for key in remove_keys.iter() {
-                if key.to_string().as_str() == found_key {
-                    return true;
-                }
-            }
-            false
-        };
-
-        let mut found = false;
-
-        while fields_len > 0 {
-            fields_len -= 1;
-
-            let str_len = rmp::decode::read_str_len(&mut cur)?;
-
-            let mut field_buff = vec![0u8; str_len as usize];
-
-            cur.read_exact(&mut field_buff)?;
-
-            let find_key = String::from_utf8(field_buff)?;
-
-            let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize;
-            let start = cur.position() as usize;
-            let end = start + bin_len;
-            cur.set_position(end as u64);
-
-            let find_value = &buf[start..end];
-
-            if !remove_key(&find_key) {
-                values.push(find_value.to_vec());
-                keys.push(find_key);
-            } else {
-                found = true;
-            }
+        let mut encoded_keys = Vec::with_capacity(remove_keys.len());
+        for key in remove_keys {
+            let mut buf = Uuid::encode_buffer();
+            encoded_keys.push(key.hyphenated().encode_lower(&mut buf).to_string().into_bytes());
         }
 
-        if !found {
-            return Ok(false);
-        }
+        self.remove_keys_by(|candidate| encoded_keys.iter().any(|key| candidate == key.as_slice()))
+    }
 
-        if keys.is_empty() {
-            self.0 = Vec::new();
-            return Ok(true);
-        }
-
-        self.serialize(keys, values)?;
-        Ok(true)
+    pub fn remove_two(&mut self, first: Uuid, second: Uuid) -> Result<bool> {
+        let mut first_buf = Uuid::encode_buffer();
+        let mut second_buf = Uuid::encode_buffer();
+        let first_key = first.hyphenated().encode_lower(&mut first_buf).as_bytes();
+        let second_key = second.hyphenated().encode_lower(&mut second_buf).as_bytes();
+        self.remove_two_keys_by_bytes(first_key, second_key)
     }
     fn serialize(&mut self, keys: Vec<String>, values: Vec<Vec<u8>>) -> Result<()> {
         assert_eq!(keys.len(), values.len(), "InlineData serialize: keys/values not match");
@@ -319,3 +368,44 @@ impl InlineData {
         Ok(())
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn remove_key_miss_keeps_inline_data_unchanged() {
+        let mut data = InlineData::new();
+        data.replace("keep", b"value".to_vec()).expect("seed inline data");
+        let before = data.as_slice().to_vec();
+
+        let removed = data.remove_key("missing").expect("remove_key should succeed");
+
+        assert!(!removed);
+        assert_eq!(data.as_slice(), before.as_slice());
+    }
+
+    #[test]
+    fn remove_two_removes_only_matching_keys() {
+        let first = Uuid::new_v4();
+        let second = Uuid::new_v4();
+        let keep = Uuid::new_v4();
+        let mut data = InlineData::new();
+        data.replace(first.hyphenated().to_string().as_str(), b"first".to_vec())
+            .expect("seed first key");
+        data.replace(second.hyphenated().to_string().as_str(), b"second".to_vec())
+            .expect("seed second key");
+        data.replace(keep.hyphenated().to_string().as_str(), b"keep".to_vec())
+            .expect("seed keep key");
+
+        let removed = data.remove_two(first, second).expect("remove_two should succeed");
+
+        assert!(removed);
+        assert_eq!(data.find(first.hyphenated().to_string().as_str()).expect("find first"), None);
+        assert_eq!(data.find(second.hyphenated().to_string().as_str()).expect("find second"), None);
+        assert_eq!(
+            data.find(keep.hyphenated().to_string().as_str()).expect("find keep"),
+            Some(b"keep".to_vec())
+        );
+    }
+}

crates/io-core/src/direct_io.rs

@@ -287,7 +287,7 @@ mod tests {
         #[cfg(not(target_os = "linux"))]
         {
             // Non-Linux should return UnsupportedPlatform
-            let file = std::fs::File::open("/dev/null").unwrap();
+            let file = std::fs::File::open(std::env::current_exe().unwrap()).unwrap();
             assert!(matches!(DirectIoReader::new(file, 0, 512), Err(DirectIoError::UnsupportedPlatform)));
         }
     }

crates/io-core/src/io_profile.rs

@@ -219,6 +219,15 @@ pub fn detect_storage_media(storage_detection_enabled: bool, storage_media_overr
         }
     }
 
+    #[cfg(not(any(target_os = "linux", target_os = "macos")))]
+    {
+        if let Ok(media) = detect_platform_storage_media()
+            && media != StorageMedia::Unknown
+        {
+            return media;
+        }
+    }
+
     StorageMedia::Unknown
 }
 

crates/io-metrics/Cargo.toml

@@ -30,6 +30,7 @@ harness = false
 
 [dependencies]
 metrics = { workspace = true }
+rustfs-s3-ops = { workspace = true }
 num_cpus = { workspace = true }
 thiserror = { workspace = true }
 tokio = { workspace = true, features = ["sync","rt"] }

crates/io-metrics/src/internode_metrics.rs

@@ -19,6 +19,19 @@ use std::sync::{
 };
 use std::time::{Duration, SystemTime, UNIX_EPOCH};
 
+pub const INTERNODE_OPERATION_READ_FILE_STREAM: &str = "read_file_stream";
+pub const INTERNODE_OPERATION_PUT_FILE_STREAM: &str = "put_file_stream";
+pub const INTERNODE_OPERATION_WALK_DIR: &str = "walk_dir";
+pub const INTERNODE_OPERATION_GRPC_READ_ALL: &str = "grpc_read_all";
+pub const INTERNODE_OPERATION_GRPC_WRITE_ALL: &str = "grpc_write_all";
+
+const OPERATION_LABEL: &str = "operation";
+const INTERNODE_OPERATION_SENT_BYTES_TOTAL: &str = "rustfs_system_network_internode_operation_sent_bytes_total";
+const INTERNODE_OPERATION_RECV_BYTES_TOTAL: &str = "rustfs_system_network_internode_operation_recv_bytes_total";
+const INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_outgoing_total";
+const INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL: &str = "rustfs_system_network_internode_operation_requests_incoming_total";
+const INTERNODE_OPERATION_ERRORS_TOTAL: &str = "rustfs_system_network_internode_operation_errors_total";
+
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
 pub struct InternodeMetricsSnapshot {
     pub sent_bytes_total: u64,
@@ -54,6 +67,16 @@ impl InternodeMetrics {
         counter!("rustfs_system_network_internode_sent_bytes_total").increment(bytes);
     }
 
+    pub fn record_sent_bytes_for_operation(&self, operation: &'static str, bytes: usize) {
+        self.record_sent_bytes(bytes);
+
+        let bytes = bytes as u64;
+        if bytes == 0 {
+            return;
+        }
+        counter!(INTERNODE_OPERATION_SENT_BYTES_TOTAL, OPERATION_LABEL => operation).increment(bytes);
+    }
+
     pub fn record_recv_bytes(&self, bytes: usize) {
         let bytes = bytes as u64;
         if bytes == 0 {
@@ -63,21 +86,46 @@ impl InternodeMetrics {
         counter!("rustfs_system_network_internode_recv_bytes_total").increment(bytes);
     }
 
+    pub fn record_recv_bytes_for_operation(&self, operation: &'static str, bytes: usize) {
+        self.record_recv_bytes(bytes);
+
+        let bytes = bytes as u64;
+        if bytes == 0 {
+            return;
+        }
+        counter!(INTERNODE_OPERATION_RECV_BYTES_TOTAL, OPERATION_LABEL => operation).increment(bytes);
+    }
+
     pub fn record_outgoing_request(&self) {
         self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
         counter!("rustfs_system_network_internode_requests_outgoing_total").increment(1);
     }
 
+    pub fn record_outgoing_request_for_operation(&self, operation: &'static str) {
+        self.record_outgoing_request();
+        counter!(INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL, OPERATION_LABEL => operation).increment(1);
+    }
+
     pub fn record_incoming_request(&self) {
         self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
         counter!("rustfs_system_network_internode_requests_incoming_total").increment(1);
     }
 
+    pub fn record_incoming_request_for_operation(&self, operation: &'static str) {
+        self.record_incoming_request();
+        counter!(INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL, OPERATION_LABEL => operation).increment(1);
+    }
+
     pub fn record_error(&self) {
         self.errors_total.fetch_add(1, Ordering::Relaxed);
         counter!("rustfs_system_network_internode_errors_total").increment(1);
     }
 
+    pub fn record_error_for_operation(&self, operation: &'static str) {
+        self.record_error();
+        counter!(INTERNODE_OPERATION_ERRORS_TOTAL, OPERATION_LABEL => operation).increment(1);
+    }
+
     pub fn record_dial_result(&self, duration: Duration, success: bool) {
         let elapsed_nanos = duration.as_nanos().min(u128::from(u64::MAX)) as u64;
         self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
@@ -163,4 +211,22 @@ mod tests {
 
         metrics.reset_for_test();
     }
+
+    #[test]
+    fn operation_metrics_also_update_aggregate_snapshot() {
+        let metrics = InternodeMetrics::default();
+
+        metrics.record_sent_bytes_for_operation(INTERNODE_OPERATION_READ_FILE_STREAM, 128);
+        metrics.record_recv_bytes_for_operation(INTERNODE_OPERATION_PUT_FILE_STREAM, 256);
+        metrics.record_outgoing_request_for_operation(INTERNODE_OPERATION_GRPC_WRITE_ALL);
+        metrics.record_incoming_request_for_operation(INTERNODE_OPERATION_GRPC_READ_ALL);
+        metrics.record_error_for_operation(INTERNODE_OPERATION_WALK_DIR);
+
+        let snapshot = metrics.snapshot();
+        assert_eq!(snapshot.sent_bytes_total, 128);
+        assert_eq!(snapshot.recv_bytes_total, 256);
+        assert_eq!(snapshot.outgoing_requests_total, 1);
+        assert_eq!(snapshot.incoming_requests_total, 1);
+        assert_eq!(snapshot.errors_total, 1);
+    }
 }

crates/io-metrics/src/lib.rs

@@ -60,10 +60,12 @@ pub mod capacity_metrics;
 pub mod collector;
 pub mod config;
 pub mod deadlock_metrics;
+pub mod internode_metrics;
 pub mod io_metrics;
 pub mod lock_metrics;
 pub mod performance;
 pub mod process_lock_metrics;
+pub mod s3_api_metrics;
 pub mod sampler;
 pub mod system_path_metrics;
 pub mod timeout_metrics;
@@ -118,6 +120,7 @@ pub use process_lock_metrics::{
     record_write_lock_held_acquire, record_write_lock_held_release, snapshot_process_lock_counts,
     snapshot_process_platform_stats,
 };
+pub use s3_api_metrics::{init_s3_metrics, record_s3_op};
 pub use sampler::{
     ProcessResourceSnapshot, ProcessStatusSnapshot, ProcessSystemSnapshot, snapshot_process_platform, snapshot_process_resource,
     snapshot_process_resource_and_system, snapshot_process_system,