feat(table-catalog): add product API surface (#3320)

* feat(table-catalog): add product API surface

* fix(table-catalog): harden product API commits

* fix(table-catalog): close product API review gaps

* fix(table-catalog): validate rollback metadata before commit

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>

.agents/skills/code-change-verification/SKILL.md

@@ -41,6 +41,19 @@ Use this skill to review code changes consistently before merge, before release,
   - hidden coupling to shared helpers/constants/features
 - If a point is uncertain, mark it as an open question instead of guessing.
 
+#### Rust-specific checks (apply to all Rust changes)
+
+- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
+- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
+- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
+- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
+- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
+- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
+- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
+- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
+- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
+- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
+
 ### 4) Findings-first output
 - Order findings by severity:
   - P0: critical failure, security breach, or data loss risk

.agents/skills/rust-code-quality/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: rust-code-quality
+description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
+---
+
+# Rust Code Quality Gate
+
+Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
+
+## Quick Start
+
+1. Identify changed `.rs` files.
+2. Run automated checks on changed files.
+3. Run manual review checklist on the diff.
+4. Report findings; block merge if P0/P1 issues exist.
+
+## Automated Checks
+
+Run these on every changed `.rs` file (excluding test modules):
+
+```bash
+# 1. unwrap/expect in production code
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
+
+# 2. Silent type truncation via `as` cast
+rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
+
+# 3. String as error type
+rg -n 'Result<.*String>' <changed-files> | grep -v test
+
+# 4. Box<dyn Error> in public APIs
+rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
+
+# 5. println/eprintln in production
+rg -n 'println!\|eprintln!' <changed-files> | grep -v test
+
+# 6. Ordering::Relaxed usage (verify each is intentional)
+rg -n 'Ordering::Relaxed' <changed-files>
+```
+
+## Manual Review Checklist
+
+For every Rust code change, verify:
+
+### Error Handling
+- [ ] No `unwrap()` or `expect()` in production code without justification comment
+- [ ] No `Result<_, String>` in public API signatures
+- [ ] No `Box<dyn Error>` in public trait/struct methods
+- [ ] `Error::source()` is overridden when inner error is stored
+- [ ] Error messages are actionable (what failed, with what input)
+
+### Type Safety
+- [ ] No silent `as` truncation (negative→unsigned, large→small)
+- [ ] `try_into()` or explicit clamping used for numeric conversions
+- [ ] No `f64 as usize` without prior clamping
+
+### Concurrency
+- [ ] Lock acquisition order is documented when multiple locks are used
+- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
+- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
+- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
+
+### Memory and Performance
+- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
+- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
+- [ ] Large buffers wrapped in `Arc` rather than cloned
+- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
+
+### Recursion Safety
+- [ ] Recursive functions have a depth limit or use iterative traversal
+- [ ] Tree/cache traversals handle corrupted/cyclic input safely
+
+### Testing
+- [ ] Every test function has at least one `assert!`
+- [ ] Tests use `.expect("context")` not bare `.unwrap()`
+- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
+
+### Serde
+- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
+- [ ] `#[serde(default)]` not used on security-critical fields without validation
+
+### Code Hygiene
+- [ ] No `#![allow(dead_code)]` at crate root
+- [ ] No camelCase statics or Hungarian notation
+- [ ] New string literals don't duplicate existing constants
+
+## Severity Classification
+
+- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
+
+## Output Template
+
+```
+## Rust Code Quality Report
+
+### Automated Scan
+- unwrap/expect in production: N found
+- as casts: N found
+- String errors: N found
+- println/eprintln: N found
+
+### Findings
+- [P1] `path:line` — description
+  - Fix: ...
+  - Validation: ...
+
+### Verdict
+PASS / BLOCKED (list blocking findings)
+```

.agents/skills/rust-code-quality/references/checklist.md

@@ -0,0 +1,52 @@
+# Rust Code Quality Checklist
+
+Use this as a quick pre-merge checklist for every Rust code change.
+
+## Critical (P0 — block merge)
+
+| Check | Command |
+|-------|---------|
+| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
+| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
+| Lock order consistent across call sites | Manual: trace all lock acquisitions |
+| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
+| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
+
+## High (P1 — must fix)
+
+| Check | Command |
+|-------|---------|
+| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
+| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
+| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
+| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
+| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
+
+## Medium (P2 — should fix)
+
+| Check | Command |
+|-------|---------|
+| Tests have assertions | Manual: check for `assert` in test functions |
+| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
+| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
+| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
+
+## Low (P3 — nice to fix)
+
+| Check | Command |
+|-------|---------|
+| No camelCase statics | `rg 'static ref [a-z]' <files>` |
+| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
+| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
+
+## Quick One-Liner
+
+```bash
+# Run all automated checks on changed files
+CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
+echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
+echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
+echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
+echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
+echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
+```

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,22 +1,32 @@
 ---
 name: rustfs-release-version-bump
-description: Prepare a RustFS release branch like PR #2957 by bumping versioned files, aligning release assets, running required verification, and finishing with commit, push, and gh PR creation. Use when publishing a new RustFS alpha, beta, or stable release.
+description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
 ---
-
 # RustFS Release Version Bump
 
-Use this skill when the task is to prepare a new RustFS version release branch following the pattern validated in PR `#2957`.
+Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
 
-## Read first
+Validated baseline: release pattern used in PR `#2957`.
 
-- Read `AGENTS.md`.
-- Read `.github/pull_request_template.md`.
-- Inspect the current branch diff against `origin/main`.
-- Do not assume every release file should change in the same way every time.
+## Required inputs
 
-## Scope validated by PR #2957
+- Exact target version, for example `1.0.0-beta.4`.
+- Delivery scope:
+- Local only (`edit/verify`).
+- Local + git (`commit/push`).
+- Full GitHub flow (`commit/push/PR`).
 
-The `1.0.0-beta.3` release branch updated these files:
+If target version is missing or ambiguous, stop and ask before editing.
+
+## Read before editing
+
+- `AGENTS.md` (root and nearest path-specific files).
+- `.github/pull_request_template.md`.
+- Current branch status and diff against `origin/main`.
+
+## Default release file scope
+
+Treat the following file list as the default checklist for each release bump:
 
 - `Cargo.toml`
 - `Cargo.lock`
@@ -26,78 +36,89 @@ The `1.0.0-beta.3` release branch updated these files:
 - `helm/rustfs/Chart.yaml`
 - `rustfs.spec`
 
-Treat this file list as the default checklist for future release bumps. Only drop a file if the repository's current release pattern clearly says it is no longer part of the publish flow.
+Only drop a file when the current repository release process clearly no longer requires it.
 
-## Workflow
+## Hard release policy
 
-1. Confirm release intent
-- Identify the target version string exactly, for example `1.0.0-beta.4`.
-- Check whether the user wants only local commit preparation or the full `commit + push + PR` flow.
-- Compare the branch against `origin/main` and isolate only release-related edits.
-- If the target version is not explicit, stop and ask for it before editing.
+- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
+- Helm chart version mapping follows `beta.N -> 0.N.0`.
+- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
+- Do not change these rules without explicit confirmation.
 
-2. Update core Rust workspace versions
-- Bump the workspace version in `Cargo.toml`.
-- Bump all internal workspace crate versions in `Cargo.toml`.
-- Ensure `Cargo.lock` reflects the same release version for workspace packages.
-- Re-read the diff and verify there are no partial version leftovers.
+## Step-by-step workflow
 
-3. Align release assets
-- Update versioned Docker examples in `README.md` and `README_ZH.md` using the `<version>` tag form, for example `rustfs/rustfs:1.0.0-beta.4`.
-- Update `flake.nix` package version.
-- Update `helm/rustfs/Chart.yaml` `appVersion`.
-- Update `helm/rustfs/Chart.yaml` `version` using the release-chart rule:
+1. Confirm intent and isolate scope
+- Confirm target version string exactly.
+- Confirm whether user requested local-only or full GitHub flow.
+- Inspect current branch and ensure only release-related files are touched for this task.
+
+2. Update workspace versions
+- Bump `[workspace.package].version` in `Cargo.toml`.
+- Bump internal workspace crate dependency versions in `Cargo.toml`.
+- Update `Cargo.lock` so workspace package versions match target version.
+- Re-scan for partial leftovers.
+
+3. Update release assets
+- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
+- `flake.nix`: update package version to target version.
+- `helm/rustfs/Chart.yaml`:
+- `appVersion` = target version.
+- `version` follows chart mapping rule, for example:
 - `1.0.0-beta.3` -> `0.3.0`
 - `1.0.0-beta.4` -> `0.4.0`
-- Follow the same pattern for later beta releases unless the repository rule changes.
-- Update `rustfs.spec` release metadata and changelog entry.
-- Set `rustfs.spec` `Release` to the prerelease suffix without the base version, for example `beta.3` for `1.0.0-beta.3`.
-- Keep the release asset changes in a separate commit from the core Rust workspace version bump when the branch contains both.
+- `rustfs.spec`:
+- Set `Release` to prerelease suffix (example `beta.4`).
+- Add/update top changelog entry with exact format:
+- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
+- `- Update RPM package to RustFS 1.0.0-beta.4`
+- Changelog identity and time must come from current environment:
+- `git config --get user.name`
+- `git config --get user.email`
+- `date '+%a %b %d %Y'`
+- Changelog version text must match target release version exactly.
 
-4. Stop and discuss before changing release policy
-- Ask before changing the established Docker tag style away from `<version>`.
-- Ask before changing the established Helm chart version mapping away from `beta.N -> 0.N.0`.
-- Ask before changing the established `rustfs.spec` `Release` rule away from the release suffix form such as `beta.N`.
-- Ask before widening the release scope beyond the files already validated in PR `#2957`.
+4. Verify before shipping
+- Run:
+- `cargo fmt --all`
+- `cargo fmt --all --check`
+- `make pre-commit`
+- If verification passes, run `cargo clean`.
+- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
 
-5. Verify before shipping
-- Run `make pre-commit`.
-- If verification succeeds, run `cargo clean` to remove generated build artifacts before wrapping up.
-- If `make pre-commit` fails, stop and return `BLOCKED`.
+5. Commit strategy
+- Preferred split when both parts changed:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
+- `chore(release): align release assets for <version>` for docs and packaging files.
+- If user asks for one commit, use one commit.
+- Stage only intended release files; do not include unrelated working tree changes.
 
-6. Commit structure
-- Prefer two commits when the change naturally splits:
-- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`
-- `chore(release): align release assets for <version>` for docs and packaging metadata
-- If the user asks for a single commit, follow that request.
+6. Push and PR
+- Push branch:
+- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
+- Create PR with template headings unchanged:
+- `gh pr create --base main --head <branch> --title ... --body-file ...`
+- PR title/body must be English.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands and any `BLOCKED` reason clearly.
 
-7. Push and PR
-- Push the release branch with `git push -u origin <branch>` or `git push` if upstream already exists.
-- Create the PR with `gh pr create --base main --head <branch> --title ... --body-file ...`.
-- Keep the PR title and body in English.
-- Keep the `.github/pull_request_template.md` headings exactly.
-
-## Ready-to-check commands
+## Recommended check commands
 
+- `git status --short --branch`
 - `git diff --name-only origin/main...HEAD`
 - `git diff --stat origin/main...HEAD`
+- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
 - `cargo clean`
-- `git status --short --branch`
 
-## Output expectations
+## Output contract
 
-When using this skill, return:
+When using this skill, always report:
 
-- The files changed for the release bump
-- Any uncertainty that needs user confirmation before editing
-- Verification status
-- Commit messages used
-- Push status and PR URL when the GitHub flow was requested
-
-## Established release policy
-
-- Docs use Docker tags in `<version>` form, not `v<version>`.
-- `helm/rustfs/Chart.yaml` `version` follows `beta.N -> 0.N.0` based on the current release policy.
-- `rustfs.spec` `Release` follows the release suffix form, for example `beta.3`.
-- If any of these rules need to change in the future, pause and confirm before editing.
+- Target version.
+- Files changed.
+- Any assumptions or uncertainties requiring confirmation.
+- Verification result (`PASSED` or `BLOCKED`) with key evidence.
+- Commit message(s) used.
+- Push status and PR URL when GitHub flow is requested.

.agents/skills/security-advisory-lessons/SKILL.md

@@ -53,6 +53,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
 - Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
 - Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough.
 - Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
 
 ### IAM and service accounts
@@ -70,6 +71,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
 - Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
 - Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
 
 ### Secrets, default credentials, and crypto
 - Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
@@ -86,12 +88,13 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 - Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
 - Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
 - RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
 
 ### Browser, CORS, and console surfaces
 - Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
 - Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
 - Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
-- License/version-like metadata endpoints should expose only coarse public data unless authenticated.
+- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
 
 ### Profiling, debug, and health endpoints
 - Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
@@ -115,5 +118,6 @@ Use these prompts while reviewing a diff:
 - Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
 - Does a public/default/empty config change security behavior from fail-closed to fail-open?
 - Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
 - Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -20,7 +20,8 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 - `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
 - `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
 - `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-- `GHSA-x5xv-223c-8vm7`: console license metadata endpoint was public. Lesson: public metadata endpoints should be coarse or authenticated.
+- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
+- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
 
 ### IAM import, service accounts, and privilege boundaries
 
@@ -38,6 +39,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
 - `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
 
 ### Secrets, defaults, and cryptographic misuse
 
@@ -55,6 +57,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
 - `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
 
 ### Browser, CORS, and console isolation
 
@@ -69,6 +72,13 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
+### Serde deserialization and input validation
+
+- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
+- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
+- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
+- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
+
 ## Useful Search Seeds
 
 Use these targeted searches when a diff touches security-sensitive code:
@@ -76,10 +86,12 @@ Use these targeted searches when a diff touches security-sensitive code:
 ```bash
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
 ## Minimum Regression Test Expectations
@@ -87,7 +99,8 @@ rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-
 - Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
+- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.

.config/make/build-docker.mak

@@ -9,7 +9,7 @@ build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
 build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
 build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
 	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
-	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
+	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) --load .
 	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
 
 .PHONY: docker-inspect-multiarch
@@ -19,4 +19,4 @@ docker-inspect-multiarch: ## Check image architecture support
 		exit 1; \
 	fi
 	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
-	docker buildx imagetools inspect $(IMAGE)
+	docker buildx imagetools inspect $(IMAGE)

.config/make/help.mak

@@ -4,7 +4,7 @@
 .PHONY: help
 help: ## Shows This Help Menu
 	echo -e "$$HEADER"
-	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} ; {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}' | sed -e 's/\[36m##/\n[32m##/'
+	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
 
 .PHONY: help-build
 help-build: ## Shows RustFS build help

.config/make/lint-fmt.mak

@@ -21,6 +21,11 @@ unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
 	@echo "🔒 Checking unsafe_code allowances..."
 	./scripts/check_unsafe_code_allowances.sh
 
+.PHONY: architecture-migration-check
+architecture-migration-check: ## Check architecture migration guardrails
+	@echo "🏗️ Checking architecture migration guardrails..."
+	./scripts/check_architecture_migration_rules.sh
+
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -7,5 +7,5 @@ setup-hooks: ## Set up git hooks
 	@echo "✅ Git hooks setup complete!"
 
 .PHONY: pre-commit
-pre-commit: fmt unsafe-code-check clippy-check compilation-check test ## Run pre-commit checks
+pre-commit: fmt unsafe-code-check architecture-migration-check clippy-check compilation-check test ## Run pre-commit checks
 	@echo "✅ All pre-commit checks passed!"

.docker/compose/README.md

@@ -54,6 +54,46 @@ Strict mode is available when you explicitly want `/health/ready == 200` as the
 WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
 ```
 
+### Profiling + Trace Validation
+
+The profiling-focused 4-node compose keeps profiling enabled and points RustFS
+to an OTLP/HTTP collector endpoint:
+
+```bash
+docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
+```
+
+Important behavior notes:
+
+- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
+  traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
+- Startup usually produces logs and metrics first. That does not guarantee
+  visible traces yet.
+- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
+- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
+  many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
+  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
+
+Minimal trace verification flow:
+
+```bash
+# 1. Start the profiling compose with richer span visibility.
+RUSTFS_OBS_LOGGER_LEVEL=debug \
+docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
+
+# 2. Generate real request traffic after startup.
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 3. Then inspect Tempo or Jaeger.
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+If logs and metrics are present but traces are sparse, the most common cause is
+"no real request traffic yet" or "`info` level filtered nested spans", not an
+OTLP routing failure.
+
 ### (Deprecated) Minimal Observability
 
 ```bash

.docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml

@@ -0,0 +1,236 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Profiling-first 4-node local-build compose.
+#
+# Goals:
+# - force linux/amd64 runtime/build on Apple Silicon hosts;
+# - enable RustFS built-in CPU profiling;
+# - keep all tuning knobs host-overridable via env.
+#
+# Observability notes:
+# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
+#   `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
+# - Logs and metrics usually appear during startup. Traces mainly appear after
+#   real HTTP/S3/gRPC requests create spans.
+# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
+#   filters many `debug`-level nested spans. Use `debug` when validating trace
+#   richness rather than collector reachability.
+
+services:
+  node1:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node1_data_0:/data/rustfs0
+      - node1_data_1:/data/rustfs1
+      - node1_data_2:/data/rustfs2
+      - node1_data_3:/data/rustfs3
+    ports:
+      - "9000:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node2:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node2_data_0:/data/rustfs0
+      - node2_data_1:/data/rustfs1
+      - node2_data_2:/data/rustfs2
+      - node2_data_3:/data/rustfs3
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node3:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node3_data_0:/data/rustfs0
+      - node3_data_1:/data/rustfs1
+      - node3_data_2:/data/rustfs2
+      - node3_data_3:/data/rustfs3
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node4:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node4
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node4_data_0:/data/rustfs0
+      - node4_data_1:/data/rustfs1
+      - node4_data_2:/data/rustfs2
+      - node4_data_3:/data/rustfs3
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-cluster-net
+
+volumes:
+  node1_data_0:
+  node1_data_1:
+  node1_data_2:
+  node1_data_3:
+  node2_data_0:
+  node2_data_1:
+  node2_data_2:
+  node2_data_3:
+  node3_data_0:
+  node3_data_1:
+  node3_data_2:
+  node3_data_3:
+  node4_data_0:
+  node4_data_1:
+  node4_data_2:
+  node4_data_3:
+
+networks:
+  rustfs-cluster-net:
+    driver: bridge

.docker/compose/docker-compose.cluster.local-build.yml

@@ -20,13 +20,18 @@ services:
       dockerfile: Dockerfile.source
     hostname: node1
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
       - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
       - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -47,13 +52,18 @@ services:
       dockerfile: Dockerfile.source
     hostname: node2
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
       - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
       - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -74,13 +84,18 @@ services:
       dockerfile: Dockerfile.source
     hostname: node3
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
       - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
       - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"
@@ -101,13 +116,18 @@ services:
       dockerfile: Dockerfile.source
     hostname: node4
     environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
       - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
       - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
       - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
     extra_hosts:
       - "host.docker.internal:host-gateway"

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-latency-stable.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-throughput-max.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60

.docker/compose/docker-compose.cluster.perf-round1-linux.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40

.docker/compose/docker-compose.cluster.perf-round1.override.yml

@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30

.docker/observability/README.md

@@ -90,6 +90,54 @@ docker compose down -v
 - **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
 - **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
 
+### Verifying RustFS Traces
+
+When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
+OTLP/HTTP base URL, for example:
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+RustFS automatically expands that base URL to:
+
+- `/v1/traces`
+- `/v1/metrics`
+- `/v1/logs`
+
+Important behavior notes:
+
+- Logs and metrics usually appear during startup, so seeing those two signals
+  first is expected.
+- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
+  startup, because request-path spans are created on demand.
+- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
+  many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
+  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
+
+Minimal validation flow:
+
+```bash
+# 1. Start this observability stack.
+docker compose up -d
+
+# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+export RUSTFS_OBS_LOGGER_LEVEL=debug
+
+# 3. Generate real request traffic.
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 4. Inspect Grafana or Jaeger.
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+If logs and metrics are present but traces are sparse, the most common cause is
+"no real request traffic yet" or "`info` level filtered nested spans", not an
+OTLP routing failure.
+
 ## Troubleshooting
 
 - **Service Health**: Check the health of services using `docker compose ps`.

.docker/observability/README_ZH.md

@@ -90,6 +90,50 @@ docker compose down -v
 -   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
 -   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
 
+### 验证 RustFS Trace
+
+当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
+OTLP/HTTP 的基础 URL，例如：
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+RustFS 会自动在该基础 URL 后补全：
+
+- `/v1/traces`
+- `/v1/metrics`
+- `/v1/logs`
+
+需要注意：
+
+- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
+- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
+- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
+  如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
+
+最小验证流程：
+
+```bash
+# 1. 启动本目录下的可观测性技术栈。
+docker compose up -d
+
+# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+export RUSTFS_OBS_LOGGER_LEVEL=debug
+
+# 3. 产生真实请求流量。
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 4. 到 Grafana 或 Jaeger 中检查。
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
+“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
+
 ## 故障排除
 
 -   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。

.docker/observability/grafana/dashboards/rustfs.json

@@ -4551,7 +4551,7 @@
                   "index": 0,
                   "text": "INACTIVE"
                 },
-                "to": 1e-09
+                "to": 1e-9
               },
               "type": "range"
             }
@@ -5281,7 +5281,10 @@
       "id": 102,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5368,7 +5371,10 @@
       "id": 103,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5455,7 +5461,10 @@
       "id": 104,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5533,7 +5542,9 @@
       "id": 105,
       "options": {
         "legend": {
-          "calcs": ["mean"],
+          "calcs": [
+            "mean"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5629,7 +5640,10 @@
       "id": 106,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5725,7 +5739,10 @@
       "id": 107,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -5803,7 +5820,10 @@
       "id": 108,
       "options": {
         "legend": {
-          "calcs": ["mean", "max"],
+          "calcs": [
+            "mean",
+            "max"
+          ],
           "displayMode": "table",
           "placement": "bottom"
         },
@@ -8071,6 +8091,374 @@
         "x": 0,
         "y": 195
       },
+      "id": 150,
+      "panels": [],
+      "title": "Object Lock Diagnostics",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "orange",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 196
+      },
+      "id": 151,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (op, mode) (increase(rustfs_object_lock_diag_slow_acquire_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Object Lock Acquires",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "orange",
+                "value": 1
+              },
+              {
+                "color": "red",
+                "value": 5
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 196
+      },
+      "id": 152,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (op, mode) (increase(rustfs_object_lock_diag_slow_hold_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Object Lock Holds",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 204
+      },
+      "id": 153,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "1000 * histogram_quantile(0.95, sum by (le, op, mode) (rate(rustfs_object_lock_diag_acquire_duration_seconds_bucket{job=~\"$job\"}[5m])))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Acquire Duration P95",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ms"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 204
+      },
+      "id": 154,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "1000 * histogram_quantile(0.95, sum by (le, op, mode) (rate(rustfs_object_lock_diag_hold_duration_seconds_bucket{job=~\"$job\"}[5m])))",
+          "legendFormat": "{{op}} | {{mode}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Hold Duration P95",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "mappings": [
+            {
+              "options": {
+                "0": {
+                  "text": "Disabled"
+                },
+                "1": {
+                  "text": "Enabled"
+                }
+              },
+              "type": "value"
+            }
+          ],
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "red",
+                "value": 0
+              },
+              {
+                "color": "green",
+                "value": 1
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 24,
+        "x": 0,
+        "y": 212
+      },
+      "id": 155,
+      "options": {
+        "colorMode": "background",
+        "graphMode": "none",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "percentChangeColorMode": "standard",
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "showPercentChange": false,
+        "textMode": "auto",
+        "wideLayout": true
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "max(rustfs_object_lock_diag_enabled{job=~\"$job\"})",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Object Lock Diagnostics Enabled",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 213
+      },
       "id": 137,
       "panels": [],
       "title": "Debug / Raw Explorer",
@@ -8138,7 +8526,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 196
+        "y": 214
       },
       "id": 138,
       "options": {
@@ -8235,7 +8623,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 196
+        "y": 214
       },
       "id": 139,
       "options": {
@@ -8332,7 +8720,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 204
+        "y": 222
       },
       "id": 140,
       "options": {
@@ -8429,7 +8817,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 204
+        "y": 222
       },
       "id": 141,
       "options": {
@@ -8526,7 +8914,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 212
+        "y": 230
       },
       "id": 142,
       "options": {
@@ -8623,7 +9011,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 212
+        "y": 230
       },
       "id": 143,
       "options": {
@@ -8720,7 +9108,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 220
+        "y": 238
       },
       "id": 144,
       "options": {
@@ -8817,7 +9205,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 220
+        "y": 238
       },
       "id": 145,
       "options": {
@@ -8914,7 +9302,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 228
+        "y": 246
       },
       "id": 146,
       "options": {
@@ -9011,7 +9399,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 228
+        "y": 246
       },
       "id": 147,
       "options": {
@@ -9037,13 +9425,13 @@
             "uid": "${datasource}"
           },
           "editorMode": "code",
-          "expr": "{__name__=~\"rustfs_(notification|audit)_.*\",job=~\"$job\"}",
+          "expr": "{__name__=~\"rustfs_(notification|audit|log_chain)_.*\",job=~\"$job\"}",
           "legendFormat": "{{__name__}}",
           "range": true,
           "refId": "A"
         }
       ],
-      "title": "Notification / Audit (All)",
+      "title": "Notification / Audit / LogChain (All)",
       "type": "timeseries"
     },
     {
@@ -9108,7 +9496,7 @@
         "h": 8,
         "w": 12,
         "x": 0,
-        "y": 236
+        "y": 254
       },
       "id": 148,
       "options": {
@@ -9205,7 +9593,7 @@
         "h": 8,
         "w": 12,
         "x": 12,
-        "y": 236
+        "y": 254
       },
       "id": 149,
       "options": {
@@ -9239,6 +9627,416 @@
       ],
       "title": "Cluster Config / API Rejected (All)",
       "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 262
+      },
+      "id": 507,
+      "panels": [],
+      "title": "Internode Reliability",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ops"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 263
+      },
+      "id": 508,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, classification) (rate(rustfs_system_network_internode_operation_classified_errors_total{job=~\"$job\", operation=~\"put_file_stream|walk_dir|read_file_stream\"}[$__rate_interval]))",
+          "legendFormat": "{{operation}} | {{classification}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Internode Classified Errors",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ops"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 263
+      },
+      "id": 509,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, classification) (rate(rustfs_system_network_internode_operation_retries_total{job=~\"$job\", operation=\"put_file_stream\"}[$__rate_interval]))",
+          "legendFormat": "retry | {{classification}}",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (operation, classification) (rate(rustfs_system_network_internode_operation_retry_successes_total{job=~\"$job\", operation=\"put_file_stream\"}[$__rate_interval]))",
+          "legendFormat": "retry success | {{classification}}",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Internode Open Write Retries",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              },
+              {
+                "color": "yellow",
+                "value": 0.25
+              },
+              {
+                "color": "red",
+                "value": 0.5
+              }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 271
+      },
+      "id": 510,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (classification) (rate(rustfs_system_network_internode_operation_retry_successes_total{job=~\"$job\", operation=\"put_file_stream\"}[$__rate_interval])) / clamp_min(sum by (classification) (rate(rustfs_system_network_internode_operation_retries_total{job=~\"$job\", operation=\"put_file_stream\"}[$__rate_interval])), 1e-9)",
+          "legendFormat": "{{classification}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Internode Retry Success Ratio",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": true,
+            "axisColorMode": "text",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "smooth",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": 0
+              }
+            ]
+          },
+          "unit": "ops"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 271
+      },
+      "id": 511,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "right",
+          "showLegend": true
+        },
+        "tooltip": {
+          "hideZeros": false,
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "12.3.2",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (stage, dominant_error) (rate(rustfs_system_storage_erasure_write_quorum_failures_total{job=~\"$job\"}[$__rate_interval]))",
+          "legendFormat": "{{stage}} | {{dominant_error}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Erasure Write Quorum Failures",
+      "type": "timeseries"
     }
   ],
   "preload": false,
@@ -9382,5 +10180,5 @@
   "timezone": "browser",
   "title": "RustFS",
   "uid": "rustfs-s3",
-  "version": 13
+  "version": 14
 }

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -38,3 +38,16 @@ groups:
         expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
       - record: rustfs:scanner_cycles_success:rate5m
         expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
+
+      - record: rustfs:log_chain_op_event_mismatch:rate5m
+        expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
+
+      - alert: RustFSLogChainOpEventMismatchDetected
+        expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
+        for: 10m
+        labels:
+          severity: warning
+          component: s3-log-chain
+        annotations:
+          summary: "RustFS log-chain op/event mismatch detected"
+          description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."

.github/AGENTS.md

@@ -32,5 +32,9 @@ Current `test-and-lint` gate includes:
 - `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes`
 - `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path`
 - `cargo fmt --all --check`
-- `cargo clippy --all-targets --all-features -- -D warnings`
+- `cargo clippy --all-targets -- -D warnings`
+- `cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2`
+- `cargo test -p rustfs --doc --features rio-v2`
+- `cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings`
 - `./scripts/check_layer_dependencies.sh`
+- `./scripts/check_architecture_migration_rules.sh`

.github/actions/setup/action.yml

@@ -53,18 +53,21 @@ runs:
           musl-tools \
           build-essential \
           pkg-config \
-          libssl-dev
+          libssl-dev \
+          ripgrep \
+          unzip \
+          protobuf-compiler
 
     - name: Install protoc
-      uses: arduino/setup-protoc@v3
+      uses: rustfs/setup-protoc@v3.0.1
       with:
-        version: "33.1"
-        repo-token: ${{ inputs.github-token }}
+        version: "34.1"
+        repo-token: ${{ github.token }}
 
     - name: Install flatc
-      uses: Nugine/setup-flatc@v1.2.4
+      uses: Nugine/setup-flatc@v1
       with:
-        version: "25.9.23"
+        version: "25.12.19"
 
     - name: Install Rust toolchain
       uses: dtolnay/rust-toolchain@stable
@@ -82,7 +85,7 @@ runs:
       uses: taiki-e/install-action@cargo-zigbuild
 
     - name: Install cargo-nextest
-      uses: taiki-e/install-action@cargo-nextest
+      uses: taiki-e/install-action@nextest
 
     - name: Setup Rust cache
       uses: Swatinem/rust-cache@v2

.github/dependabot.yml

@@ -40,6 +40,8 @@ updates:
         versions: [ "1.x" ]
       - dependency-name: "ratelimit"
         versions: [ "2.x" ]
+      - dependency-name: "pyroscope"
+        versions: [ "2.x" ]
     groups:
       s3s:
         update-types:

.github/s3tests/s3tests.conf

@@ -11,8 +11,8 @@
 # host set for RustFS - will be substituted via envsubst
 host = ${S3_HOST}
 
-# port for RustFS
-port = 9000
+# port for RustFS - will be substituted via envsubst
+port = ${S3_PORT}
 
 ## say "False" to disable TLS
 is_secure = False

.github/workflows/architecture-migration-rules.yml

@@ -0,0 +1,43 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Architecture Migration Rules
+
+on:
+  pull_request:
+    branches: [ main ]
+    paths:
+      - "ARCHITECTURE.md"
+      - "docs/architecture/**"
+      - "scripts/check_architecture_migration_rules.sh"
+      - ".github/workflows/architecture-migration-rules.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  architecture-migration-rules:
+    name: Architecture Migration Rules
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install ripgrep
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y ripgrep
+
+      - name: Check architecture migration rules
+        run: ./scripts/check_architecture_migration_rules.sh

.github/workflows/audit.yml

@@ -40,7 +40,7 @@ env:
 jobs:
   security-audit:
     name: Security Audit
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 15
     steps:
       - name: Checkout repository
@@ -65,7 +65,7 @@ jobs:
 
   dependency-review:
     name: Dependency Review
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     if: github.event_name == 'pull_request'
     permissions:
       contents: read
@@ -75,7 +75,7 @@ jobs:
         uses: actions/checkout@v6
 
       - name: Dependency Review
-        uses: actions/dependency-review-action@v4
+        uses: actions/dependency-review-action@v5
         with:
           fail-on-severity: moderate
-          comment-summary-in-pr: true
+          comment-summary-in-pr: always

.github/workflows/build.yml

@@ -73,7 +73,7 @@ jobs:
   # Build strategy check - determine build type based on trigger
   build-check:
     name: Build Strategy Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       build_type: ${{ steps.check.outputs.build_type }}
@@ -146,7 +146,7 @@ jobs:
   # Build RustFS binaries
   prepare-platform-matrix:
     name: Prepare Platform Matrix
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       matrix: ${{ steps.select.outputs.matrix }}
       selected: ${{ steps.select.outputs.selected }}
@@ -169,7 +169,7 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-26-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -202,7 +202,7 @@ jobs:
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
     runs-on: ${{ matrix.os }}
-    timeout-minutes: 60
+    timeout-minutes: 90
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
       # Always enable Tokio unstable features (required by dial9-tokio-telemetry).
@@ -234,8 +234,7 @@ jobs:
         run: |
           mkdir -p ./rustfs/static
           if [[ "${{ matrix.platform }}" == "windows" ]]; then
-            curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300
-            if [[ $? -eq 0 ]]; then
+            if curl.exe --fail -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300; then
               unzip -o console.zip -d ./rustfs/static
               rm console.zip
             else
@@ -244,9 +243,8 @@ jobs:
             fi
           else
             chmod +w ./rustfs/static/LICENSE || true
-            curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-              -o console.zip --retry 3 --retry-delay 5 --max-time 300
-            if [[ $? -eq 0 ]]; then
+            if curl --fail -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
+              -o console.zip --retry 3 --retry-delay 5 --max-time 300; then
               unzip -o console.zip -d ./rustfs/static
               rm console.zip
             else
@@ -551,7 +549,7 @@ jobs:
     name: Build Summary
     needs: [ build-check, build-rustfs ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Build completion summary
         shell: bash
@@ -603,7 +601,7 @@ jobs:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
     outputs:
@@ -689,7 +687,7 @@ jobs:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
       actions: read
@@ -770,7 +768,7 @@ jobs:
     name: Update Latest Version
     needs: [ build-check, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/')
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Update latest.json
         env:
@@ -795,9 +793,10 @@ jobs:
 
           curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
           unzip "$OSSUTIL_ZIP"
-          mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
+          OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
+          mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
           rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
-          chmod +x /usr/local/bin/ossutil
+          chmod +x "$OSSUTIL_BIN"
 
           # Create latest.json
           cat > latest.json << EOF
@@ -811,7 +810,7 @@ jobs:
           EOF
 
           # Upload to OSS
-          ossutil cp latest.json oss://rustfs-version/latest.json --force
+          "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
 
           echo "✅ Updated latest.json for stable release $VERSION"
 
@@ -820,7 +819,7 @@ jobs:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
     if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     permissions:
       contents: write
     steps:

.github/workflows/ci.yml

@@ -54,7 +54,7 @@ on:
       - ".github/workflows/audit.yml"
       - ".github/workflows/performance.yml"
   merge_group:
-    types: [checks_requested]
+    types: [ checks_requested ]
   schedule:
     - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
   workflow_dispatch:
@@ -78,7 +78,7 @@ jobs:
     permissions:
       actions: write
       contents: read
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_skip: ${{ steps.skip_check.outputs.should_skip }}
     steps:
@@ -95,7 +95,7 @@ jobs:
     name: Typos
     needs: skip-check
     if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
       - uses: dtolnay/rust-toolchain@stable
@@ -134,11 +134,42 @@ jobs:
         run: ./scripts/check_unsafe_code_allowances.sh
 
       - name: Run clippy lints
-        run: cargo clippy --all-targets --all-features -- -D warnings
+        run: cargo clippy --all-targets -- -D warnings
 
       - name: Check layered dependencies
         run: ./scripts/check_layer_dependencies.sh
 
+      - name: Check architecture migration rules
+        run: ./scripts/check_architecture_migration_rules.sh
+
+  test-and-lint-rio-v2:
+    name: Test and Lint (rio-v2)
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-test-rio-v2-${{ hashFiles('**/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Run rio-v2 feature tests
+        run: |
+          cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2
+          cargo test -p rustfs --doc --features rio-v2
+
+      - name: Run rio-v2 clippy lints
+        run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
+
   build-rustfs-debug-binary:
     name: Build RustFS Debug Binary
     needs: skip-check
@@ -172,6 +203,39 @@ jobs:
           if-no-files-found: error
           retention-days: 1
 
+  build-rustfs-debug-binary-rio-v2:
+    name: Build RustFS Debug Binary (rio-v2)
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-4
+    timeout-minutes: 30
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-rustfs-debug-binary-rio-v2-${{ hashFiles('**/Cargo.lock') }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build debug binary with rio-v2
+        run: |
+          touch rustfs/build.rs
+          cargo build -p rustfs --bins --features rio-v2 --jobs 2
+
+      - name: Upload debug binary
+        uses: actions/upload-artifact@v6
+        with:
+          name: rustfs-debug-binary-rio-v2
+          path: target/debug/rustfs
+          if-no-files-found: error
+          retention-days: 1
+
   e2e-tests:
     name: End-to-End Tests
     needs: [ skip-check, build-rustfs-debug-binary ]
@@ -182,11 +246,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v6
 
-      - name: Clean up previous test run
-        run: |
-          rm -rf /tmp/rustfs
-          rm -f /tmp/rustfs.log
-
       - name: Download debug binary
         uses: actions/download-artifact@v7
         with:
@@ -204,7 +263,59 @@ jobs:
         with:
           tool: s3s-e2e
           git: https://github.com/s3s-project/s3s.git
-          rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
+          rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
+
+      - name: Run end-to-end tests
+        run: |
+          s3s-e2e --version
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          RUSTFS_TEST_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
+          RUSTFS_TEST_PORT="${RUSTFS_TEST_PORT}" \
+          RUSTFS_TEST_LOG="${RUN_ROOT}/rustfs.log" \
+          ./scripts/e2e-run.sh ./target/debug/rustfs "${RUN_ROOT}/data"
+
+      - name: Upload test logs
+        if: failure()
+        uses: actions/upload-artifact@v6
+        with:
+          name: e2e-test-logs-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
+          retention-days: 3
+
+  e2e-tests-rio-v2:
+    name: End-to-End Tests (rio-v2)
+    needs: [ skip-check, build-rustfs-debug-binary-rio-v2 ]
+    if: needs.skip-check.outputs.should_skip != 'true'
+    runs-on: ubicloud-standard-2
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Clean up previous test run
+        run: |
+          rm -rf /tmp/rustfs
+          rm -f /tmp/rustfs.log
+
+      - name: Download debug binary
+        uses: actions/download-artifact@v7
+        with:
+          name: rustfs-debug-binary-rio-v2
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Setup Rust toolchain for s3s-e2e installation
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Install s3s-e2e test tool
+        uses: taiki-e/cache-cargo-install-action@v2
+        with:
+          tool: s3s-e2e
+          git: https://github.com/s3s-project/s3s.git
+          rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
 
       - name: Run end-to-end tests
         run: |
@@ -215,13 +326,13 @@ jobs:
         if: failure()
         uses: actions/upload-artifact@v6
         with:
-          name: e2e-test-logs-${{ github.run_number }}
+          name: e2e-test-logs-rio-v2-${{ github.run_number }}
           path: /tmp/rustfs.log
           retention-days: 3
 
   s3-implemented-tests:
     name: S3 Implemented Tests
-    needs: [ skip-check, build-rustfs-debug-binary ]
+    needs: [ skip-check, build-rustfs-debug-binary, e2e-tests ]
     if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubicloud-standard-4
     timeout-minutes: 60
@@ -240,10 +351,16 @@ jobs:
 
       - name: Run implemented s3-tests
         run: |
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
           DEPLOY_MODE=binary \
           RUSTFS_BINARY=./target/debug/rustfs \
           TEST_MODE=single \
           MAXFAIL=1 \
+          S3_PORT="${S3_PORT}" \
+          DATA_ROOT="${RUN_ROOT}" \
+          S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
           ./scripts/s3-tests/run.sh
 
       - name: Upload s3 test artifacts

.github/workflows/docker.yml

@@ -73,7 +73,7 @@ jobs:
   # Check if we should build Docker images
   build-check:
     name: Docker Build Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       should_push: ${{ steps.check.outputs.should_push }}
@@ -421,7 +421,7 @@ jobs:
     name: Docker Build Summary
     needs: [ build-check, build-docker ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     steps:
       - name: Docker build completion summary
         run: |

.github/workflows/helm-package.yml

@@ -31,7 +31,7 @@ permissions:
 
 jobs:
   build-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     if: |
       github.event_name == 'workflow_dispatch' ||
       (
@@ -99,7 +99,7 @@ jobs:
           retention-days: 1
 
   publish-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
     needs: [ build-helm-package ]
     if: needs.build-helm-package.result == 'success'
 

.github/workflows/issue-translator.yml

@@ -25,7 +25,7 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubicloud-standard-4
+    runs-on: ubuntu-latest
     steps:
       - uses: usthe/issues-translate-action@v2.7
         with:

.github/workflows/nix-flake-update.yml

@@ -31,7 +31,9 @@ jobs:
   update-flake:
     name: Update flake.lock
     runs-on: ubuntu-latest
-    timeout-minutes: 45
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
@@ -39,6 +41,9 @@ jobs:
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@v3
 
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@v3.20.0
+
       - name: Check Nix flake inputs
         uses: DeterminateSystems/flake-checker-action@v12
 
@@ -56,7 +61,7 @@ jobs:
             nix
             automated
           commit-msg: "chore(deps): update flake.lock"
-          pr-reviewers: houseme, overtrue, majinghe
+          pr-reviewers: overtrue, majinghe
           token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
 
       - name: Log PR details

.github/workflows/nix.yml

@@ -38,39 +38,68 @@ concurrency:
   cancel-in-progress: true
 permissions:
   contents: read
+  id-token: write
 
 jobs:
   nix-validation:
     name: Nix Build & Check
     runs-on: ubuntu-latest
-    timeout-minutes: 45
+    timeout-minutes: 60
+    permissions:
+      contents: read
+      id-token: write
+    env:
+      NIX_CURL_FLAGS: -A cargo/stable
     steps:
       - name: Checkout repository
         uses: actions/checkout@v6
 
       - name: Install Nix
-        uses: cachix/install-nix-action@v31
+        uses: DeterminateSystems/determinate-nix-action@v3.21.0
         with:
-          github_access_token: ${{ secrets.GITHUB_TOKEN }}
-          extra_nix_config: |
-            experimental-features = nix-command flakes
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          extra-conf: |
+            experimental-features = nix-command flakes configurable-impure-env
+            impure-env = NIX_CURL_FLAGS
+            max-jobs = 1
 
-      - name: Setup Magic Nix Cache
-        uses: DeterminateSystems/magic-nix-cache-action@v13
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@v3.21.0
 
-      - name: Setup Flake Checker
+      - name: Check Nix Flake Inputs
         uses: DeterminateSystems/flake-checker-action@v12
+        with:
+          fail-mode: true
+          ignore-missing-flake-lock: false
 
       - name: Verify Flake
         run: |
           echo "Checking flake structure and evaluation..."
           nix flake show
-          nix flake check --print-build-logs
+          for attempt in 1 2 3; do
+            if nix flake check --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix flake check failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done
 
       - name: Build RustFS
         run: |
           echo "Building the default package..."
-          nix build .#default --print-build-logs
+          for attempt in 1 2 3; do
+            if nix build .#default --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix build failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done
 
       - name: Test Binary
         run: |

.github/workflows/performance.yml

@@ -86,7 +86,7 @@ jobs:
 
       - name: Build with profiling optimizations
         run: |
-          RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off" \
+          RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off --cfg tokio_unstable" \
           cargo +nightly build --profile profiling -p rustfs --bins
 
       - name: Run performance profiling

.gitignore

@@ -42,6 +42,11 @@ artifacts/
 PR_DESCRIPTION.md
 scripts/s3-tests/selected_tests.txt
 docs
+__pycache__/
+!docs/
+docs/*
+!docs/architecture/
+!docs/architecture/**
 
 # nix stuff
 result*
@@ -53,3 +58,4 @@ rustfs-webdav.code-workspace
 # Local test/benchmark artifacts
 benchmarks.logs
 tmp/
+crates/*/docs

.vscode/launch.json

@@ -224,6 +224,47 @@
                 "rust"
             ],
         },
+        {
+            "type": "lldb",
+            "request": "launch",
+            "name": "Debug executable 'rustfs' with rio-v2 full matrix",
+            "cargo": {
+                "args": [
+                    "build",
+                    "--bin=rustfs",
+                    "--package=rustfs",
+                    "--features=full,rio-v2"
+                ],
+                "filter": {
+                    "name": "rustfs",
+                    "kind": "bin"
+                }
+            },
+            "env": {
+                "RUST_LOG": "rustfs=error,ecstore=error,s3s=error,iam=error",
+                "RUSTFS_ALLOW_INSECURE_DEFAULT_CREDENTIALS": "true",
+                "RUST_BACKTRACE": "full",
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                "RUSTFS_VOLUMES": "./target/rio-v2/test{1...4}",
+                "RUSTFS_SCANNER_ENABLED": "false",
+                "RUSTFS_HEAL_ENABLED": "false",
+                "RUSTFS_REGION": "us-east-1",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs/rio-v2",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
+                "RUSTFS_COMPRESSION_ENABLED": "true",
+                "RUSTFS_KMS_ENABLE": "true",
+                "RUSTFS_KMS_BACKEND": "local",
+                "RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
+                "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
+                "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}"
+        },
         {
             "name": "E2E test executable target/debug/rustfs",
             "type":  "lldb",
@@ -256,4 +297,4 @@
             ],
         },
     ]
-}
+}

AGENTS.md

@@ -103,6 +103,26 @@ Do not open a PR with code changes when the required checks fail.
 - Use environment variables or vault tooling for sensitive configuration.
 - For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
+## Tools
+
+### xl.meta decode tool Quick Use
+
+```
+cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
+```
+
+## Serde Safety
+
+- Add `#[serde(deny_unknown_fields)]` to structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs).
+- When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level.
+- Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value.
+
+## Naming Conventions
+
+- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
+- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
+- If existing code violates naming conventions, do not widen the violation in new code. Fix opportunistically when touching the surrounding area.
+
 ## Scoped Guidance in This Repository
 
 - `.github/AGENTS.md`

ARCHITECTURE.md

@@ -6,6 +6,8 @@
 > If you want to familiarize yourself with the code base, you are in the right place!
 >
 > See also [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow.
+> See also [docs/architecture](docs/architecture/overview.md) for active
+> architecture migration guardrails.
 
 ## Bird's Eye View
 

CLAUDE.md

@@ -0,0 +1,188 @@
+# RustFS — CLAUDE.md
+
+S3-compatible object store in Rust, derived from MinIO. Erasure-coded, multi-pool, supports ILM tiering/lifecycle.
+
+## Commands
+
+```bash
+cargo build --release --bin rustfs        # production binary
+cargo build                               # dev build
+cargo check -p <crate>                    # fast type-check one crate
+cargo test -p <crate>                     # test one crate
+cargo fmt --all                           # format (required before PR)
+make pre-commit                           # full pre-PR gate (fmt + clippy + test)
+make build-docker BUILD_OS=ubuntu22.04    # Docker cross-build
+```
+
+> **Docker build note**: `buildx build` without `--load` keeps the image in the buildx cache only — `docker run` will use a stale local image. The Makefile already includes `--load`; if you suspect a stale binary, add `--no-cache` to the `buildx build` invocation inside `.config/make/build-docker.mak`.
+
+> Agent/PR rules: see `.github/copilot-instructions.md`.
+> Crate membership: `Cargo.toml` `[workspace].members`.
+> CI gates: `.github/workflows/ci.yml`.
+
+## Workspace layout
+
+```
+rustfs/src/main.rs                          # binary entry point
+crates/ecstore/src/
+  set_disk.rs                               # ErasureSet: transition_object, restore_transitioned_object
+  store.rs / store_api/                     # ECStore trait + ObjectInfo / TransitionedObject types
+  bucket/lifecycle/
+    bucket_lifecycle_ops.rs                 # ILM actions: transition_object, expire_transitioned_object,
+                                            #   get_transitioned_object_reader, gen_transition_objname
+    tier_sweeper.rs                         # background sweep: delete_object_from_remote_tier
+  tier/
+    warm_backend.rs                         # WarmBackend trait (put/get/remove/in_use)
+    warm_backend_s3.rs                      # HTTP-client based (TransitionClient) — used for S3/MinIO
+    warm_backend_s3sdk.rs                   # aws-sdk-s3 based — alternative S3 backend
+    warm_backend_minio.rs / _rustfs.rs / …  # per-provider wrappers (all delegate to _s3 or _s3sdk)
+    tier.rs                                 # TierConfigMgr, new_warm_backend dispatch
+  client/transition_api.rs                  # TransitionClient HTTP plumbing; UploadInfo, to_object_info
+  client/api_put_object_streaming.rs        # put_object_do → UploadInfo (version_id from x-amz-version-id)
+crates/filemeta/src/
+  filemeta.rs                               # FileMeta (xl.meta top-level), is_skip_meta_key
+  filemeta/version.rs                       # FileMetaVersion, MetaObject, MetaDeleteMarker
+                                            #   → to_fileinfo() reads transition_version_id
+                                            #   → set_transition() writes raw UUID bytes
+                                            #   → From<FileInfo> for MetaObject writes all meta
+  fileinfo.rs                               # FileInfo struct (transition_version_id: Option<Uuid>)
+  examples/
+    dump_fileinfo.rs                        # CLI: parse xl.meta, print transition fields + metadata
+    dump_versions.rs                        # CLI: list all versions in xl.meta
+crates/utils/src/http/metadata_compat.rs    # SUFFIX_* constants, insert_bytes/get_bytes (dual RustFS+MinIO keys)
+```
+
+## Metadata key conventions
+
+Internal metadata is stored under **both** `x-rustfs-internal-<suffix>` and `x-minio-internal-<suffix>` for MinIO interoperability. `get_bytes` prefers the RustFS key with MinIO fallback.
+
+Key suffixes (from `metadata_compat.rs`):
+| Suffix | Meaning |
+|--------|---------|
+| `transition-status` | `"complete"` when tiered |
+| `transitioned-object` | tier key path (without prefix) |
+| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) |
+| `transition-tier` | tier name |
+| `tier-free-versionID` | delete-marker version for free-version sweep |
+
+## Tier / ILM transition architecture
+
+### Transition flow (hot → cold)
+1. `transition_object` (lifecycle_ops) → `ECStore::transition_object` → `set_disk.rs`
+2. `gen_transition_objname(bucket)` → `{sha256_hash[0..16]}/{uuid[0..2]}/{uuid[2..4]}/{uuid}` (unique per object version)
+3. `tgt_client.put_with_meta(dest_obj, …)` → returns `rv: String` (remote S3 version_id, or `""`)
+4. `fi.transition_version_id = if rv.is_empty() { None } else { Some(Uuid::parse_str(&rv)?) }`
+5. `fi.transitioned_objname = dest_obj` (without tier prefix)
+6. Written to xl.meta via `MetaObject::from(FileInfo)` → `insert_bytes(SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes())` (16 raw bytes)
+
+### Tier GET flow (restore/read)
+`get_transitioned_object_reader` (lifecycle_ops):
+- reads `oi.transitioned_object.name` (= `fi.transitioned_objname`)
+- reads `oi.transitioned_object.version_id` (= `fi.transition_version_id.to_string()` or `""`)
+- calls `warm_backend.get(name, version_id, opts)`
+- `warm_backend_s3.rs::get`: adds `?versionId=…` only when `rv != ""`
+
+### Tier prefix handling
+`WarmBackendS3::get_dest(object)` prepends `self.prefix` to the object name.
+`transitioned_objname` is stored **without** the prefix — `get_dest` adds it on every call.
+
+### xl.meta on disk
+Path: `{disk}/{bucket}/{object}/xl.meta` — one per erasure shard disk.
+All shards should be identical for a healthy object.
+
+## Known bugs & fixes
+
+### Bug 1: `NoSuchVersion` on tier GET — nil UUID sent as versionId
+**Root cause**: `transitioned-versionID` metadata key exists with empty string value (0 bytes). Old reading code:
+```rust
+// OLD — unwrap_or_default() converts 0-byte or wrong-length slice to Uuid::nil()
+get_bytes(…).map(|v| Uuid::from_slice(v.as_slice()).unwrap_or_default())
+// → Some(Uuid::nil()) → sends ?versionId=00000000-… → NoSuchVersion
+```
+**Fix** (version.rs, `MetaObject::to_fileinfo` + `MetaDeleteMarker::to_fileinfo`):
+```rust
+get_bytes(…)
+    .and_then(|v| Uuid::from_slice(v.as_slice()).ok())  // None for wrong-length bytes
+    .filter(|u| !u.is_nil())                            // None for nil UUID (old write-back)
+```
+**Regression tests** (`crates/filemeta/src/filemeta/version.rs` `mod tests`): 6 tests cover absent key, empty bytes, nil UUID, and valid UUID round-trip for both `MetaObject` and `MetaDeleteMarker` paths.
+
+### Bug 2: `warm_backend_s3sdk.rs` ignored rv and range opts
+**Fix**: added `req.version_id(rv)` and `req.range(…)` to GET; `req.version_id(rv)` to DELETE.
+
+### Bug 4: `set_disk::copy_object` returns 501 for tiered objects (storage class restore)
+**Root cause**: `set_disk::copy_object` immediately returns `StorageError::NotImplemented` when `src_info.metadata_only = false`. For tiered objects, `metadata_only` is never set to `true` (guarded by `transitioned_object.tier.is_empty()`). So `mc cp --storage-class STANDARD obj obj` on a tiered object always returns 501.
+**Fix** (`crates/ecstore/src/set_disk.rs`, `copy_object`):
+```rust
+if !src_info.metadata_only {
+    if path_join_buf(&[src_bucket, src_object]) == path_join_buf(&[dst_bucket, dst_object]) {
+        if let Some(mut put_reader) = src_info.put_object_reader.take() {
+            return self.put_object(dst_bucket, dst_object, &mut put_reader, dst_opts).await;
+        }
+    }
+    return Err(StorageError::NotImplemented);
+}
+```
+When a self-copy has a `put_object_reader` (data already fetched from tier in `execute_copy_object`), writes it back locally via `put_object`, effectively de-tiering the object.
+**How `mc cp --storage-class STANDARD` flows**:
+1. mc sends `PUT /bucket/key` with `x-amz-copy-source`, `x-amz-metadata-directive: REPLACE`, `x-amz-storage-class: STANDARD`
+2. `execute_copy_object` → `get_object_reader` fetches data from tier backend → stores in `src_info.put_object_reader`
+3. `store.copy_object(...)` → now calls `put_object` with tier data and STANDARD storage class in `dst_opts`
+4. New xl.meta written locally with STANDARD class, no tier metadata → object de-tiered
+
+### Bug 3 (open): race in `expire_transitioned_object`
+Order is: delete remote tier version → delete local object.
+A concurrent GET between those two steps fetches a valid stored version_id but the tier version is already gone → `NoSuchVersion`.
+The proper fix is to delete local metadata first (making the object unreachable) before deleting the remote tier version.
+
+## Debugging tier issues
+
+### Inspect xl.meta directly
+```bash
+cargo build -p rustfs-filemeta --example dump_fileinfo
+./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta
+# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id
+```
+`transition_ver_id: <none>` → no versionId will be sent to tier (correct for non-versioned tier bucket).
+`transition_ver_id: <uuid>` → that UUID will be sent as `?versionId=<uuid>`.
+
+### Check what versionId is being sent at runtime
+Enable debug logging:
+```bash
+RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs …
+```
+Log line: `fetching transitioned object from tier` (DEBUG before request).
+Log line: `tier GET failed` (ERROR on failure, includes `tier_version_id`).
+
+### Metadata key to watch
+```
+x-minio-internal-transitioned-versionID=   ← empty string = will cause NoSuchVersion with old code
+x-rustfs-internal-transitioned-versionID=  ← same
+```
+If both are empty string, the object was transitioned to a non-versioned tier bucket. The versionId should NOT be sent — fixed by Bug 1 above.
+
+## Common patterns
+
+### Writing internal metadata (binary values)
+```rust
+insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes().to_vec());
+// stores under both x-rustfs-internal-* and x-minio-internal-* keys
+```
+
+### Reading internal metadata (binary values)
+```rust
+get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID)
+    .and_then(|v| Uuid::from_slice(v.as_slice()).ok())
+    .filter(|u| !u.is_nil())
+// Returns None for: absent, wrong-length bytes, nil UUID
+```
+
+### WarmBackend trait
+```rust
+put_with_meta(object, reader, length, meta) -> Result<String>  // returns S3 version_id or ""
+put(object, reader, length) -> Result<String>
+get(object, rv, opts) -> Result<ReadCloser>  // rv="" means no versionId
+remove(object, rv) -> Result<()>
+in_use() -> Result<bool>
+```
+`rv` = remote version, always pass as empty string when `transition_version_id` is None.

CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 ## 📋 Code Quality Requirements
 
-For instructions on setting up and running the local development environment, please see [Development Guide](docs/DEVELOPMENT.md).
+This guide covers the local development environment and the checks expected before contributing.
 
 ### 🔧 Code Formatting Rules
 

Cargo.toml

@@ -15,10 +15,10 @@
 [workspace]
 members = [
     "rustfs", # Core file system implementation
-    "crates/appauth", # Application authentication and authorization
     "crates/audit", # Audit target management system with multi-target fan-out
     "crates/checksums", # client checksums
     "crates/common", # Shared utilities and data structures
+    "crates/data-usage", # Shared data usage models and algorithms
     "crates/config", # Configuration management
     "crates/credentials", # Credential management system
     "crates/crypto", # Cryptography and security features
@@ -38,16 +38,20 @@ members = [
     "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
+    "crates/rio-v2", # Next-generation Rust I/O compatibility layer
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
-    "crates/s3-common", # Common utilities and data structures for S3 compatibility
+    "crates/s3-types", # S3 event type definitions
+    "crates/s3-ops", # S3 operation definitions and mapping
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
     "crates/scanner", # Scanner for data integrity checks and health monitoring
+    "crates/security-governance", # Security governance contracts
     "crates/signer", # client signer
+    "crates/storage-api", # Storage API contracts
     "crates/targets", # Target-specific configurations and utilities
     "crates/trusted-proxies", # Trusted proxies management
+    "crates/tls-runtime", # Project-wide TLS runtime foundation
     "crates/utils", # Utility functions and helpers
-    "crates/workers", # Worker thread pools and task scheduling
     "crates/io-metrics", # Zero-copy metrics collection for performance analysis
     "crates/io-core", # Zero-copy core reader and writer implementations
     "crates/zip", # ZIP file handling and compression
@@ -58,8 +62,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.95.0"
-version = "1.0.0-beta.3"
+rust-version = "1.96.0"
+version = "1.0.0-beta.8"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -76,76 +80,80 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.3" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.3" }
-rustfs-appauth = { path = "crates/appauth", version = "1.0.0-beta.3" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.3" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.3" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.3" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.3" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.3" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.3" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.3" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.3" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.3" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.3" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.3" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.3" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.3" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.3" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.3" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.3" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.3" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.3" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.3" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.3" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.3" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.3" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.3" }
-rustfs-s3-common = { path = "crates/s3-common", version = "1.0.0-beta.3" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.3" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.3" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.3" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.3" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.3" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.3" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.3" }
-rustfs-workers = { path = "crates/workers", version = "1.0.0-beta.3" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.3" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.8" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.8" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.8" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.8" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.8" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.8" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.8" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.8" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.8" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.8" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.8" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.8" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.8" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.8" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.8" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.8" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.8" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.8" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.8" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.8" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.8" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.8" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.8" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.8" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.8" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.8" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.8" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.8" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.8" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.8" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.8" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.8" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.8" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.8" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.8" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.8" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.8" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.8" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.8" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.8" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-mysql_async = { version = "0.36.1", default-features = false, features = ["default-rustls", "tracing"], git = "https://github.com/blackbeam/mysql_async", rev = "98d3d8067efdf97d3e93cdca7b9231753c904aca" }
+mysql_async = { version = "0.37", default-features = false, features = ["default-rustls", "tracing"] }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
-async-nats = "0.48.0"
+async-nats = "0.49.1"
 axum = "0.8.9"
 futures = "0.3.32"
 futures-core = "0.3.32"
 futures-util = "0.3.32"
 pollster = "0.4.0"
-pulsar = { version = "6.7.2", default-features = false, features = ["tokio-rustls-runtime"] }
-lapin = { version = "4.7.2", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
-hyper = { version = "1.9.0", features = ["http2", "http1", "server"] }
+pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime"] }
+lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
+hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
 hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
 hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
-http = "1.4.0"
+http = "1.4.2"
 http-body = "1.0.1"
 http-body-util = "0.1.3"
-reqwest = { version = "0.13.3", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
 rustfs-kafka-async = { version = "1.2.0" }
-socket2 = { version = "0.6.3", features = ["all"] }
+socket2 = { version = "0.6.4", features = ["all"] }
 tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
 tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
 tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic = { version = "0.14.6", features = ["gzip"] }
+tonic = { version = "0.14.6", features = ["gzip", "deflate"] }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
 tower = { version = "0.5.3", features = ["timeout"] }
-tower-http = { version = "0.6.10", features = ["cors"] }
+tower-http = { version = "0.6.11", features = ["cors"] }
 
 # Serialization and Data Formats
 bytes = { version = "1.11.1", features = ["serde"] }
@@ -153,22 +161,22 @@ bytesize = "2.3.1"
 byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
-prost = "0.14.3"
-quick-xml = "0.39.4"
+prost = "0.14.4"
+quick-xml = "0.40.1"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.228", features = ["derive"] }
-serde_json = { version = "1.0.149", features = ["raw_value"] }
+serde_json = { version = "1.0.150", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
-aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
+aes-gcm = { version = "0.11.0-rc.4", features = ["rand_core"] }
 argon2 = { version = "0.6.0-rc.8" }
 blake2 = "0.11.0-rc.6"
 chacha20poly1305 = { version = "0.11.0-rc.3" }
-crc-fast = "1.9.0"
+crc-fast = "1.10.0"
 hmac = { version = "0.13.0" }
-jsonwebtoken = { version = "10.3.0", features = ["aws_lc_rs"] }
+jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
 openidconnect = { version = "4.0", default-features = false }
 pbkdf2 = "0.13.0"
 rsa = { version = "0.10.0-rc.18" }
@@ -181,31 +189,30 @@ subtle = "2.6"
 zeroize = { version = "1.8.2", features = ["derive"] }
 
 # Time and Date
-chrono = { version = "0.4.44", features = ["serde"] }
+chrono = { version = "0.4.45", features = ["serde"] }
 humantime = "2.3.0"
-jiff = { version = "0.2.24", features = ["serde"] }
+jiff = { version = "0.2.28", features = ["serde"] }
 time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Database
 deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
 tokio-postgres = { version = "0.7", default-features = false, features = ["runtime", "with-serde_json-1"] }
-tokio-postgres-rustls = "0.13"
+tokio-postgres-rustls = "0.14.0"
 
 # Utilities and Tools
 anyhow = "1.0.102"
 arc-swap = "1.9.1"
-astral-tokio-tar = "0.6.1"
-atoi = "2.0.0"
+astral-tokio-tar = "0.6.2"
+atoi = "3.0.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.8.16" }
+aws-config = { version = "1.8.18" }
 aws-credential-types = { version = "1.2.14" }
-aws-sdk-s3 = { version = "1.132.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-http-client = { version = "1.1.12", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-types = { version = "1.4.7" }
-backtrace = "0.3.76"
+aws-sdk-s3 = { version = "1.135.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.1.13", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-types = { version = "1.4.9" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
-brotli = "8.0.2"
+brotli = "8.0.3"
 clap = { version = "4.6.1", features = ["derive", "env"] }
 const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
@@ -214,14 +221,14 @@ crossbeam-queue = "0.3.12"
 crossbeam-channel = "0.5.15"
 crossbeam-deque = "0.8.6"
 crossbeam-utils = "0.8.21"
-datafusion = "53.1.0"
+datafusion = "54.0.0"
 derive_builder = "0.20.2"
-enumset = "1.1.12"
+enumset = "1.1.13"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.12.0"
-google-cloud-auth = "1.10.0"
+google-cloud-storage = "1.14.0"
+google-cloud-auth = "1.12.0"
 hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
 hex = "0.4.3"
 hex-simd = "0.8.0"
@@ -230,7 +237,7 @@ ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
 libc = "0.2.186"
 libsystemd = "0.7.2"
-local-ip-address = "0.6.12"
+local-ip-address = "0.6.13"
 memmap2 = "0.9.10"
 lz4 = "1.28.1"
 matchit = "0.9.2"
@@ -253,23 +260,23 @@ ratelimit = "0.10.1"
 rayon = "1.12.0"
 reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
 reed-solomon-simd = "3.1.0"
-regex = { version = "1.12.3" }
-rumqttc = { package = "rumqttc-next", version = "0.33.0", features = ["websocket"] }
-redis = { version = "1.2.1", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
+regex = { version = "1.12.4" }
+rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
+redis = { version = "1.2.2", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.2" }
-s3s = { git = "https://github.com/rustfs/s3s", rev = "a3b16608df35aaeed8fff08b4988d03f4ca9445b", features = ["minio"] }
-serial_test = "3.4.0"
+s3s = { git = "https://github.com/rustfs/s3s", rev = "507e1312b211c3ddc214b03875d6fabd15d22ed5", features = ["minio"] }
+serial_test = "3.5.0"
 shadow-rs = { version = "2.0.0", default-features = false }
 siphasher = "1.0.3"
 smallvec = { version = "1.15.1", features = ["serde"] }
 smartstring = "1.0.1"
-snafu = "0.9.0"
+snafu = "0.9.1"
 snap = "1.1.1"
-starshard = { version = "1.1.0", features = ["rayon", "async", "serde"] }
+starshard = { version = "2.2.0", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.39.0"
+sysinfo = "0.39.3"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -277,12 +284,12 @@ thiserror = "2.0.18"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
-tracing-opentelemetry = "0.32.1"
+tracing-opentelemetry = { version = "0.33" }
 tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.23.1", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.23.3", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
 wildmatch = { version = "2.6.1", features = ["serde"] }
@@ -292,23 +299,23 @@ zip = "8.6.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
-metrics = "0.24.5"
+metrics = "0.24.6"
 dial9-tokio-telemetry = "0.3"
-opentelemetry = { version = "0.31.0" }
-opentelemetry-appender-tracing = { version = "0.31.1", features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes", "spec_unstable_logs_enabled"] }
-opentelemetry-otlp = { version = "0.31.1", features = ["gzip-http", "reqwest-rustls"] }
-opentelemetry_sdk = { version = "0.31.0" }
-opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
-opentelemetry-stdout = { version = "0.31.0" }
-pyroscope = { version = "2.0.3", features = ["backend-pprof-rs"] }
+opentelemetry = { version = "0.32.0" }
+opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
+opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
+opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
+opentelemetry-semantic-conventions = { version = "0.32.0", features = ["semconv_experimental"] }
+opentelemetry-stdout = { version = "0.32.0" }
+pyroscope = { version = "=2.0.5", features = ["backend-pprof-rs", "backend-jemalloc"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
-suppaftp = { version = "8.0.3", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
-rcgen = "0.14.7"
-russh = "0.60.2"
-russh-sftp = "2.1.2"
+suppaftp = { version = "8.0.4", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+rcgen = "0.14.8"
+russh = { version = "0.61.2", features = ["serde"] }
+russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
@@ -323,18 +330,14 @@ tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profilin
 jemalloc_pprof = { version = "0.8.2", features = ["symbolize", "flamegraph"] }
 # Used to generate CPU performance analysis data and flame diagrams
 # pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
-# Pyroscope uses a patched pprof, until they merge back upstream, replace all references. Otherwise, two pprof libs with symbol collision.
-pprof = { package = "pprof-pyroscope-fork", version = "0.1500.3", features = ["flamegraph", "protobuf-codec"] }
+# Use pprof-pyroscope-fork to match pyroscope 2.0.5's internal pprof dependency (v0.1500.4).
+# Cargo unifies them into a single instance, avoiding duplicate perf_signal_handler symbols.
+# NOTE: Do NOT upgrade pyroscope to >=2.0.6 — it vendors pprof-rs internally, causing symbol collision.
+pprof = { package = "pprof-pyroscope-fork", version = "0.1500.4", features = ["flamegraph", "protobuf-codec"] }
 
 [workspace.metadata.cargo-shear]
 ignored = ["rustfs"]
 
-[patch.crates-io]
-# Pinned to the simon-escapecode/russh fork carrying the upstream fix at
-# https://github.com/Eugeny/russh/pull/702. Drops out when a russh release
-# containing the fix is published.
-russh = { git = "https://github.com/simon-escapecode/russh", rev = "5cac2ed84945f9b80a52b673e058f2032bbe98ec" }
-
 [profile.release]
 opt-level = 3
 

Dockerfile.decommission-local

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM rust:1.93-trixie
+FROM rust:1.95-trixie
 
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \

Dockerfile.source

@@ -31,7 +31,7 @@ ARG TARGETARCH
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.93-trixie AS builder
+FROM rust:1.95-trixie AS builder
 
 # Re-declare args after FROM
 ARG TARGETPLATFORM

README.md

@@ -7,6 +7,7 @@
   <a href="https://github.com/rustfs/rustfs/actions/workflows/docker.yml"><img alt="Build and Push Docker Images" src="https://github.com/rustfs/rustfs/actions/workflows/docker.yml/badge.svg" /></a>
   <img alt="GitHub commit activity" src="https://img.shields.io/github/commit-activity/m/rustfs/rustfs"/>
   <img alt="Github Last Commit" src="https://img.shields.io/github/last-commit/rustfs/rustfs"/>
+  <a href="https://discord.gg/NcKBCEJp6P"><img alt="Discord" src="https://img.shields.io/badge/Discord-Join%20Chat-5865F2?logo=discord&logoColor=white" /></a>
   <a href="https://hellogithub.com/repository/rustfs/rustfs" target="_blank"><img src="https://abroad.hellogithub.com/v1/widgets/recommend.svg?rid=b95bcb72bdc340b68f16fdf6790b7d5b&claim_uid=MsbvjYeLDKAH457&theme=small" alt="Featured｜HelloGitHub" /></a>
 </p>
 
@@ -102,7 +103,7 @@ curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
 
 ### 2. Docker Quick Start (Option 2)
 
-The RustFS container runs as a non-root user `rustfs` (UID `10001`). If you run Docker with `-v` to mount a host directory, please ensure the host directory owner is set to `10001`, otherwise you will encounter permission denied errors.
+The RustFS container runs as a non-root user `rustfs` (UID/GID `10001:10001`). If you bind-mount host directories with Docker or Compose, every mounted path must be writable by that user, otherwise startup may fail with permission denied errors. This applies to data directories, log directories, and TLS certificate directories when `RUSTFS_TLS_PATH` is enabled.
 
 ```bash
 # Create data and logs directories
@@ -115,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -124,12 +125,26 @@ If you use [podman](https://github.com/containers/podman) instead of docker, you
 podman run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 ```
 
+If you enable TLS with a bind-mounted certificate directory, prepare that mount the same way:
+
+```bash
+mkdir -p certs
+chown -R 10001:10001 certs
+```
+
 You can also use Docker Compose. Using the `docker-compose.yml` file in the root directory:
 
 ```bash
 docker compose --profile observability up -d
 ```
 
+Before running Compose with host bind mounts:
+
+- Ensure every mounted host path is writable by `10001:10001`.
+- If you enable TLS, ensure the certificate mount for `/opt/tls` is also readable by `10001:10001`.
+- If matching host ownership is not practical, run the `rustfs` service with `user: "<host-uid>:<host-gid>"` instead.
+- `docker-compose-simple.yml` includes a `volume-permission-helper` service for named volumes. `docker-compose.yml` relies on you to prepare bind-mounted host paths in advance.
+
 Similarly, you can run the command with podman
 
 ```bash
@@ -295,6 +310,7 @@ If you have any questions or need assistance:
 - [Documentation](https://docs.rustfs.com) - The manual you should read
 - [Changelog](https://github.com/rustfs/rustfs/releases) - What we broke and fixed
 - [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Where the community lives
+- [Discord](https://discord.gg/NcKBCEJp6P) - Chat with the RustFS community
 
 ## Contact
 

README_ZH.md

@@ -1,4 +1,4 @@
-[![RustFS](https://github.com/user-attachments/assets/1b5afcd6-a2c3-47ff-8bc3-ce882b0ddca7)](https://rustfs.com.cn)
+[![RustFS](https://repository-images.githubusercontent.com/722597620/0fa936a2-8164-4f53-867f-def4beb64b21)](https://rustfs.com.cn)
 
 <p align="center">RustFS 是一个基于 Rust 构建的高性能分布式对象存储系统。</p>
 
@@ -100,20 +100,27 @@ curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
 
 ### 2\. Docker 快速启动 (选项 2)
 
-RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用 Docker 的 `-v` 参数挂载宿主机目录，请务必确保宿主机目录的所有者已更改为 `10001`，否则会遇到权限拒绝错误。
+RustFS 容器以非 root 用户 `rustfs` (UID/GID `10001:10001`) 运行。如果您通过 Docker 或 Compose 绑定挂载宿主机目录，请务必确保所有挂载路径都对该用户可写，否则启动时可能出现权限拒绝错误。这不仅适用于数据目录和日志目录，也适用于启用 `RUSTFS_TLS_PATH` 时挂载的 TLS 证书目录。
 
 ```bash
- # 创建数据和日志目录
- mkdir -p data logs
+# 创建数据和日志目录
+mkdir -p data logs
 
- # 更改这两个目录的所有者
- chown -R 10001:10001 data logs
+# 更改这两个目录的所有者
+chown -R 10001:10001 data logs
 
- # 使用最新版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
+# 使用最新版本运行
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
- # 使用指定版本运行
- docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.3
+# 使用指定版本运行
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
+```
+
+如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
+
+```bash
+mkdir -p certs
+chown -R 10001:10001 certs
 ```
 
 您也可以使用 Docker Compose。使用根目录下的 `docker-compose.yml` 文件：
@@ -122,6 +129,13 @@ RustFS 容器以非 root 用户 `rustfs` (UID `10001`) 运行。如果您使用
 docker compose --profile observability up -d
 ```
 
+在使用 Compose 且包含宿主机绑定挂载时，请先确认：
+
+- 所有挂载到容器内的宿主机路径都对 `10001:10001` 可写。
+- 如果启用了 TLS，挂载到 `/opt/tls` 的证书目录也需要对 `10001:10001` 可读。
+- 如果不方便调整宿主机目录归属，可以为 `rustfs` 服务显式设置 `user: "<host-uid>:<host-gid>"`。
+- `docker-compose-simple.yml` 为命名 volume 提供了 `volume-permission-helper` 服务；`docker-compose.yml` 使用宿主机绑定挂载，因此需要您在启动前自行准备目录权限。
+
 **注意**: 我们建议您在运行前查看 `docker-compose.yml` 文件。该文件定义了包括 Grafana、Prometheus 和 Jaeger 在内的多个服务，有助于 RustFS 的可观测性监控。如果您还想启动 Redis 或 Nginx 容器，可以指定相应的 profile。
 
 ### 3\. 源码编译 (选项 3) - 进阶用户

build-rustfs.sh

@@ -217,7 +217,7 @@ setup_rust_environment() {
     # Set up environment variables for musl targets
     if [[ "$PLATFORM" == *"musl"* ]]; then
         print_message $YELLOW "Setting up environment for musl target..."
-        export RUSTFLAGS="'--cfg tokio_unstable -C target-feature=-crt-static'"
+        export RUSTFLAGS="--cfg tokio_unstable -C target-feature=-crt-static"
 
         # For cargo-zigbuild, set up additional environment variables
         if command -v cargo-zigbuild &> /dev/null; then

crates/AGENTS.md

@@ -14,6 +14,41 @@ Applies to all paths under `crates/`.
 - Keep integration tests under each crate's `tests/` directory.
 - Add regression tests for bug fixes and behavior changes.
 
+## Error Type Design
+
+- Public API functions must return a typed error enum (preferably `thiserror`-derived), never `Result<_, String>`.
+- Do not use `Box<dyn Error>` or `Box<dyn Error + Send + Sync>` in public trait methods or struct methods. Define a concrete error type with specific variants.
+- When implementing `std::error::Error`, always override `fn source()` if you store an inner error. Breaking the error chain makes debugging impossible.
+- Internal helpers that return `Result<_, String>` and are immediately wrapped via `.map_err(Error::other)` should return the actual error type directly.
+
+## Concurrency
+
+- Document lock acquisition order when a module uses multiple locks. Never acquire the same set of locks in different orders across code paths.
+- Never hold a `tokio::sync::RwLock`/`Mutex` write guard across `.await` points unless the critical section is unavoidably async and the hold time is bounded.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters (peak values, adaptive heuristics).
+- When resetting multi-field atomic statistics, use a version/sequence counter or accept that concurrent readers may see partial snapshots; document the tradeoff.
+- `std::sync::Mutex` is acceptable in async context only when held for a brief, non-`await`-containing critical section. If in doubt, use `tokio::sync::Mutex`.
+
+## Recursion Safety
+
+- Recursive tree/graph traversals must have a depth limit (e.g., `max_depth` counter) or use an iterative approach with an explicit `Vec` stack.
+- This applies to cache trees, directory walks, and any user-influenced hierarchy.
+- A corrupted or malicious input must not be able to overflow the thread stack.
+
+## Type Casting
+
+- Never use `as` for numeric conversions that may truncate or overflow. Use `try_into()` with explicit error handling, or clamp with `value.max(0) as usize` when the domain is bounded.
+- `f64 as usize` saturates but is fragile; clamp to `[0, usize::MAX as f64]` first.
+- Treat every `as` cast in a PR review as a potential bug; require justification.
+
+## Testing
+
+- Keep unit tests close to the module they test.
+- Keep integration tests under each crate's `tests/` directory.
+- Add regression tests for bug fixes and behavior changes.
+- Every test function must contain at least one `assert!`/`assert_eq!`/`assert_matches!`. A test that only calls code without asserting is not a test.
+- In tests, prefer `.expect("context: what was being tested")` over bare `.unwrap()`. A test failure should tell you which operation failed and with what input.
+
 ## Async and Performance
 
 - Keep async paths non-blocking.

crates/appauth/README.md

@@ -1,37 +0,0 @@
-[![RustFS](https://rustfs.com/images/rustfs-github.png)](https://rustfs.com)
-
-# RustFS AppAuth - Application Authentication
-
-<p align="center">
-  <strong>Application-level authentication and authorization module for RustFS distributed object storage</strong>
-</p>
-
-<p align="center">
-  <a href="https://github.com/rustfs/rustfs/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/rustfs/rustfs/actions/workflows/ci.yml/badge.svg" /></a>
-  <a href="https://docs.rustfs.com/">📖 Documentation</a>
-  · <a href="https://github.com/rustfs/rustfs/issues">🐛 Bug Reports</a>
-  · <a href="https://github.com/rustfs/rustfs/discussions">💬 Discussions</a>
-</p>
-
----
-
-## 📖 Overview
-
-**RustFS AppAuth** provides application-level authentication and authorization capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
-
-## ✨ Features
-
-- JWT-based authentication with secure token management
-- RBAC (Role-Based Access Control) for fine-grained permissions
-- Multi-tenant application isolation and management
-- OAuth 2.0 and OpenID Connect integration
-- API key management and rotation
-- Session management with configurable expiration
-
-## 📚 Documentation
-
-For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
-
-## 📄 License
-
-This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.

crates/audit/Cargo.toml

@@ -29,7 +29,7 @@ categories = ["web-programming", "development-tools", "asynchronous", "api-bindi
 rustfs-targets = { workspace = true }
 rustfs-config = { workspace = true, features = ["audit", "constants"] }
 rustfs-ecstore = { workspace = true }
-rustfs-s3-common = { workspace = true }
+rustfs-s3-types = { workspace = true }
 chrono = { workspace = true }
 const-str = { workspace = true }
 futures = { workspace = true }

crates/audit/src/entity.rs

@@ -14,7 +14,7 @@
 
 use chrono::{DateTime, Utc};
 use hashbrown::HashMap;
-use rustfs_s3_common::EventName;
+use rustfs_s3_types::EventName;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
 

crates/checksums/Cargo.toml

@@ -23,7 +23,7 @@ homepage.workspace = true
 description = "Checksum calculation and verification callbacks for HTTP request and response bodies sent by service clients generated by RustFS, ensuring data integrity and authenticity."
 keywords = ["checksum-calculation", "verification", "integrity", "authenticity", "rustfs"]
 categories = ["web-programming", "development-tools", "network-programming"]
-documentation = "https://docs.rs/rustfs-signer/latest/rustfs_checksum/"
+documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
 
 [dependencies]
 bytes = { workspace = true }

crates/common/Cargo.toml

@@ -33,12 +33,8 @@ tonic = { workspace = true }
 uuid = { workspace = true }
 chrono = { workspace = true }
 metrics = { workspace = true }
-rustfs-madmin = { workspace = true }
-rustfs-filemeta = { workspace = true }
 serde = { workspace = true }
-path-clean = { workspace = true }
 rmp-serde = { workspace = true }
-async-trait = { workspace = true }
 s3s = { workspace = true }
 tracing = { workspace = true }
 

crates/common/src/globals.rs

@@ -17,6 +17,7 @@
 use chrono::{DateTime, Utc};
 use std::collections::HashMap;
 use std::sync::LazyLock;
+use std::sync::atomic::{AtomicU64, Ordering};
 use tokio::sync::RwLock;
 use tonic::transport::Channel;
 
@@ -27,6 +28,7 @@ pub static GLOBAL_RUSTFS_ADDR: LazyLock<RwLock<String>> = LazyLock::new(|| RwLoc
 pub static GLOBAL_CONN_MAP: LazyLock<RwLock<HashMap<String, Channel>>> = LazyLock::new(|| RwLock::new(HashMap::new()));
 pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|| RwLock::new(None));
 pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
+pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
 /// Global initialization time of the RustFS node.
 pub static GLOBAL_INIT_TIME: LazyLock<RwLock<Option<DateTime<Utc>>>> = LazyLock::new(|| RwLock::new(None));
 
@@ -88,6 +90,16 @@ pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
     *GLOBAL_MTLS_IDENTITY.write().await = identity;
 }
 
+/// Set the global outbound TLS generation.
+pub fn set_global_outbound_tls_generation(generation: u64) {
+    GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
+}
+
+/// Get the global outbound TLS generation.
+pub fn get_global_outbound_tls_generation() -> u64 {
+    GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
+}
+
 /// Evict a stale/dead connection from the global connection cache.
 /// This is critical for cluster recovery when a node dies unexpectedly (e.g., power-off).
 /// By removing the cached connection, subsequent requests will establish a fresh connection.

crates/common/src/heal_channel.rs

@@ -110,6 +110,25 @@ pub enum HealScanMode {
     Deep = 2,
 }
 
+impl HealScanMode {
+    pub const fn as_str(self) -> &'static str {
+        match self {
+            Self::Unknown => "unknown",
+            Self::Normal => "normal",
+            Self::Deep => "deep",
+        }
+    }
+
+    pub const fn from_u8(value: u8) -> Option<Self> {
+        match value {
+            0 => Some(Self::Unknown),
+            1 => Some(Self::Normal),
+            2 => Some(Self::Deep),
+            _ => None,
+        }
+    }
+}
+
 impl Serialize for HealScanMode {
     fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
     where
@@ -137,12 +156,7 @@ impl<'de> Deserialize<'de> for HealScanMode {
             where
                 E: serde::de::Error,
             {
-                match value {
-                    0 => Ok(HealScanMode::Unknown),
-                    1 => Ok(HealScanMode::Normal),
-                    2 => Ok(HealScanMode::Deep),
-                    _ => Err(E::custom(format!("invalid HealScanMode value: {value}"))),
-                }
+                HealScanMode::from_u8(value).ok_or_else(|| E::custom(format!("invalid HealScanMode value: {value}")))
             }
 
             fn visit_u64<E>(self, value: u64) -> Result<Self::Value, E>
@@ -260,9 +274,17 @@ pub enum HealChannelCommand {
         response_tx: oneshot::Sender<Result<HealAdmissionResult, String>>,
     },
     /// Query heal task status
-    Query { heal_path: String, client_token: String },
+    Query {
+        heal_path: String,
+        client_token: String,
+        response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
+    },
     /// Cancel heal task
-    Cancel { heal_path: String },
+    Cancel {
+        heal_path: String,
+        client_token: String,
+        response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
+    },
 }
 
 /// Heal request from admin to ahm
@@ -379,7 +401,9 @@ fn heal_response_sender() -> &'static HealResponseSender {
 
 /// Publish a heal response to subscribers.
 pub fn publish_heal_response(response: HealChannelResponse) -> Result<(), broadcast::error::SendError<HealChannelResponse>> {
-    heal_response_sender().send(response).map(|_| ())
+    let sender = heal_response_sender();
+    let _ = sender.send(response);
+    Ok(())
 }
 
 /// Subscribe to heal responses.
@@ -405,14 +429,36 @@ pub async fn send_heal_request(request: HealChannelRequest) -> Result<(), String
     }
 }
 
+async fn receive_heal_channel_response(
+    response_rx: oneshot::Receiver<Result<HealChannelResponse, String>>,
+) -> Result<HealChannelResponse, String> {
+    response_rx
+        .await
+        .map_err(|e| format!("Failed to receive heal channel response: {e}"))?
+}
+
 /// Send heal query request
-pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<(), String> {
-    send_heal_command(HealChannelCommand::Query { heal_path, client_token }).await
+pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    send_heal_command(HealChannelCommand::Query {
+        heal_path,
+        client_token,
+        response_tx,
+    })
+    .await?;
+    receive_heal_channel_response(response_rx).await
 }
 
 /// Send heal cancel request
-pub async fn cancel_heal_task(heal_path: String) -> Result<(), String> {
-    send_heal_command(HealChannelCommand::Cancel { heal_path }).await
+pub async fn cancel_heal_task(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
+    let (response_tx, response_rx) = oneshot::channel();
+    send_heal_command(HealChannelCommand::Cancel {
+        heal_path,
+        client_token,
+        response_tx,
+    })
+    .await?;
+    receive_heal_channel_response(response_rx).await
 }
 
 /// Create a new heal request
@@ -626,5 +672,10 @@ mod tests {
         let received = receiver.recv().await.expect("should receive heal response");
         assert_eq!(received.request_id, response.request_id);
         assert!(received.success);
+
+        drop(receiver);
+        let response = create_heal_response("req-no-subscriber".to_string(), true, None, None);
+
+        publish_heal_response(response).expect("publish without subscribers should be ignored");
     }
 }

crates/common/src/internode_metrics.rs

@@ -1,166 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use metrics::{counter, gauge};
-use std::sync::{
-    Arc, LazyLock,
-    atomic::{AtomicU64, Ordering},
-};
-use std::time::{Duration, SystemTime, UNIX_EPOCH};
-
-#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
-pub struct InternodeMetricsSnapshot {
-    pub sent_bytes_total: u64,
-    pub recv_bytes_total: u64,
-    pub outgoing_requests_total: u64,
-    pub incoming_requests_total: u64,
-    pub errors_total: u64,
-    pub dial_errors_total: u64,
-    pub dial_avg_time_nanos: u64,
-    pub last_dial_unix_millis: u64,
-}
-
-#[derive(Debug, Default)]
-pub struct InternodeMetrics {
-    sent_bytes_total: AtomicU64,
-    recv_bytes_total: AtomicU64,
-    outgoing_requests_total: AtomicU64,
-    incoming_requests_total: AtomicU64,
-    errors_total: AtomicU64,
-    dial_errors_total: AtomicU64,
-    dial_total_time_nanos: AtomicU64,
-    dial_samples_total: AtomicU64,
-    last_dial_unix_millis: AtomicU64,
-}
-
-impl InternodeMetrics {
-    pub fn record_sent_bytes(&self, bytes: usize) {
-        let bytes = bytes as u64;
-        if bytes == 0 {
-            return;
-        }
-        self.sent_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_sent_bytes_total").increment(bytes);
-    }
-
-    pub fn record_recv_bytes(&self, bytes: usize) {
-        let bytes = bytes as u64;
-        if bytes == 0 {
-            return;
-        }
-        self.recv_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_recv_bytes_total").increment(bytes);
-    }
-
-    pub fn record_outgoing_request(&self) {
-        self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_requests_outgoing_total").increment(1);
-    }
-
-    pub fn record_incoming_request(&self) {
-        self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_requests_incoming_total").increment(1);
-    }
-
-    pub fn record_error(&self) {
-        self.errors_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_errors_total").increment(1);
-    }
-
-    pub fn record_dial_result(&self, duration: Duration, success: bool) {
-        let elapsed_nanos = duration.as_nanos().min(u128::from(u64::MAX)) as u64;
-        self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
-        let samples = self.dial_samples_total.fetch_add(1, Ordering::Relaxed) + 1;
-        let total = self.dial_total_time_nanos.load(Ordering::Relaxed);
-        gauge!("rustfs_system_network_internode_dial_avg_time_nanos").set(total as f64 / samples as f64);
-
-        if !success {
-            self.dial_errors_total.fetch_add(1, Ordering::Relaxed);
-            counter!("rustfs_system_network_internode_dial_errors_total").increment(1);
-        }
-
-        let now_ms = SystemTime::now()
-            .duration_since(UNIX_EPOCH)
-            .unwrap_or_default()
-            .as_millis()
-            .min(u128::from(u64::MAX)) as u64;
-        self.last_dial_unix_millis.store(now_ms, Ordering::Relaxed);
-    }
-
-    pub fn snapshot(&self) -> InternodeMetricsSnapshot {
-        let dial_samples_total = self.dial_samples_total.load(Ordering::Relaxed);
-        let dial_total_time_nanos = self.dial_total_time_nanos.load(Ordering::Relaxed);
-        let dial_avg_time_nanos = dial_total_time_nanos.checked_div(dial_samples_total).unwrap_or(0);
-
-        InternodeMetricsSnapshot {
-            sent_bytes_total: self.sent_bytes_total.load(Ordering::Relaxed),
-            recv_bytes_total: self.recv_bytes_total.load(Ordering::Relaxed),
-            outgoing_requests_total: self.outgoing_requests_total.load(Ordering::Relaxed),
-            incoming_requests_total: self.incoming_requests_total.load(Ordering::Relaxed),
-            errors_total: self.errors_total.load(Ordering::Relaxed),
-            dial_errors_total: self.dial_errors_total.load(Ordering::Relaxed),
-            dial_avg_time_nanos,
-            last_dial_unix_millis: self.last_dial_unix_millis.load(Ordering::Relaxed),
-        }
-    }
-
-    #[doc(hidden)]
-    pub fn reset_for_test(&self) {
-        self.sent_bytes_total.store(0, Ordering::Relaxed);
-        self.recv_bytes_total.store(0, Ordering::Relaxed);
-        self.outgoing_requests_total.store(0, Ordering::Relaxed);
-        self.incoming_requests_total.store(0, Ordering::Relaxed);
-        self.errors_total.store(0, Ordering::Relaxed);
-        self.dial_errors_total.store(0, Ordering::Relaxed);
-        self.dial_total_time_nanos.store(0, Ordering::Relaxed);
-        self.dial_samples_total.store(0, Ordering::Relaxed);
-        self.last_dial_unix_millis.store(0, Ordering::Relaxed);
-    }
-}
-
-pub fn global_internode_metrics() -> &'static Arc<InternodeMetrics> {
-    static GLOBAL_INTERNODE_METRICS: LazyLock<Arc<InternodeMetrics>> = LazyLock::new(|| Arc::new(InternodeMetrics::default()));
-    &GLOBAL_INTERNODE_METRICS
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn snapshot_reports_recorded_values() {
-        let metrics = global_internode_metrics();
-        metrics.reset_for_test();
-
-        metrics.record_sent_bytes(64);
-        metrics.record_recv_bytes(32);
-        metrics.record_outgoing_request();
-        metrics.record_incoming_request();
-        metrics.record_error();
-        metrics.record_dial_result(Duration::from_millis(9), true);
-        metrics.record_dial_result(Duration::from_millis(3), false);
-
-        let snapshot = metrics.snapshot();
-        assert_eq!(snapshot.sent_bytes_total, 64);
-        assert_eq!(snapshot.recv_bytes_total, 32);
-        assert_eq!(snapshot.outgoing_requests_total, 1);
-        assert_eq!(snapshot.incoming_requests_total, 1);
-        assert_eq!(snapshot.errors_total, 1);
-        assert_eq!(snapshot.dial_errors_total, 1);
-        assert_eq!(snapshot.dial_avg_time_nanos, 6_000_000);
-        assert!(snapshot.last_dial_unix_millis > 0);
-
-        metrics.reset_for_test();
-    }
-}

crates/common/src/lib.rs

@@ -13,12 +13,9 @@
 // limitations under the License.
 
 pub mod bucket_stats;
-pub mod capacity_scope;
 // pub mod error;
-pub mod data_usage;
 pub mod globals;
 pub mod heal_channel;
-pub mod internode_metrics;
 pub mod last_minute;
 pub mod metrics;
 mod readiness;

crates/concurrency/src/backpressure.rs

@@ -14,15 +14,17 @@
 
 //! Backpressure management
 
-use rustfs_io_core::{BackpressureMonitor as CoreBackpressureMonitor, BackpressureState};
+use rustfs_io_core::{
+    BackpressureConfig as CoreBackpressureConfig, BackpressureMonitor as CoreBackpressureMonitor, BackpressureState,
+};
 use rustfs_io_metrics::backpressure_metrics;
 use std::sync::Arc;
 use std::time::Instant;
 use tokio::io::{DuplexStream, duplex};
 
-/// Backpressure configuration
-#[derive(Debug, Clone)]
-pub struct BackpressureConfig {
+/// Facade policy for duplex-pipe watermark backpressure.
+#[derive(Debug, Clone, Copy)]
+pub struct PipeBackpressurePolicy {
     /// Buffer size in bytes
     pub buffer_size: usize,
     /// High watermark percentage
@@ -31,7 +33,7 @@ pub struct BackpressureConfig {
     pub low_watermark: u32,
 }
 
-impl Default for BackpressureConfig {
+impl Default for PipeBackpressurePolicy {
     fn default() -> Self {
         Self {
             buffer_size: 4 * 1024 * 1024, // 4MB
@@ -41,7 +43,7 @@ impl Default for BackpressureConfig {
     }
 }
 
-impl BackpressureConfig {
+impl PipeBackpressurePolicy {
     /// Calculate high watermark threshold in bytes
     pub fn high_watermark_bytes(&self) -> usize {
         (self.buffer_size as u64 * self.high_watermark as u64 / 100) as usize
@@ -51,42 +53,59 @@ impl BackpressureConfig {
     pub fn low_watermark_bytes(&self) -> usize {
         (self.buffer_size as u64 * self.low_watermark as u64 / 100) as usize
     }
+
+    /// Convert the facade policy into the reusable io-core admission-pressure config.
+    ///
+    /// The concurrency layer still owns duplex buffer sizing, but the shared
+    /// overload/admission primitive lives in `io-core`.
+    pub fn to_core_config(&self) -> CoreBackpressureConfig {
+        CoreBackpressureConfig {
+            max_concurrent: 32,
+            high_water_mark: self.high_watermark as f64 / 100.0,
+            low_water_mark: self.low_watermark as f64 / 100.0,
+            cooldown: std::time::Duration::from_millis(100),
+            enabled: true,
+        }
+    }
 }
 
 /// Backpressure manager
 pub struct BackpressureManager {
-    config: BackpressureConfig,
+    config: PipeBackpressurePolicy,
+    core_config: CoreBackpressureConfig,
     monitor: Arc<CoreBackpressureMonitor>,
 }
 
 impl BackpressureManager {
     /// Create a new backpressure manager
     pub fn new(buffer_size: usize, high_watermark: u32, low_watermark: u32) -> Self {
-        let config = BackpressureConfig {
+        Self::from_policy(PipeBackpressurePolicy {
             buffer_size,
             high_watermark,
             low_watermark,
-        };
-
-        let core_config = rustfs_io_core::BackpressureConfig {
-            max_concurrent: 32,
-            high_water_mark: high_watermark as f64 / 100.0,
-            low_water_mark: low_watermark as f64 / 100.0,
-            cooldown: std::time::Duration::from_millis(100),
-            enabled: true,
-        };
+        })
+    }
 
+    /// Create a new backpressure manager from the facade policy type.
+    pub fn from_policy(config: PipeBackpressurePolicy) -> Self {
+        let core_config = config.to_core_config();
         Self {
             config,
+            core_config: core_config.clone(),
             monitor: Arc::new(CoreBackpressureMonitor::new(core_config)),
         }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &BackpressureConfig {
+    pub fn config(&self) -> &PipeBackpressurePolicy {
         &self.config
     }
 
+    /// Get the derived io-core admission-pressure configuration.
+    pub fn core_config(&self) -> &CoreBackpressureConfig {
+        &self.core_config
+    }
+
     /// Get the monitor
     pub fn monitor(&self) -> Arc<CoreBackpressureMonitor> {
         self.monitor.clone()
@@ -94,7 +113,7 @@ impl BackpressureManager {
 
     /// Create a backpressure pipe
     pub fn create_pipe(&self) -> BackpressurePipe {
-        BackpressurePipe::new(self.config.clone(), self.monitor.clone())
+        BackpressurePipe::new(self.config, self.monitor.clone())
     }
 
     /// Get current state
@@ -112,13 +131,24 @@ impl BackpressureManager {
 pub struct BackpressurePipe {
     reader: DuplexStream,
     writer: DuplexStream,
-    config: BackpressureConfig,
+    config: PipeBackpressurePolicy,
     monitor: Arc<CoreBackpressureMonitor>,
     created_at: Instant,
 }
 
+/// Shared pipe metadata snapshot for facade-level backpressure pipes.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct BackpressurePipeMeta {
+    /// Configured duplex buffer capacity in bytes.
+    pub buffer_capacity: usize,
+    /// Current backpressure state reported by the shared core monitor.
+    pub state: BackpressureState,
+    /// Age of the pipe since creation.
+    pub age: std::time::Duration,
+}
+
 impl BackpressurePipe {
-    fn new(config: BackpressureConfig, monitor: Arc<CoreBackpressureMonitor>) -> Self {
+    fn new(config: PipeBackpressurePolicy, monitor: Arc<CoreBackpressureMonitor>) -> Self {
         let (reader, writer) = duplex(config.buffer_size);
 
         Self {
@@ -146,7 +176,7 @@ impl BackpressurePipe {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &BackpressureConfig {
+    pub fn config(&self) -> &PipeBackpressurePolicy {
         &self.config
     }
 
@@ -160,6 +190,15 @@ impl BackpressurePipe {
         self.created_at.elapsed()
     }
 
+    /// Get a compact metadata snapshot for the pipe.
+    pub fn meta(&self) -> BackpressurePipeMeta {
+        BackpressurePipeMeta {
+            buffer_capacity: self.config.buffer_size,
+            state: self.state(),
+            age: self.age(),
+        }
+    }
+
     /// Check if should apply backpressure
     pub fn should_apply_backpressure(&self) -> bool {
         let should = self.monitor.should_apply_backpressure();
@@ -170,45 +209,26 @@ impl BackpressurePipe {
     }
 }
 
-/// Backpressure event
-#[allow(dead_code)]
-#[derive(Debug, Clone)]
-pub struct BackpressureEvent {
-    /// Event timestamp
-    pub timestamp: Instant,
-    /// Event type
-    pub event_type: BackpressureEventType,
-    /// Buffer usage
-    pub buffer_usage: usize,
-    /// Buffer capacity
-    pub buffer_capacity: usize,
-}
-
-/// Backpressure event type
-#[allow(dead_code)]
-#[derive(Debug, Clone, Copy)]
-pub enum BackpressureEventType {
-    /// High watermark reached
-    HighWatermarkReached,
-    /// High watermark exited
-    HighWatermarkExited,
-    /// Backpressure applied
-    BackpressureApplied,
-    /// Backpressure released
-    BackpressureReleased,
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
 
     #[test]
     fn test_backpressure_config() {
-        let config = BackpressureConfig::default();
+        let config = PipeBackpressurePolicy::default();
         assert_eq!(config.buffer_size, 4 * 1024 * 1024);
         assert!(config.high_watermark > config.low_watermark);
     }
 
+    #[test]
+    fn test_backpressure_policy_to_core_config() {
+        let policy = PipeBackpressurePolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.high_water_mark, policy.high_watermark as f64 / 100.0);
+        assert_eq!(core.low_water_mark, policy.low_watermark as f64 / 100.0);
+        assert!(core.enabled);
+    }
+
     #[test]
     fn test_backpressure_manager() {
         let manager = BackpressureManager::new(1024, 80, 50);
@@ -220,5 +240,6 @@ mod tests {
         let manager = BackpressureManager::new(1024, 80, 50);
         let pipe = manager.create_pipe();
         assert_eq!(pipe.state(), BackpressureState::Normal);
+        assert_eq!(pipe.meta().buffer_capacity, 1024);
     }
 }

crates/concurrency/src/config.rs

@@ -14,6 +14,10 @@
 
 //! Configuration for concurrency management
 
+use crate::{
+    backpressure::PipeBackpressurePolicy, deadlock::DeadlockMonitorPolicy, scheduler::SchedulerPolicy,
+    timeout::TimeoutManagerPolicy,
+};
 use std::time::Duration;
 
 /// Feature flags for concurrency modules
@@ -72,84 +76,39 @@ impl ConcurrencyFeatures {
     }
 }
 
+/// Facade policy for lock manager behavior.
+#[derive(Debug, Clone, Copy)]
+pub struct LockManagerPolicy {
+    /// Enable lock optimization.
+    pub enabled: bool,
+    /// Lock acquisition timeout.
+    pub acquire_timeout: Duration,
+}
+
+impl Default for LockManagerPolicy {
+    fn default() -> Self {
+        Self {
+            enabled: true,
+            acquire_timeout: Duration::from_secs(5),
+        }
+    }
+}
+
 /// Main configuration for concurrency management
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Default)]
 pub struct ConcurrencyConfig {
     /// Feature flags
     pub features: ConcurrencyFeatures,
-
-    // Timeout configuration
-    /// Default timeout duration
-    pub default_timeout: Duration,
-    /// Maximum timeout duration
-    pub max_timeout: Duration,
-    /// Enable dynamic timeout
-    pub enable_dynamic_timeout: bool,
-
-    // Lock configuration
-    /// Enable lock optimization
-    pub enable_lock_optimization: bool,
-    /// Lock acquisition timeout
-    pub lock_acquire_timeout: Duration,
-
-    // Deadlock configuration
-    /// Enable deadlock detection
-    pub enable_deadlock_detection: bool,
-    /// Deadlock check interval
-    pub deadlock_check_interval: Duration,
-    /// Hang threshold
-    pub hang_threshold: Duration,
-
-    // Backpressure configuration
-    /// Buffer size for backpressure
-    pub backpressure_buffer_size: usize,
-    /// High watermark percentage
-    pub high_watermark: u32,
-    /// Low watermark percentage
-    pub low_watermark: u32,
-
-    // Scheduler configuration
-    /// Base buffer size for I/O
-    pub io_buffer_size: usize,
-    /// Maximum buffer size
-    pub max_buffer_size: usize,
-    /// High priority size threshold
-    pub high_priority_threshold: usize,
-    /// Low priority size threshold
-    pub low_priority_threshold: usize,
-}
-
-impl Default for ConcurrencyConfig {
-    fn default() -> Self {
-        Self {
-            features: ConcurrencyFeatures::default(),
-
-            // Timeout defaults
-            default_timeout: Duration::from_secs(30),
-            max_timeout: Duration::from_secs(300),
-            enable_dynamic_timeout: true,
-
-            // Lock defaults
-            enable_lock_optimization: true,
-            lock_acquire_timeout: Duration::from_secs(5),
-
-            // Deadlock defaults
-            enable_deadlock_detection: false,
-            deadlock_check_interval: Duration::from_secs(10),
-            hang_threshold: Duration::from_secs(60),
-
-            // Backpressure defaults
-            backpressure_buffer_size: 4 * 1024 * 1024, // 4MB
-            high_watermark: 80,
-            low_watermark: 50,
-
-            // Scheduler defaults
-            io_buffer_size: 64 * 1024,                // 64KB
-            max_buffer_size: 4 * 1024 * 1024,         // 4MB
-            high_priority_threshold: 1024 * 1024,     // 1MB
-            low_priority_threshold: 10 * 1024 * 1024, // 10MB
-        }
-    }
+    /// Timeout facade policy.
+    pub timeout_policy: TimeoutManagerPolicy,
+    /// Lock facade policy.
+    pub lock_policy: LockManagerPolicy,
+    /// Deadlock facade policy.
+    pub deadlock_policy: DeadlockMonitorPolicy,
+    /// Backpressure facade policy.
+    pub backpressure_policy: PipeBackpressurePolicy,
+    /// Scheduler facade policy.
+    pub scheduler_policy: SchedulerPolicy,
 }
 
 impl ConcurrencyConfig {
@@ -161,25 +120,25 @@ impl ConcurrencyConfig {
         if let Ok(val) = std::env::var("RUSTFS_TIMEOUT_DEFAULT")
             && let Ok(secs) = val.parse::<u64>()
         {
-            config.default_timeout = Duration::from_secs(secs);
+            config.timeout_policy.default_timeout = Duration::from_secs(secs);
         }
 
         if let Ok(val) = std::env::var("RUSTFS_TIMEOUT_MAX")
             && let Ok(secs) = val.parse::<u64>()
         {
-            config.max_timeout = Duration::from_secs(secs);
+            config.timeout_policy.max_timeout = Duration::from_secs(secs);
         }
 
         if let Ok(val) = std::env::var("RUSTFS_BACKPRESSURE_BUFFER_SIZE")
             && let Ok(size) = val.parse::<usize>()
         {
-            config.backpressure_buffer_size = size;
+            config.backpressure_policy.buffer_size = size;
         }
 
         if let Ok(val) = std::env::var("RUSTFS_IO_BUFFER_SIZE")
             && let Ok(size) = val.parse::<usize>()
         {
-            config.io_buffer_size = size;
+            config.scheduler_policy.base_buffer_size = size;
         }
 
         config
@@ -187,18 +146,25 @@ impl ConcurrencyConfig {
 
     /// Validate configuration
     pub fn validate(&self) -> Result<(), ConfigError> {
-        if self.default_timeout > self.max_timeout {
+        if self.timeout_policy.default_timeout > self.timeout_policy.max_timeout {
             return Err(ConfigError::InvalidTimeout("default_timeout cannot exceed max_timeout".to_string()));
         }
+        if self.timeout_policy.min_timeout > self.timeout_policy.max_timeout {
+            return Err(ConfigError::InvalidTimeout("min_timeout cannot exceed max_timeout".to_string()));
+        }
 
-        if self.high_watermark <= self.low_watermark || self.high_watermark > 100 {
+        if self.backpressure_policy.high_watermark <= self.backpressure_policy.low_watermark
+            || self.backpressure_policy.high_watermark > 100
+        {
             return Err(ConfigError::InvalidBackpressure(
                 "high_watermark must be > low_watermark and <= 100".to_string(),
             ));
         }
 
-        if self.io_buffer_size > self.max_buffer_size {
-            return Err(ConfigError::InvalidScheduler("io_buffer_size cannot exceed max_buffer_size".to_string()));
+        if self.scheduler_policy.base_buffer_size > self.scheduler_policy.max_buffer_size {
+            return Err(ConfigError::InvalidScheduler(
+                "base_buffer_size cannot exceed max_buffer_size".to_string(),
+            ));
         }
 
         Ok(())
@@ -235,8 +201,12 @@ mod tests {
     #[test]
     fn test_invalid_timeout() {
         let config = ConcurrencyConfig {
-            default_timeout: Duration::from_secs(100),
-            max_timeout: Duration::from_secs(50),
+            timeout_policy: TimeoutManagerPolicy {
+                default_timeout: Duration::from_secs(100),
+                max_timeout: Duration::from_secs(50),
+                enable_dynamic: true,
+                ..Default::default()
+            },
             ..Default::default()
         };
         assert!(
@@ -245,6 +215,22 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_invalid_min_timeout() {
+        let config = ConcurrencyConfig {
+            timeout_policy: TimeoutManagerPolicy {
+                min_timeout: Duration::from_secs(100),
+                max_timeout: Duration::from_secs(50),
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+        assert!(
+            config.validate().is_err(),
+            "validate() should return an error when min_timeout > max_timeout"
+        );
+    }
+
     #[test]
     fn test_features() {
         let features = ConcurrencyFeatures::all();

crates/concurrency/src/deadlock.rs

@@ -14,15 +14,15 @@
 
 //! Deadlock detection management
 
-use rustfs_io_core::{DeadlockDetector as CoreDeadlockDetector, LockType};
+use rustfs_io_core::{DeadlockDetector as CoreDeadlockDetector, DeadlockDetectorConfig as CoreDeadlockConfig, LockType};
 use rustfs_io_metrics::deadlock_metrics;
 use std::collections::HashMap;
 use std::sync::Arc;
 use std::time::{Duration, Instant};
 
-/// Deadlock configuration
-#[derive(Debug, Clone)]
-pub struct DeadlockConfig {
+/// Facade policy for the concurrency-layer deadlock monitor.
+#[derive(Debug, Clone, Copy)]
+pub struct DeadlockMonitorPolicy {
     /// Enable deadlock detection
     pub enabled: bool,
     /// Check interval
@@ -31,7 +31,7 @@ pub struct DeadlockConfig {
     pub hang_threshold: Duration,
 }
 
-impl Default for DeadlockConfig {
+impl Default for DeadlockMonitorPolicy {
     fn default() -> Self {
         Self {
             enabled: false,
@@ -41,9 +41,20 @@ impl Default for DeadlockConfig {
     }
 }
 
+impl DeadlockMonitorPolicy {
+    /// Convert the facade policy into the reusable io-core deadlock config.
+    pub fn to_core_config(&self) -> CoreDeadlockConfig {
+        CoreDeadlockConfig {
+            enabled: self.enabled,
+            detection_interval: self.check_interval,
+            max_hold_time: self.hang_threshold,
+        }
+    }
+}
+
 /// Deadlock manager
 pub struct DeadlockManager {
-    config: DeadlockConfig,
+    config: DeadlockMonitorPolicy,
     detector: Arc<CoreDeadlockDetector>,
     running: Arc<tokio::sync::Mutex<bool>>,
 }
@@ -51,18 +62,16 @@ pub struct DeadlockManager {
 impl DeadlockManager {
     /// Create a new deadlock manager
     pub fn new(enabled: bool, check_interval: Duration, hang_threshold: Duration) -> Self {
-        let config = DeadlockConfig {
+        Self::from_policy(DeadlockMonitorPolicy {
             enabled,
             check_interval,
             hang_threshold,
-        };
-
-        let core_config = rustfs_io_core::DeadlockDetectorConfig {
-            enabled,
-            detection_interval: check_interval,
-            max_hold_time: hang_threshold,
-        };
+        })
+    }
 
+    /// Create a new deadlock manager from the facade policy type.
+    pub fn from_policy(config: DeadlockMonitorPolicy) -> Self {
+        let core_config = config.to_core_config();
         Self {
             config,
             detector: Arc::new(CoreDeadlockDetector::new(core_config)),
@@ -71,7 +80,7 @@ impl DeadlockManager {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &DeadlockConfig {
+    pub fn config(&self) -> &DeadlockMonitorPolicy {
         &self.config
     }
 
@@ -129,7 +138,11 @@ impl DeadlockManager {
     }
 }
 
-/// Request tracker for tracking resources
+/// Lightweight compatibility wrapper for request-scoped deadlock bookkeeping.
+///
+/// This type intentionally stays minimal in the concurrency layer. Rich
+/// request-level lock/resource diagnostics belong to
+/// `rustfs::storage::deadlock_detector::RequestResourceTracker`.
 pub struct RequestTracker {
     request_id: String,
     description: String,
@@ -174,6 +187,11 @@ impl RequestTracker {
         deadlock_metrics::record_lock_acquisition("read");
     }
 
+    /// Return a read-only view of tracked resource names.
+    pub fn resources(&self) -> &HashMap<String, Vec<String>> {
+        &self.resources
+    }
+
     /// Record a lock release
     pub fn record_lock_release(&mut self, lock_id: u64) {
         self.detector.record_release(lock_id);
@@ -196,12 +214,24 @@ mod tests {
         assert!(!manager.config().enabled);
     }
 
+    #[test]
+    fn test_deadlock_policy_to_core_config() {
+        let policy = DeadlockMonitorPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.enabled, policy.enabled);
+        assert_eq!(core.detection_interval, policy.check_interval);
+        assert_eq!(core.max_hold_time, policy.hang_threshold);
+    }
+
     #[tokio::test]
     async fn test_request_tracker() {
         let manager = DeadlockManager::new(true, Duration::from_secs(10), Duration::from_secs(60));
-        let tracker = manager.track_request("req-1".to_string(), "test request".to_string());
+        let mut tracker = manager.track_request("req-1".to_string(), "test request".to_string());
+        let lock_id = manager.register_lock(LockType::Mutex);
+        tracker.record_lock_acquire(lock_id, "bucket/key".to_string());
 
         assert_eq!(tracker.request_id(), "req-1");
         assert_eq!(tracker.description(), "test request");
+        assert_eq!(tracker.resources().get("locks").map(Vec::len), Some(1));
     }
 }

crates/concurrency/src/lib.rs

@@ -124,21 +124,23 @@ mod backpressure;
 #[cfg(feature = "scheduler")]
 mod scheduler;
 
+pub mod workers;
+
 // Public module exports with feature gates
 #[cfg(feature = "timeout")]
-pub use timeout::{TimeoutConfig, TimeoutGuard, TimeoutManager};
+pub use timeout::{TimeoutGuard, TimeoutManager, TimeoutManagerPolicy};
 
 #[cfg(feature = "lock")]
 pub use lock::{LockConfig, LockManager, LockScopeGuard, OptimizedLockGuard};
 
 #[cfg(feature = "deadlock")]
-pub use deadlock::{DeadlockConfig, DeadlockManager, RequestTracker};
+pub use deadlock::{DeadlockManager, DeadlockMonitorPolicy, RequestTracker};
 
 #[cfg(feature = "backpressure")]
-pub use backpressure::{BackpressureConfig, BackpressureManager, BackpressurePipe};
+pub use backpressure::{BackpressureManager, BackpressurePipe, PipeBackpressurePolicy};
 
 #[cfg(feature = "scheduler")]
-pub use scheduler::{IoStrategy, SchedulerConfig, SchedulerManager};
+pub use scheduler::{IoStrategy, SchedulerManager, SchedulerPolicy};
 
 // Configuration
 mod config;
@@ -153,19 +155,19 @@ pub mod prelude {
     //! Prelude module for convenient imports
 
     #[cfg(feature = "timeout")]
-    pub use crate::timeout::{TimeoutConfig, TimeoutGuard, TimeoutManager};
+    pub use crate::timeout::{TimeoutGuard, TimeoutManager, TimeoutManagerPolicy};
 
     #[cfg(feature = "lock")]
     pub use crate::lock::{LockConfig, LockManager, LockScopeGuard, OptimizedLockGuard};
 
     #[cfg(feature = "deadlock")]
-    pub use crate::deadlock::{DeadlockConfig, DeadlockManager, RequestTracker};
+    pub use crate::deadlock::{DeadlockManager, DeadlockMonitorPolicy, RequestTracker};
 
     #[cfg(feature = "backpressure")]
-    pub use crate::backpressure::{BackpressureConfig, BackpressureManager, BackpressurePipe};
+    pub use crate::backpressure::{BackpressureManager, BackpressurePipe, PipeBackpressurePolicy};
 
     #[cfg(feature = "scheduler")]
-    pub use crate::scheduler::{IoStrategy, SchedulerConfig, SchedulerManager};
+    pub use crate::scheduler::{IoStrategy, SchedulerManager, SchedulerPolicy};
 
     pub use crate::{ConcurrencyConfig, ConcurrencyFeatures, ConcurrencyManager};
 }

crates/concurrency/src/manager.rs

@@ -85,39 +85,22 @@ impl ConcurrencyManager {
 
         Self {
             #[cfg(feature = "timeout")]
-            timeout: Arc::new(crate::timeout::TimeoutManager::new(
-                config.default_timeout,
-                config.max_timeout,
-                config.enable_dynamic_timeout,
-            )),
+            timeout: Arc::new(crate::timeout::TimeoutManager::from_policy(config.timeout_policy)),
 
             #[cfg(feature = "lock")]
             lock: Arc::new(crate::lock::LockManager::new(
-                config.enable_lock_optimization,
-                config.lock_acquire_timeout,
+                config.lock_policy.enabled,
+                config.lock_policy.acquire_timeout,
             )),
 
             #[cfg(feature = "deadlock")]
-            deadlock: Arc::new(crate::deadlock::DeadlockManager::new(
-                config.enable_deadlock_detection,
-                config.deadlock_check_interval,
-                config.hang_threshold,
-            )),
+            deadlock: Arc::new(crate::deadlock::DeadlockManager::from_policy(config.deadlock_policy)),
 
             #[cfg(feature = "backpressure")]
-            backpressure: Arc::new(crate::backpressure::BackpressureManager::new(
-                config.backpressure_buffer_size,
-                config.high_watermark,
-                config.low_watermark,
-            )),
+            backpressure: Arc::new(crate::backpressure::BackpressureManager::from_policy(config.backpressure_policy)),
 
             #[cfg(feature = "scheduler")]
-            scheduler: Arc::new(crate::scheduler::SchedulerManager::new(
-                config.io_buffer_size,
-                config.max_buffer_size,
-                config.high_priority_threshold,
-                config.low_priority_threshold,
-            )),
+            scheduler: Arc::new(crate::scheduler::SchedulerManager::from_policy(config.scheduler_policy)),
 
             config,
         }
@@ -244,7 +227,7 @@ impl ConcurrencyManager {
     pub async fn start(&self) {
         #[cfg(feature = "deadlock")]
         {
-            if self.config.enable_deadlock_detection {
+            if self.config.deadlock_policy.enabled {
                 self.deadlock.start().await;
             }
         }

crates/concurrency/src/scheduler.rs

@@ -22,9 +22,9 @@ use rustfs_io_metrics::io_metrics;
 use std::sync::Arc;
 use std::time::Duration;
 
-/// Scheduler configuration
-#[derive(Debug, Clone)]
-pub struct SchedulerConfig {
+/// Facade policy for the concurrency-layer scheduler manager.
+#[derive(Debug, Clone, Copy)]
+pub struct SchedulerPolicy {
     /// Base buffer size
     pub base_buffer_size: usize,
     /// Maximum buffer size
@@ -35,7 +35,7 @@ pub struct SchedulerConfig {
     pub low_priority_threshold: usize,
 }
 
-impl Default for SchedulerConfig {
+impl Default for SchedulerPolicy {
     fn default() -> Self {
         Self {
             base_buffer_size: 64 * 1024,              // 64KB
@@ -46,9 +46,23 @@ impl Default for SchedulerConfig {
     }
 }
 
+impl SchedulerPolicy {
+    /// Convert facade policy to io-core scheduler config.
+    pub fn to_core_config(&self) -> rustfs_io_core::IoSchedulerConfig {
+        rustfs_io_core::IoSchedulerConfig {
+            base_buffer_size: self.base_buffer_size,
+            max_buffer_size: self.max_buffer_size,
+            high_priority_size_threshold: self.high_priority_threshold,
+            low_priority_size_threshold: self.low_priority_threshold,
+            ..rustfs_io_core::IoSchedulerConfig::default()
+        }
+    }
+}
+
 /// Scheduler manager
 pub struct SchedulerManager {
-    config: SchedulerConfig,
+    config: SchedulerPolicy,
+    core_config: rustfs_io_core::IoSchedulerConfig,
     scheduler: Arc<CoreIoScheduler>,
 }
 
@@ -60,26 +74,35 @@ impl SchedulerManager {
         high_priority_threshold: usize,
         low_priority_threshold: usize,
     ) -> Self {
-        let config = SchedulerConfig {
+        Self::from_policy(SchedulerPolicy {
             base_buffer_size,
             max_buffer_size,
             high_priority_threshold,
             low_priority_threshold,
-        };
+        })
+    }
 
-        let core_config = rustfs_io_core::IoSchedulerConfig::default();
+    /// Create a scheduler manager from facade policy.
+    pub fn from_policy(config: SchedulerPolicy) -> Self {
+        let core_config = config.to_core_config();
 
         Self {
             config,
+            core_config: core_config.clone(),
             scheduler: Arc::new(CoreIoScheduler::new(core_config)),
         }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &SchedulerConfig {
+    pub fn config(&self) -> &SchedulerPolicy {
         &self.config
     }
 
+    /// Get the derived io-core scheduler config.
+    pub fn core_config(&self) -> &rustfs_io_core::IoSchedulerConfig {
+        &self.core_config
+    }
+
     /// Get the scheduler
     pub fn scheduler(&self) -> Arc<CoreIoScheduler> {
         self.scheduler.clone()
@@ -87,7 +110,7 @@ impl SchedulerManager {
 
     /// Create an I/O strategy
     pub fn create_strategy(&self) -> IoStrategy {
-        IoStrategy::new(self.config.clone(), self.scheduler.clone())
+        IoStrategy::new(self.config, self.scheduler.clone())
     }
 
     /// Calculate buffer size
@@ -111,12 +134,12 @@ impl SchedulerManager {
 
 /// I/O strategy
 pub struct IoStrategy {
-    config: SchedulerConfig,
+    config: SchedulerPolicy,
     scheduler: Arc<CoreIoScheduler>,
 }
 
 impl IoStrategy {
-    fn new(config: SchedulerConfig, scheduler: Arc<CoreIoScheduler>) -> Self {
+    fn new(config: SchedulerPolicy, scheduler: Arc<CoreIoScheduler>) -> Self {
         Self { config, scheduler }
     }
 
@@ -191,7 +214,7 @@ impl IoStrategy {
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &SchedulerConfig {
+    pub fn config(&self) -> &SchedulerPolicy {
         &self.config
     }
 }
@@ -202,10 +225,20 @@ mod tests {
 
     #[test]
     fn test_scheduler_config() {
-        let config = SchedulerConfig::default();
+        let config = SchedulerPolicy::default();
         assert!(config.base_buffer_size < config.max_buffer_size);
     }
 
+    #[test]
+    fn test_scheduler_policy_to_core_config() {
+        let policy = SchedulerPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.base_buffer_size, policy.base_buffer_size);
+        assert_eq!(core.max_buffer_size, policy.max_buffer_size);
+        assert_eq!(core.high_priority_size_threshold, policy.high_priority_threshold);
+        assert_eq!(core.low_priority_size_threshold, policy.low_priority_threshold);
+    }
+
     #[test]
     fn test_scheduler_manager() {
         let manager = SchedulerManager::new(1024, 4096, 512, 2048);

crates/concurrency/src/timeout.rs

@@ -14,60 +14,101 @@
 
 //! Timeout management for operations
 
-use rustfs_io_core::{TimeoutError, calculate_adaptive_timeout};
+use rustfs_io_core::{TimeoutConfig as CoreTimeoutConfig, TimeoutError, calculate_adaptive_timeout};
 use std::time::{Duration, Instant};
 use tokio_util::sync::CancellationToken;
 
-/// Timeout configuration
-#[derive(Debug, Clone)]
-pub struct TimeoutConfig {
+/// Facade policy for the concurrency-layer timeout manager.
+#[derive(Debug, Clone, Copy)]
+pub struct TimeoutManagerPolicy {
     /// Default timeout duration
     pub default_timeout: Duration,
     /// Maximum timeout duration
     pub max_timeout: Duration,
+    /// Minimum timeout floor (prevents dynamic calculation from going too low).
+    pub min_timeout: Duration,
     /// Enable dynamic timeout calculation
     pub enable_dynamic: bool,
 }
 
-impl Default for TimeoutConfig {
+impl Default for TimeoutManagerPolicy {
     fn default() -> Self {
         Self {
             default_timeout: Duration::from_secs(30),
             max_timeout: Duration::from_secs(300),
+            min_timeout: Duration::from_secs(5),
             enable_dynamic: true,
         }
     }
 }
 
+impl TimeoutManagerPolicy {
+    /// Convert the facade policy into the reusable io-core timeout configuration.
+    ///
+    /// This keeps the concurrency layer explicitly wired to the shared core
+    /// timeout primitives without changing the facade's public behavior.
+    pub fn to_core_config(&self) -> CoreTimeoutConfig {
+        CoreTimeoutConfig {
+            base_timeout: self.default_timeout,
+            timeout_per_mb: Duration::ZERO,
+            max_timeout: self.max_timeout,
+            min_timeout: self.min_timeout,
+            get_object_timeout: self.default_timeout,
+            put_object_timeout: self.max_timeout,
+            list_objects_timeout: self.default_timeout,
+            enable_dynamic_timeout: self.enable_dynamic,
+        }
+    }
+}
+
 /// Timeout manager
 pub struct TimeoutManager {
-    config: TimeoutConfig,
+    config: TimeoutManagerPolicy,
+    core_config: CoreTimeoutConfig,
 }
 
 impl TimeoutManager {
     /// Create a new timeout manager
     pub fn new(default_timeout: Duration, max_timeout: Duration, enable_dynamic: bool) -> Self {
-        Self {
-            config: TimeoutConfig {
-                default_timeout,
-                max_timeout,
-                enable_dynamic,
-            },
-        }
+        let min_timeout = default_timeout.min(max_timeout);
+        Self::from_policy(TimeoutManagerPolicy {
+            default_timeout,
+            max_timeout,
+            min_timeout,
+            enable_dynamic,
+        })
+    }
+
+    /// Create a new timeout manager from the facade policy type.
+    pub fn from_policy(config: TimeoutManagerPolicy) -> Self {
+        let config = TimeoutManagerPolicy {
+            // Guard clamp(min, max) from panic when callers provide an
+            // out-of-order policy (or very small max_timeout).
+            min_timeout: config.min_timeout.min(config.max_timeout),
+            ..config
+        };
+        let core_config = config.to_core_config();
+        Self { config, core_config }
     }
 
     /// Get the configuration
-    pub fn config(&self) -> &TimeoutConfig {
+    pub fn config(&self) -> &TimeoutManagerPolicy {
         &self.config
     }
 
+    /// Get the derived io-core timeout configuration.
+    pub fn core_config(&self) -> &CoreTimeoutConfig {
+        &self.core_config
+    }
+
     /// Calculate timeout for a given size
     pub fn calculate_timeout(&self, size: u64, _history: &[Duration]) -> Duration {
         if !self.config.enable_dynamic {
             return self.config.default_timeout;
         }
 
-        calculate_adaptive_timeout(self.config.default_timeout, None, 0, size).min(self.config.max_timeout)
+        calculate_adaptive_timeout(self.core_config.base_timeout, None, 0, size)
+            .clamp(self.core_config.min_timeout, self.core_config.max_timeout)
     }
 
     /// Wrap an operation with timeout control
@@ -87,7 +128,7 @@ impl TimeoutManager {
 
     /// Create a timeout guard for manual timeout control
     pub fn create_guard(&self, timeout: Option<Duration>) -> TimeoutGuard {
-        TimeoutGuard::new(timeout.unwrap_or(self.config.default_timeout))
+        TimeoutGuard::new(timeout.unwrap_or(self.core_config.base_timeout))
     }
 }
 
@@ -134,10 +175,41 @@ mod tests {
 
     #[test]
     fn test_timeout_config() {
-        let config = TimeoutConfig::default();
+        let config = TimeoutManagerPolicy::default();
         assert!(config.default_timeout < config.max_timeout);
     }
 
+    #[test]
+    fn test_timeout_policy_to_core_config() {
+        let policy = TimeoutManagerPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.base_timeout, policy.default_timeout);
+        assert_eq!(core.max_timeout, policy.max_timeout);
+        assert_eq!(core.min_timeout, policy.min_timeout);
+        assert_eq!(core.get_object_timeout, policy.default_timeout);
+        assert!(core.enable_dynamic_timeout);
+    }
+
+    #[test]
+    fn test_timeout_manager_new_sanitizes_min_timeout_with_small_max_timeout() {
+        let manager = TimeoutManager::new(Duration::from_secs(1), Duration::from_secs(1), true);
+        let timeout = manager.calculate_timeout(1024, &[]);
+        assert_eq!(timeout, Duration::from_secs(1));
+    }
+
+    #[test]
+    fn test_timeout_manager_from_policy_sanitizes_min_timeout() {
+        let manager = TimeoutManager::from_policy(TimeoutManagerPolicy {
+            default_timeout: Duration::from_secs(30),
+            max_timeout: Duration::from_secs(1),
+            min_timeout: Duration::from_secs(5),
+            enable_dynamic: true,
+        });
+
+        assert_eq!(manager.config().min_timeout, Duration::from_secs(1));
+        assert_eq!(manager.core_config().min_timeout, Duration::from_secs(1));
+    }
+
     #[tokio::test]
     async fn test_wrap_operation_success() {
         let manager = TimeoutManager::new(Duration::from_secs(5), Duration::from_secs(10), true);

crates/concurrency/src/workers.rs

@@ -12,10 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+//! Worker slot limiter used by long-running background workflows.
+
 use std::sync::Arc;
 use tokio::sync::{Mutex, Notify};
 use tracing::info;
 
+/// Cooperative worker-slot controller for async tasks.
 pub struct Workers {
     available: Mutex<usize>, // Available working slots
     notify: Notify,          // Used to notify waiting tasks
@@ -23,20 +26,20 @@ pub struct Workers {
 }
 
 impl Workers {
-    // Create a Workers object that allows up to n jobs to execute concurrently
-    pub fn new(n: usize) -> Result<Arc<Workers>, &'static str> {
+    /// Create a [`Workers`] object that allows up to `n` jobs to execute concurrently.
+    pub fn new(n: usize) -> Result<Arc<Self>, &'static str> {
         if n == 0 {
             return Err("n must be > 0");
         }
 
-        Ok(Arc::new(Workers {
+        Ok(Arc::new(Self {
             available: Mutex::new(n),
             notify: Notify::new(),
             limit: n,
         }))
     }
 
-    // Give a job a chance to be executed
+    /// Acquire a worker slot, waiting until one becomes available.
     pub async fn take(&self) {
         loop {
             let mut available = self.available.lock().await;
@@ -51,7 +54,7 @@ impl Workers {
         }
     }
 
-    // Release a job's slot
+    /// Release a worker slot.
     pub async fn give(&self) {
         let mut available = self.available.lock().await;
         info!("worker give, {}", *available);
@@ -59,7 +62,7 @@ impl Workers {
         self.notify.notify_one(); // Notify a waiting task
     }
 
-    // Wait for all concurrent jobs to complete
+    /// Wait until all worker slots are released.
     pub async fn wait(&self) {
         loop {
             {
@@ -74,6 +77,7 @@ impl Workers {
         info!("worker wait end");
     }
 
+    /// Return the current number of available worker slots.
     pub async fn available(&self) -> usize {
         *self.available.lock().await
     }

crates/config/README.md

@@ -65,11 +65,34 @@ Current guidance:
 ## Scanner environment aliases
 
 - `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)
+- `RUSTFS_SCANNER_DELAY` (canonical)
+- `RUSTFS_SCANNER_MAX_WAIT_SECS` (canonical)
 - `RUSTFS_SCANNER_CYCLE` (canonical, also accepts `MINIO_SCANNER_CYCLE`)
 - `RUSTFS_SCANNER_START_DELAY_SECS` (canonical)
 - `RUSTFS_DATA_SCANNER_START_DELAY_SECS` (deprecated alias for compatibility)
 - `RUSTFS_SCANNER_IDLE_MODE` (canonical)
 - `RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical)
+- `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical)
+
+## Health compatibility switches
+
+- `RUSTFS_HEALTH_ENDPOINT_ENABLE`
+  - controls canonical `/health`, `/health/live`, and `/health/ready` endpoint exposure.
+- `RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE`
+  - enables minimal payload mode for GET health responses (`status`, `ready` only).
+- `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
+  - TTL for readiness cache evaluation.
+- `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
+  - enables busy protection behavior for health probes.
+  - default is `false`.
+- `RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS`
+  - max active HTTP requests; health probes return `429` when active requests reach or exceed this value.
+  - `0` disables thresholding even if busy protection is enabled.
+- `RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE`
+  - enables KMS readiness enforcement for `/health/ready`.
+  - default is `false`.
 
 ## Drive timeout environment variables
 
@@ -83,6 +106,28 @@ Legacy compatibility fallback:
 - `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION`
   This legacy variable is treated as a deprecated fallback for the operation-specific drive timeout variables above when a canonical variable is unset.
 
+Drive timeout health-action policy:
+- `RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION`
+  - `mark_failure` (default): timeout marks failure and may transition drive runtime state.
+  - `ignore_scanner`: timeout does not mark failure for scanner-sensitive operations (`walk_dir`, `read_metadata`, `list_dir`, `disk_info`).
+
+Drive timeout profile preset:
+- `RUSTFS_DRIVE_TIMEOUT_PROFILE`
+  - `default` (default): keep current timeout defaults.
+  - `high_latency`: use 60s default timeout for scanner-sensitive operations when no per-operation timeout override is set (`read_metadata`, `disk_info`, `list_dir`, `walk_dir`, `walk_dir_stall`).
+- Precedence:
+  - Explicit per-operation timeout env (`RUSTFS_DRIVE_*_TIMEOUT_SECS`) takes highest precedence.
+  - Then `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION` legacy fallback.
+  - Then the profile-derived default (`default` or `high_latency`).
+
+## Startup filesystem boundary policy
+
+- `RUSTFS_UNSUPPORTED_FS_POLICY` controls startup behavior when RustFS detects local endpoint filesystems that are outside the supported production boundary.
+  - `warn` (default): log warning and continue startup.
+  - `fail`: abort startup with an error.
+
+RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard.
+
 ## 📄 License
 
 This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.

crates/config/src/audit/kafka.rs

@@ -21,10 +21,14 @@ pub const ENV_AUDIT_KAFKA_TLS_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_TLS_ENABLE";
 pub const ENV_AUDIT_KAFKA_TLS_CA: &str = "RUSTFS_AUDIT_KAFKA_TLS_CA";
 pub const ENV_AUDIT_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_CERT";
 pub const ENV_AUDIT_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_AUDIT_KAFKA_SASL_ENABLE: &str = "RUSTFS_AUDIT_KAFKA_SASL_ENABLE";
+pub const ENV_AUDIT_KAFKA_SASL_MECHANISM: &str = "RUSTFS_AUDIT_KAFKA_SASL_MECHANISM";
+pub const ENV_AUDIT_KAFKA_SASL_USERNAME: &str = "RUSTFS_AUDIT_KAFKA_SASL_USERNAME";
+pub const ENV_AUDIT_KAFKA_SASL_PASSWORD: &str = "RUSTFS_AUDIT_KAFKA_SASL_PASSWORD";
 pub const ENV_AUDIT_KAFKA_QUEUE_DIR: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_DIR";
 pub const ENV_AUDIT_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_KAFKA_QUEUE_LIMIT";
 
-pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
+pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 14] = &[
     ENV_AUDIT_KAFKA_ENABLE,
     ENV_AUDIT_KAFKA_BROKERS,
     ENV_AUDIT_KAFKA_TOPIC,
@@ -33,6 +37,10 @@ pub const ENV_AUDIT_KAFKA_KEYS: &[&str; 10] = &[
     ENV_AUDIT_KAFKA_TLS_CA,
     ENV_AUDIT_KAFKA_TLS_CLIENT_CERT,
     ENV_AUDIT_KAFKA_TLS_CLIENT_KEY,
+    ENV_AUDIT_KAFKA_SASL_ENABLE,
+    ENV_AUDIT_KAFKA_SASL_MECHANISM,
+    ENV_AUDIT_KAFKA_SASL_USERNAME,
+    ENV_AUDIT_KAFKA_SASL_PASSWORD,
     ENV_AUDIT_KAFKA_QUEUE_DIR,
     ENV_AUDIT_KAFKA_QUEUE_LIMIT,
 ];
@@ -47,6 +55,10 @@ pub const AUDIT_KAFKA_KEYS: &[&str] = &[
     crate::KAFKA_TLS_CA,
     crate::KAFKA_TLS_CLIENT_CERT,
     crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_SASL_ENABLE,
+    crate::KAFKA_SASL_MECHANISM,
+    crate::KAFKA_SASL_USERNAME,
+    crate::KAFKA_SASL_PASSWORD,
     crate::KAFKA_QUEUE_DIR,
     crate::KAFKA_QUEUE_LIMIT,
     crate::COMMENT_KEY,

crates/config/src/constants/app.rs

@@ -155,6 +155,16 @@ pub const ENV_RUSTFS_SECRET_KEY: &str = "RUSTFS_SECRET_KEY";
 /// Environment variable for server secret key file.
 pub const ENV_RUSTFS_SECRET_KEY_FILE: &str = "RUSTFS_SECRET_KEY_FILE";
 
+/// Environment variable controlling startup behavior when unsupported filesystem types are detected.
+///
+/// Accepted values:
+/// - "warn" (default): log a warning and continue startup
+/// - "fail": abort startup with an error
+pub const ENV_RUSTFS_UNSUPPORTED_FS_POLICY: &str = "RUSTFS_UNSUPPORTED_FS_POLICY";
+pub const RUSTFS_UNSUPPORTED_FS_POLICY_WARN: &str = "warn";
+pub const RUSTFS_UNSUPPORTED_FS_POLICY_FAIL: &str = "fail";
+pub const DEFAULT_RUSTFS_UNSUPPORTED_FS_POLICY: &str = RUSTFS_UNSUPPORTED_FS_POLICY_WARN;
+
 /// Environment variable for server OBS endpoint.
 pub const ENV_RUSTFS_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
 
@@ -293,9 +303,9 @@ pub const DEFAULT_OBS_LOGS_EXPORT_ENABLED: bool = true;
 
 /// Default profiling export enabled
 /// It is used to enable or disable exporting profiles
-/// Default value: false
+/// Default value: true
 /// Environment variable: RUSTFS_OBS_PROFILING_EXPORT_ENABLED
-pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = false;
+pub const DEFAULT_OBS_PROFILING_EXPORT_ENABLED: bool = true;
 
 /// Default log local logging enabled for rustfs
 /// This is the default log local logging enabled for rustfs.

crates/config/src/constants/drive.rs

@@ -47,6 +47,16 @@ pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;
 pub const ENV_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS";
 pub const DEFAULT_DRIVE_ACTIVE_CHECK_TIMEOUT_SECS: u64 = 5;
 
+/// Timeout-to-health transition policy for drive operations.
+///
+/// Accepted values:
+/// - "mark_failure" (default): timeout marks failure and may transition runtime state.
+/// - "ignore_scanner": timeout does not mark failure for scanner-sensitive operations.
+pub const ENV_DRIVE_TIMEOUT_HEALTH_ACTION: &str = "RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION";
+pub const DRIVE_TIMEOUT_HEALTH_ACTION_MARK_FAILURE: &str = "mark_failure";
+pub const DRIVE_TIMEOUT_HEALTH_ACTION_IGNORE_SCANNER: &str = "ignore_scanner";
+pub const DEFAULT_DRIVE_TIMEOUT_HEALTH_ACTION: &str = DRIVE_TIMEOUT_HEALTH_ACTION_MARK_FAILURE;
+
 /// Number of consecutive failures before a suspect drive is classified as offline.
 pub const ENV_DRIVE_SUSPECT_FAILURE_THRESHOLD: &str = "RUSTFS_DRIVE_SUSPECT_FAILURE_THRESHOLD";
 pub const DEFAULT_DRIVE_SUSPECT_FAILURE_THRESHOLD: u64 = 2;
@@ -66,3 +76,19 @@ pub const DEFAULT_DRIVE_OFFLINE_GRACE_PERIOD_SECS: u64 = 30;
 /// Duration in seconds after which a recovered drive is classified as long offline.
 pub const ENV_DRIVE_LONG_OFFLINE_THRESHOLD_SECS: &str = "RUSTFS_DRIVE_LONG_OFFLINE_THRESHOLD_SECS";
 pub const DEFAULT_DRIVE_LONG_OFFLINE_THRESHOLD_SECS: u64 = 172_800;
+
+/// Drive timeout profile preset.
+///
+/// Accepted values:
+/// - "default": keep current timeout defaults.
+/// - "high_latency": use a higher default timeout preset for scanner-sensitive and metadata operations.
+///
+/// Explicit per-operation overrides (`RUSTFS_DRIVE_*_TIMEOUT_SECS`) still take precedence.
+pub const ENV_DRIVE_TIMEOUT_PROFILE: &str = "RUSTFS_DRIVE_TIMEOUT_PROFILE";
+pub const DRIVE_TIMEOUT_PROFILE_DEFAULT: &str = "default";
+pub const DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY: &str = "high_latency";
+pub const DEFAULT_DRIVE_TIMEOUT_PROFILE: &str = DRIVE_TIMEOUT_PROFILE_DEFAULT;
+
+/// Timeout preset (seconds) used when `RUSTFS_DRIVE_TIMEOUT_PROFILE=high_latency`
+/// and no per-operation timeout override is provided.
+pub const DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS: u64 = 60;

crates/config/src/constants/heal.rs

@@ -12,6 +12,18 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+/// Heal admin config subsystem name.
+pub const HEAL_SUB_SYS: &str = "heal";
+
+/// Heal config key setting the scanner-driven periodic deep bitrot scan cycle in seconds.
+pub const HEAL_BITROT_CYCLE: &str = "bitrot_cycle";
+
+/// Heal config keys supported by the admin config subsystem.
+pub const HEAL_KEYS: &[&str] = &[HEAL_BITROT_CYCLE];
+
+/// Default scanner-driven bitrot scan cycle used by heal/scanner runtime config.
+pub const DEFAULT_HEAL_BITROT_CYCLE_SECS: u64 = 30 * 24 * 60 * 60;
+
 /// Environment variable name that enables or disables auto-heal functionality.
 /// - Purpose: Control whether the system automatically performs heal operations.
 /// - Valid values: "true" or "false" (case insensitive).

crates/config/src/constants/health.rs

@@ -26,3 +26,20 @@ pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
 /// When enabled, only `status` and `ready` fields are returned.
 pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE";
 pub const DEFAULT_HEALTH_MINIMAL_RESPONSE_ENABLE: bool = false;
+
+/// Enable busy protection for health probes.
+/// When enabled with a positive request threshold, alias health probes may
+/// return 429 when active HTTP requests exceed the threshold.
+pub const ENV_HEALTH_COMPAT_BUSY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE";
+pub const DEFAULT_HEALTH_COMPAT_BUSY_CHECK_ENABLE: bool = false;
+
+/// Max active HTTP requests; alias health probes report busy (429) when active requests reach or exceed this value.
+/// Set to 0 to disable thresholding even when busy protection is enabled.
+pub const ENV_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: &str = "RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS";
+pub const DEFAULT_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: usize = 0;
+
+/// Enable KMS readiness check for alias readiness probes.
+/// When enabled, `/health/ready` additionally requires KMS service to be
+/// in running state if a global KMS manager exists.
+pub const ENV_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE";
+pub const DEFAULT_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: bool = false;

crates/config/src/constants/internode.rs

@@ -32,6 +32,16 @@ pub const DEFAULT_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 3;
 pub const ENV_INTERNODE_RPC_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS";
 pub const DEFAULT_INTERNODE_RPC_TIMEOUT_SECS: u64 = 10;
 
+/// Environment variable for selecting the internode data-plane transport backend.
+pub const ENV_RUSTFS_INTERNODE_DATA_TRANSPORT: &str = "RUSTFS_INTERNODE_DATA_TRANSPORT";
+pub const DEFAULT_INTERNODE_DATA_TRANSPORT: &str = "tcp-http";
+
+/// Legacy alias for "tcp-http". Both values select the TCP/HTTP transport backend.
+pub const INTERNODE_DATA_TRANSPORT_TCP: &str = "tcp";
+
+/// Known internode transport backend names accepted by the config parser.
+pub const KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS: &[&str] = &[DEFAULT_INTERNODE_DATA_TRANSPORT, INTERNODE_DATA_TRANSPORT_TCP];
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -58,5 +68,9 @@ mod tests {
             "RUSTFS_INTERNODE_HTTP2_KEEPALIVE_TIMEOUT_SECS"
         );
         assert_eq!(ENV_INTERNODE_RPC_TIMEOUT_SECS, "RUSTFS_INTERNODE_RPC_TIMEOUT_SECS");
+        assert_eq!(ENV_RUSTFS_INTERNODE_DATA_TRANSPORT, "RUSTFS_INTERNODE_DATA_TRANSPORT");
+        assert_eq!(DEFAULT_INTERNODE_DATA_TRANSPORT, "tcp-http");
+        assert_eq!(INTERNODE_DATA_TRANSPORT_TCP, "tcp");
+        assert_eq!(KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS, &["tcp-http", "tcp"]);
     }
 }

crates/config/src/constants/object.rs

@@ -281,6 +281,40 @@ pub const ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT: &str = "RUSTFS_OBJECT_LOCK_ACQUIRE_TI
 /// Default lock acquisition timeout: 5 seconds.
 pub const DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT: u64 = 5;
 
+/// Environment variable to enable object namespace lock diagnostics.
+///
+/// When enabled, RustFS emits slow lock acquisition and long lock hold
+/// warnings for object-level namespace locks. This is intended for
+/// production debugging of contention on hot object keys.
+///
+/// Default: false (disabled, can be overridden by `RUSTFS_OBJECT_LOCK_DIAG_ENABLE`).
+pub const ENV_OBJECT_LOCK_DIAG_ENABLE: &str = "RUSTFS_OBJECT_LOCK_DIAG_ENABLE";
+
+/// Default: object lock diagnostics are disabled.
+pub const DEFAULT_OBJECT_LOCK_DIAG_ENABLE: bool = false;
+
+/// Environment variable for the slow object lock acquisition threshold in milliseconds.
+///
+/// When a read or write namespace lock takes at least this long to acquire,
+/// RustFS emits a warning with the operation name and object key.
+///
+/// Default: 500 milliseconds.
+pub const ENV_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS: &str = "RUSTFS_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS";
+
+/// Default slow object lock acquisition threshold: 500 milliseconds.
+pub const DEFAULT_OBJECT_LOCK_DIAG_SLOW_ACQUIRE_MS: u64 = 500;
+
+/// Environment variable for the long object lock hold threshold in milliseconds.
+///
+/// When a namespace lock guard is held for at least this long, RustFS emits
+/// a warning when the guard is dropped.
+///
+/// Default: 1000 milliseconds.
+pub const ENV_OBJECT_LOCK_DIAG_SLOW_HOLD_MS: &str = "RUSTFS_OBJECT_LOCK_DIAG_SLOW_HOLD_MS";
+
+/// Default long object lock hold threshold: 1000 milliseconds.
+pub const DEFAULT_OBJECT_LOCK_DIAG_SLOW_HOLD_MS: u64 = 1000;
+
 // ============================================================================
 // I/O priority scheduling configuration
 // ============================================================================

crates/config/src/constants/oidc.rs

@@ -28,6 +28,7 @@ pub const OIDC_GROUPS_CLAIM: &str = "groups_claim";
 pub const OIDC_ROLES_CLAIM: &str = "roles_claim";
 pub const OIDC_EMAIL_CLAIM: &str = "email_claim";
 pub const OIDC_USERNAME_CLAIM: &str = "username_claim";
+pub const OIDC_HIDE_FROM_UI: &str = "hide_from_ui";
 
 // Environment variable names for OIDC
 pub const ENV_IDENTITY_OPENID_ENABLE: &str = "RUSTFS_IDENTITY_OPENID_ENABLE";
@@ -46,9 +47,10 @@ pub const ENV_IDENTITY_OPENID_GROUPS_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_GROUP
 pub const ENV_IDENTITY_OPENID_ROLES_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_ROLES_CLAIM";
 pub const ENV_IDENTITY_OPENID_EMAIL_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM";
 pub const ENV_IDENTITY_OPENID_USERNAME_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM";
+pub const ENV_IDENTITY_OPENID_HIDE_FROM_UI: &str = "RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI";
 
 /// List of all environment variable keys for an OIDC provider.
-pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
+pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 17] = &[
     ENV_IDENTITY_OPENID_ENABLE,
     ENV_IDENTITY_OPENID_CONFIG_URL,
     ENV_IDENTITY_OPENID_CLIENT_ID,
@@ -65,6 +67,7 @@ pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 16] = &[
     ENV_IDENTITY_OPENID_ROLES_CLAIM,
     ENV_IDENTITY_OPENID_EMAIL_CLAIM,
     ENV_IDENTITY_OPENID_USERNAME_CLAIM,
+    ENV_IDENTITY_OPENID_HIDE_FROM_UI,
 ];
 
 /// A list of all valid configuration keys for an OIDC provider.
@@ -85,6 +88,7 @@ pub const IDENTITY_OPENID_KEYS: &[&str] = &[
     OIDC_ROLES_CLAIM,
     OIDC_EMAIL_CLAIM,
     OIDC_USERNAME_CLAIM,
+    OIDC_HIDE_FROM_UI,
     crate::COMMENT_KEY,
 ];
 

crates/config/src/constants/runtime.rs

@@ -59,6 +59,29 @@ pub const DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT: usize = 10;
 pub const DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE: f64 = 1.0; // 100% sampling
 // Note: S3 bucket/prefix have no default; absence means upload is disabled (modeled as Option<String>)
 
+/// Maximum transition workers used as a local fallback when runtime env is unset.
+pub const DEFAULT_TRANSITION_WORKERS_CAP: i64 = 16;
+/// Absolute upper bound for transition workers accepted from runtime env.
+pub const DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX: i64 = 32;
+/// Default capacity for the transition queue.
+pub const DEFAULT_TRANSITION_QUEUE_CAPACITY: usize = 1000;
+/// Default send timeout for transition queue enqueue attempts, in milliseconds.
+pub const DEFAULT_TRANSITION_QUEUE_SEND_TIMEOUT_MS: usize = 100;
+/// Test-only fault injection env var that forces the immediate transition enqueue timeout path.
+pub const ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT: &str = "RUSTFS_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT";
+/// Test-only fault injection env var that forces a number of IAM bootstrap failures.
+pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS";
+/// Test-only env var that overrides the deferred IAM retry interval in debug builds.
+pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
+/// Runtime env var controlling the transition worker count.
+pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
+/// Runtime env var controlling the absolute maximum transition workers.
+pub const ENV_TRANSITION_WORKERS_ABSOLUTE_MAX: &str = "RUSTFS_ABSOLUTE_MAX_WORKERS";
+/// Runtime env var controlling the transition queue capacity.
+pub const ENV_TRANSITION_QUEUE_CAPACITY: &str = "RUSTFS_TRANSITION_QUEUE_CAPACITY";
+/// Runtime env var controlling the transition queue send timeout in milliseconds.
+pub const ENV_TRANSITION_QUEUE_SEND_TIMEOUT_MS: &str = "RUSTFS_TRANSITION_QUEUE_SEND_TIMEOUT_MS";
+
 // Allocator reclaim configuration
 pub const ENV_ALLOCATOR_RECLAIM_ENABLED: &str = "RUSTFS_ALLOCATOR_RECLAIM_ENABLED";
 pub const ENV_ALLOCATOR_RECLAIM_INTERVAL_SECS: &str = "RUSTFS_ALLOCATOR_RECLAIM_INTERVAL_SECS";

crates/config/src/constants/scanner.rs

@@ -14,6 +14,84 @@
 
 use std::time::Duration;
 
+/// Scanner admin config subsystem name.
+pub const SCANNER_SUB_SYS: &str = "scanner";
+
+/// Scanner config key selecting the speed preset.
+pub const SCANNER_SPEED: &str = "speed";
+
+/// Scanner config key overriding the scanner sleep multiplier.
+pub const SCANNER_DELAY: &str = "delay";
+
+/// Scanner config key overriding the maximum scanner sleep in seconds.
+pub const SCANNER_MAX_WAIT: &str = "max_wait";
+
+/// Scanner config key overriding the cycle interval in seconds.
+pub const SCANNER_CYCLE: &str = "cycle";
+
+/// Scanner config key setting the startup delay in seconds.
+///
+/// For compatibility, this also acts as the scanner cycle interval when
+/// `cycle` is unset.
+pub const SCANNER_START_DELAY: &str = "start_delay";
+
+/// Scanner config key capping one cycle's runtime in seconds.
+pub const SCANNER_CYCLE_MAX_DURATION: &str = "cycle_max_duration";
+
+/// Scanner config key capping objects processed by one cycle.
+pub const SCANNER_CYCLE_MAX_OBJECTS: &str = "cycle_max_objects";
+
+/// Scanner config key capping directories entered by one cycle.
+pub const SCANNER_CYCLE_MAX_DIRECTORIES: &str = "cycle_max_directories";
+
+/// Scanner config key setting the periodic bitrot scan cycle in seconds.
+pub const SCANNER_BITROT_CYCLE: &str = "bitrot_cycle";
+
+/// Scanner config key controlling whether scanner throttling is enabled.
+pub const SCANNER_IDLE_MODE: &str = "idle_mode";
+
+/// Scanner config key controlling scanner cache save timeout in seconds.
+pub const SCANNER_CACHE_SAVE_TIMEOUT: &str = "cache_save_timeout";
+
+/// Scanner config key capping concurrent scanner set tasks.
+pub const SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "max_concurrent_set_scans";
+
+/// Scanner config key capping concurrent scanner disk bucket walks per set.
+pub const SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "max_concurrent_disk_scans";
+
+/// Scanner config key controlling how often object loops yield.
+pub const SCANNER_YIELD_EVERY_N_OBJECTS: &str = "yield_every_n_objects";
+
+/// Scanner config key controlling object version count alerts.
+pub const SCANNER_ALERT_EXCESS_VERSIONS: &str = "alert_excess_versions";
+
+/// Scanner config key controlling retained version size alerts.
+pub const SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "alert_excess_version_size";
+
+/// Scanner config key controlling direct subfolder count alerts.
+pub const SCANNER_ALERT_EXCESS_FOLDERS: &str = "alert_excess_folders";
+
+/// Scanner config keys supported by the admin config subsystem.
+pub const SCANNER_KEYS: &[&str] = &[
+    SCANNER_SPEED,
+    SCANNER_DELAY,
+    SCANNER_MAX_WAIT,
+    SCANNER_CYCLE,
+    SCANNER_START_DELAY,
+    SCANNER_CYCLE_MAX_DURATION,
+    SCANNER_CYCLE_MAX_OBJECTS,
+    SCANNER_CYCLE_MAX_DIRECTORIES,
+    SCANNER_BITROT_CYCLE,
+    SCANNER_IDLE_MODE,
+    SCANNER_CACHE_SAVE_TIMEOUT,
+    SCANNER_MAX_CONCURRENT_SET_SCANS,
+    SCANNER_MAX_CONCURRENT_DISK_SCANS,
+    SCANNER_YIELD_EVERY_N_OBJECTS,
+    SCANNER_ALERT_EXCESS_VERSIONS,
+    SCANNER_ALERT_EXCESS_VERSION_SIZE,
+    SCANNER_ALERT_EXCESS_FOLDERS,
+];
+
 /// Canonical environment variable name that specifies the scanner start delay in seconds.
 /// If set, this overrides the cycle interval derived from `RUSTFS_SCANNER_SPEED`.
 /// - Unit: seconds (u64).
@@ -31,15 +109,84 @@ pub const ENV_DATA_SCANNER_START_DELAY_SECS: &str = "RUSTFS_DATA_SCANNER_START_D
 /// - Example: `export RUSTFS_SCANNER_CYCLE=3600` (1 hour)
 pub const ENV_SCANNER_CYCLE: &str = "RUSTFS_SCANNER_CYCLE";
 
+/// Environment variable that caps one scanner cycle's runtime in seconds.
+/// A value of `0` disables the cycle runtime budget.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS=1800`
+pub const ENV_SCANNER_CYCLE_MAX_DURATION_SECS: &str = "RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS";
+
+/// Environment variable that caps objects processed by one scanner cycle.
+/// A value of `0` disables the per-cycle object budget.
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_OBJECTS=1000000`
+pub const ENV_SCANNER_CYCLE_MAX_OBJECTS: &str = "RUSTFS_SCANNER_CYCLE_MAX_OBJECTS";
+
+/// Environment variable that caps directories entered by one scanner cycle.
+/// A value of `0` disables the per-cycle directory budget.
+/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000`
+pub const ENV_SCANNER_CYCLE_MAX_DIRECTORIES: &str = "RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES";
+
 /// Environment variable that selects the scanner speed preset.
 /// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
 /// Controls the sleep factor, maximum sleep duration, and cycle interval.
 /// - Example: `export RUSTFS_SCANNER_SPEED=slow`
 pub const ENV_SCANNER_SPEED: &str = "RUSTFS_SCANNER_SPEED";
 
+/// Environment variable that overrides the scanner sleep multiplier.
+/// - Example: `export RUSTFS_SCANNER_DELAY=30.0`
+pub const ENV_SCANNER_DELAY: &str = "RUSTFS_SCANNER_DELAY";
+
+/// Environment variable that overrides the maximum scanner sleep in seconds.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_MAX_WAIT_SECS=15`
+pub const ENV_SCANNER_MAX_WAIT_SECS: &str = "RUSTFS_SCANNER_MAX_WAIT_SECS";
+
 /// Default scanner speed preset.
 pub const DEFAULT_SCANNER_SPEED: &str = "default";
 
+/// Default scanner cycle runtime budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_DURATION_SECS: u64 = 0;
+
+/// Default scanner per-cycle object budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_OBJECTS: u64 = 0;
+
+/// Default scanner per-cycle directory budget.
+/// `0` keeps the existing unbounded per-cycle behavior.
+pub const DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES: u64 = 0;
+
+/// Environment variable that specifies the periodic bitrot scan cycle in seconds.
+/// When set to `0`, `true`, `on`, or `yes`, every scanner cycle runs in deep mode.
+/// When set to `false`, `off`, `no`, or `disabled`, periodic deep scans are disabled.
+/// - Unit: seconds (u64).
+/// - Example: `export RUSTFS_SCANNER_BITROT_CYCLE_SECS=2592000` (30 days)
+pub const ENV_SCANNER_BITROT_CYCLE_SECS: &str = "RUSTFS_SCANNER_BITROT_CYCLE_SECS";
+
+/// Default bitrot scan cycle used by the scanner.
+pub const DEFAULT_SCANNER_BITROT_CYCLE_SECS: u64 = 30 * 24 * 60 * 60;
+
+/// Environment variable that controls how many object versions trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS=100`
+pub const ENV_SCANNER_ALERT_EXCESS_VERSIONS: &str = "RUSTFS_SCANNER_ALERT_EXCESS_VERSIONS";
+
+/// Default object version count that triggers scanner alerts.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_VERSIONS: u64 = 100;
+
+/// Environment variable that controls how many cumulative bytes across object versions trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE=1099511627776`
+pub const ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE: &str = "RUSTFS_SCANNER_ALERT_EXCESS_VERSION_SIZE";
+
+/// Default cumulative object version bytes that trigger scanner alerts.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_VERSION_SIZE: u64 = 1024 * 1024 * 1024 * 1024;
+
+/// Environment variable that controls how many subfolders trigger scanner alerts.
+/// - Example: `export RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS=65538`
+pub const ENV_SCANNER_ALERT_EXCESS_FOLDERS: &str = "RUSTFS_SCANNER_ALERT_EXCESS_FOLDERS";
+
+/// Default subfolder count that triggers scanner alerts.
+/// Allows Proxmox Backup Server's chunk namespace layout.
+pub const DEFAULT_SCANNER_ALERT_EXCESS_FOLDERS: u64 = 65_538;
+
 /// Environment variable that controls whether the scanner sleeps between operations.
 /// When `true` (default), the scanner throttles itself. When `false`, it runs at full speed.
 /// - Example: `export RUSTFS_SCANNER_IDLE_MODE=false`
@@ -51,9 +198,38 @@ pub const ENV_SCANNER_IDLE_MODE: &str = "RUSTFS_SCANNER_IDLE_MODE";
 /// - Example: `export RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS=30`
 pub const ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS: &str = "RUSTFS_SCANNER_CACHE_SAVE_TIMEOUT_SECS";
 
+/// Default scanner cache save timeout in seconds.
+pub const DEFAULT_SCANNER_CACHE_SAVE_TIMEOUT_SECS: u64 = 30;
+
+/// Environment variable that caps concurrent scanner set tasks.
+/// A value of `0` keeps the existing topology-based concurrency.
+/// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS=2`
+pub const ENV_SCANNER_MAX_CONCURRENT_SET_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS";
+
+/// Environment variable that caps concurrent scanner disk bucket walks per set.
+/// A value of `0` keeps the existing disk-count-based concurrency.
+/// - Example: `export RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS=1`
+pub const ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS: &str = "RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS";
+
+/// Environment variable that controls how often scanner object loops yield to the async runtime.
+/// A value of `0` disables this extra object-count yield.
+/// - Example: `export RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS=32`
+pub const ENV_SCANNER_YIELD_EVERY_N_OBJECTS: &str = "RUSTFS_SCANNER_YIELD_EVERY_N_OBJECTS";
+
 /// Default scanner idle mode.
 pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
 
+/// Default set scan concurrency budget.
+/// `0` means no additional limit beyond deployment topology.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 0;
+
+/// Default disk scan concurrency budget.
+/// `0` means no additional limit beyond available disks in the set.
+pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 0;
+
+/// Default object interval for cooperative scanner yields.
+pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
+
 /// Compatibility flag kept for Patch 3 rollback windows.
 ///
 /// Inline scanner heal execution has been removed in favor of heal-candidate enqueue.
@@ -78,7 +254,9 @@ pub const DEFAULT_SCANNER_INLINE_HEAL_ENABLE: bool = false;
 /// | `slow`    | 10x    | 15 seconds| 1 minute       |
 /// | `slowest` | 100x   | 15 seconds| 30 minutes     |
 ///
-/// The cycle interval can be overridden by `RUSTFS_SCANNER_CYCLE`.
+/// The sleep factor, max sleep, and cycle interval can be overridden by
+/// `RUSTFS_SCANNER_DELAY`, `RUSTFS_SCANNER_MAX_WAIT_SECS`, and
+/// `RUSTFS_SCANNER_CYCLE`.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
 pub enum ScannerSpeed {
     Fastest,
@@ -117,15 +295,20 @@ impl ScannerSpeed {
         }
     }
 
-    pub fn from_env_str(s: &str) -> Self {
+    pub fn parse_str(s: &str) -> Option<Self> {
         match s.trim().to_ascii_lowercase().as_str() {
-            "fastest" => Self::Fastest,
-            "fast" => Self::Fast,
-            "slow" => Self::Slow,
-            "slowest" => Self::Slowest,
-            _ => Self::Default,
+            "fastest" => Some(Self::Fastest),
+            "fast" => Some(Self::Fast),
+            "default" => Some(Self::Default),
+            "slow" => Some(Self::Slow),
+            "slowest" => Some(Self::Slowest),
+            _ => None,
         }
     }
+
+    pub fn from_env_str(s: &str) -> Self {
+        Self::parse_str(s).unwrap_or_default()
+    }
 }
 
 impl std::fmt::Display for ScannerSpeed {

crates/config/src/constants/targets.rs

@@ -49,6 +49,10 @@ pub const KAFKA_TLS_ENABLE: &str = "tls_enable";
 pub const KAFKA_TLS_CA: &str = "tls_ca";
 pub const KAFKA_TLS_CLIENT_CERT: &str = "tls_client_cert";
 pub const KAFKA_TLS_CLIENT_KEY: &str = "tls_client_key";
+pub const KAFKA_SASL_ENABLE: &str = "sasl_enable";
+pub const KAFKA_SASL_MECHANISM: &str = "sasl_mechanism";
+pub const KAFKA_SASL_USERNAME: &str = "sasl_username";
+pub const KAFKA_SASL_PASSWORD: &str = "sasl_password";
 
 pub const AMQP_URL: &str = "url";
 pub const AMQP_EXCHANGE: &str = "exchange";

crates/config/src/notify/kafka.rs

@@ -22,6 +22,10 @@ pub const NOTIFY_KAFKA_KEYS: &[&str] = &[
     crate::KAFKA_TLS_CA,
     crate::KAFKA_TLS_CLIENT_CERT,
     crate::KAFKA_TLS_CLIENT_KEY,
+    crate::KAFKA_SASL_ENABLE,
+    crate::KAFKA_SASL_MECHANISM,
+    crate::KAFKA_SASL_USERNAME,
+    crate::KAFKA_SASL_PASSWORD,
     crate::KAFKA_QUEUE_DIR,
     crate::KAFKA_QUEUE_LIMIT,
     crate::COMMENT_KEY,
@@ -36,10 +40,14 @@ pub const ENV_NOTIFY_KAFKA_TLS_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_TLS_ENABLE";
 pub const ENV_NOTIFY_KAFKA_TLS_CA: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CA";
 pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_CERT";
 pub const ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_KAFKA_TLS_CLIENT_KEY";
+pub const ENV_NOTIFY_KAFKA_SASL_ENABLE: &str = "RUSTFS_NOTIFY_KAFKA_SASL_ENABLE";
+pub const ENV_NOTIFY_KAFKA_SASL_MECHANISM: &str = "RUSTFS_NOTIFY_KAFKA_SASL_MECHANISM";
+pub const ENV_NOTIFY_KAFKA_SASL_USERNAME: &str = "RUSTFS_NOTIFY_KAFKA_SASL_USERNAME";
+pub const ENV_NOTIFY_KAFKA_SASL_PASSWORD: &str = "RUSTFS_NOTIFY_KAFKA_SASL_PASSWORD";
 pub const ENV_NOTIFY_KAFKA_QUEUE_DIR: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_DIR";
 pub const ENV_NOTIFY_KAFKA_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_KAFKA_QUEUE_LIMIT";
 
-pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
+pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 14] = &[
     ENV_NOTIFY_KAFKA_ENABLE,
     ENV_NOTIFY_KAFKA_BROKERS,
     ENV_NOTIFY_KAFKA_TOPIC,
@@ -48,6 +56,10 @@ pub const ENV_NOTIFY_KAFKA_KEYS: &[&str; 10] = &[
     ENV_NOTIFY_KAFKA_TLS_CA,
     ENV_NOTIFY_KAFKA_TLS_CLIENT_CERT,
     ENV_NOTIFY_KAFKA_TLS_CLIENT_KEY,
+    ENV_NOTIFY_KAFKA_SASL_ENABLE,
+    ENV_NOTIFY_KAFKA_SASL_MECHANISM,
+    ENV_NOTIFY_KAFKA_SASL_USERNAME,
+    ENV_NOTIFY_KAFKA_SASL_PASSWORD,
     ENV_NOTIFY_KAFKA_QUEUE_DIR,
     ENV_NOTIFY_KAFKA_QUEUE_LIMIT,
 ];

crates/credentials/Cargo.toml

@@ -1,3 +1,17 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 [package]
 name = "rustfs-credentials"
 edition.workspace = true
@@ -12,9 +26,11 @@ categories = ["web-programming", "development-tools", "data-structures", "securi
 
 [dependencies]
 base64-simd = { workspace = true }
+hmac = { workspace = true }
 rand = { workspace = true }
 serde = { workspace = true }
 serde_json.workspace = true
+sha2 = { workspace = true }
 time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }
 
 [lints]

crates/credentials/src/credentials.rs

@@ -12,10 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::{DEFAULT_SECRET_KEY, ENV_RPC_SECRET, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+use crate::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ENV_RPC_SECRET, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+use base64_simd::URL_SAFE_NO_PAD;
+use hmac::{Hmac, KeyInit, Mac};
 use rand::{Rng, RngExt};
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
+use sha2::Sha256;
 use std::collections::HashMap;
 use std::env;
 use std::fmt;
@@ -33,7 +36,12 @@ pub static GLOBAL_RUSTFS_RPC_SECRET: OnceLock<String> = OnceLock::new();
 pub const RPC_SECRET_REQUIRED_MESSAGE: &str = "RPC authentication secret is not configured";
 
 /// Operator-facing guidance for configuring RPC authentication safely.
-pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str = "RUSTFS_RPC_SECRET must be set to a non-default value or RUSTFS_SECRET_KEY must be changed from the default for RPC authentication";
+pub const RPC_SECRET_REQUIRED_OPERATOR_MESSAGE: &str =
+    "RUSTFS_RPC_SECRET can be set explicitly; otherwise the RPC secret is derived from the active access/secret key pair";
+
+type HmacSha256 = Hmac<Sha256>;
+
+const RPC_SECRET_DERIVATION_CONTEXT: &[u8] = b"rustfs-rpc-secret:v1";
 
 /// Error type for credentials operations
 #[derive(Debug)]
@@ -217,37 +225,59 @@ pub fn gen_secret_key(length: usize) -> std::io::Result<String> {
     Ok(encoded)
 }
 
-/// Get the RPC authentication token from environment variable
+/// Get the RPC authentication token from the environment or derive it from the active credentials.
 ///
 /// # Returns
 /// * `String` - The RPC authentication token
 ///
-fn resolve_rpc_secret(env_secret: Option<&str>, global_secret: Option<&str>) -> Option<String> {
-    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
-        return (secret != DEFAULT_SECRET_KEY).then(|| secret.to_string());
+fn normalize_rpc_secret(secret: &str) -> Option<String> {
+    let secret = secret.trim();
+    (!secret.is_empty() && secret != DEFAULT_SECRET_KEY && secret != DEFAULT_ACCESS_KEY).then(|| secret.to_string())
+}
+
+fn derive_rpc_secret(access_key: &str, secret_key: &str) -> Option<String> {
+    let access_key = access_key.trim();
+    let secret_key = secret_key.trim();
+
+    if access_key.is_empty() || secret_key.is_empty() {
+        return None;
     }
 
-    global_secret
-        .map(str::trim)
-        .filter(|secret| !secret.is_empty() && *secret != DEFAULT_SECRET_KEY)
-        .map(ToOwned::to_owned)
+    let mut mac = <HmacSha256 as KeyInit>::new_from_slice(secret_key.as_bytes()).expect("HMAC can take key of any size");
+    mac.update(RPC_SECRET_DERIVATION_CONTEXT);
+    mac.update(&[0]);
+    mac.update(access_key.as_bytes());
+
+    Some(URL_SAFE_NO_PAD.encode_to_string(mac.finalize().into_bytes()))
+}
+
+fn resolve_rpc_secret(env_secret: Option<&str>, global_access: Option<&str>, global_secret: Option<&str>) -> Option<String> {
+    if let Some(secret) = env_secret.map(str::trim).filter(|secret| !secret.is_empty()) {
+        return normalize_rpc_secret(secret);
+    }
+
+    match (global_access, global_secret) {
+        (Some(access_key), Some(secret_key)) => derive_rpc_secret(access_key, secret_key),
+        _ => None,
+    }
 }
 
 pub fn try_get_rpc_token() -> std::io::Result<String> {
     if let Some(secret) = GLOBAL_RUSTFS_RPC_SECRET.get() {
-        return resolve_rpc_secret(None, Some(secret)).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
+        return normalize_rpc_secret(secret).ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE));
     }
 
     let env_secret = env::var(ENV_RPC_SECRET).ok();
+    let global_access = get_global_access_key_opt();
     let global_secret = get_global_secret_key_opt();
-    let secret = resolve_rpc_secret(env_secret.as_deref(), global_secret.as_deref())
+    let secret = resolve_rpc_secret(env_secret.as_deref(), global_access.as_deref(), global_secret.as_deref())
         .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE))?;
 
     match GLOBAL_RUSTFS_RPC_SECRET.set(secret.clone()) {
         Ok(()) => Ok(secret),
         Err(_) => GLOBAL_RUSTFS_RPC_SECRET
             .get()
-            .and_then(|stored| resolve_rpc_secret(None, Some(stored)))
+            .and_then(|stored| normalize_rpc_secret(stored))
             .ok_or_else(|| Error::other(RPC_SECRET_REQUIRED_MESSAGE)),
     }
 }
@@ -407,7 +437,7 @@ impl Credentials {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use crate::{IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
+    use crate::{DEFAULT_ACCESS_KEY, IAM_POLICY_CLAIM_NAME_SA, INHERITED_POLICY_TYPE};
     use time::Duration;
 
     #[test]
@@ -510,11 +540,12 @@ mod tests {
         // If it hasn't already been initialized, the test automatically generates logic
         if get_global_action_cred().is_none() {
             init_global_action_credentials(None, None).ok();
-            let ak = get_global_access_key();
-            let sk = get_global_secret_key();
-            assert_eq!(ak.len(), 20);
-            assert_eq!(sk.len(), 32);
         }
+
+        let ak = get_global_access_key();
+        let sk = get_global_secret_key();
+        assert!(ak.len() >= 3);
+        assert!(sk.len() >= 8);
     }
 
     #[test]
@@ -528,10 +559,26 @@ mod tests {
     }
 
     #[test]
-    fn test_resolve_rpc_secret_rejects_default_fallback() {
-        assert!(resolve_rpc_secret(None, None).is_none());
-        assert!(resolve_rpc_secret(None, Some(DEFAULT_SECRET_KEY)).is_none());
-        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-global-secret")).is_none());
+    fn test_gen_access_key_length_and_charset() {
+        let err = gen_access_key(2).expect_err("length below 3 should fail");
+        assert_eq!(err.to_string(), "access key length is too short");
+
+        let key = gen_access_key(20).expect("access key should generate");
+        assert_eq!(key.len(), 20);
+        assert!(key.chars().all(|ch| ch.is_ascii_uppercase() || ch.is_ascii_digit()));
+    }
+
+    #[test]
+    fn test_resolve_rpc_secret_allows_default_credentials_for_derivation() {
+        assert!(resolve_rpc_secret(None, None, None).is_none());
+
+        let expected = derive_rpc_secret(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY).expect("secret should derive");
+        assert_eq!(
+            resolve_rpc_secret(None, Some(DEFAULT_ACCESS_KEY), Some(DEFAULT_SECRET_KEY)).as_deref(),
+            Some(expected.as_str())
+        );
+
+        assert!(resolve_rpc_secret(Some(DEFAULT_SECRET_KEY), Some("custom-access"), Some("custom-global-secret")).is_none());
     }
 
     #[test]
@@ -551,37 +598,64 @@ mod tests {
 
     #[test]
     fn test_resolve_rpc_secret_accepts_non_default_secret() {
-        assert_eq!(resolve_rpc_secret(Some("custom-rpc-secret"), None).as_deref(), Some("custom-rpc-secret"));
         assert_eq!(
-            resolve_rpc_secret(None, Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("custom-rpc-secret"), None, None).as_deref(),
+            Some("custom-rpc-secret")
+        );
+        let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
+        assert_eq!(
+            resolve_rpc_secret(None, Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
     }
 
     #[test]
     fn test_resolve_rpc_secret_trims_and_falls_back_from_blank_env() {
+        let expected = derive_rpc_secret("custom-access", "custom-global-secret").expect("secret should derive");
         assert_eq!(
-            resolve_rpc_secret(Some("  custom-rpc-secret  "), None).as_deref(),
+            resolve_rpc_secret(Some("  custom-rpc-secret  "), None, None).as_deref(),
             Some("custom-rpc-secret")
         );
         assert_eq!(
-            resolve_rpc_secret(Some(""), Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some(""), Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
         assert_eq!(
-            resolve_rpc_secret(Some("  "), Some("  custom-global-secret  ")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("  "), Some("  custom-access  "), Some("  custom-global-secret  ")).as_deref(),
+            Some(expected.as_str())
         );
         assert_eq!(
-            resolve_rpc_secret(Some("  "), Some("custom-global-secret")).as_deref(),
-            Some("custom-global-secret")
+            resolve_rpc_secret(Some("  "), Some("custom-access"), Some("custom-global-secret")).as_deref(),
+            Some(expected.as_str())
         );
     }
 
     #[test]
     fn test_resolve_rpc_secret_returns_none_for_trimmed_default_secret() {
         let padded_default_secret = format!("  {}  ", DEFAULT_SECRET_KEY);
-        assert!(resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-global-secret")).is_none());
+        assert!(
+            resolve_rpc_secret(Some(padded_default_secret.as_str()), Some("custom-access"), Some("custom-global-secret"))
+                .is_none()
+        );
+
+        let padded_default_access = format!("  {}  ", DEFAULT_ACCESS_KEY);
+        assert!(
+            resolve_rpc_secret(Some(padded_default_access.as_str()), Some("custom-access"), Some("custom-global-secret"))
+                .is_none()
+        );
+    }
+
+    #[test]
+    fn test_derive_rpc_secret_is_stable_and_not_plaintext() {
+        let first = derive_rpc_secret("custom-access", "custom-secret").expect("secret should derive");
+        let second = derive_rpc_secret("custom-access", "custom-secret").expect("secret should derive");
+
+        assert_eq!(first, second);
+        assert_ne!(first, "custom-access");
+        assert_ne!(first, "custom-secret");
+        assert_ne!(first, format!("{}{}", "custom-access", "custom-secret"));
+        assert!(!first.contains("custom-access"));
+        assert!(!first.contains("custom-secret"));
     }
 
     #[test]

crates/crypto/Cargo.toml

@@ -33,8 +33,11 @@ aes-gcm = { workspace = true, optional = true }
 argon2 = { workspace = true, optional = true }
 chacha20poly1305 = { workspace = true, optional = true }
 jsonwebtoken = { workspace = true }
+base64-simd = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }
 rand = { workspace = true, optional = true }
+rsa = { workspace = true, features = ["sha2"] }
+serde = { workspace = true, features = ["derive"] }
 sha2 = { workspace = true, optional = true }
 thiserror.workspace = true
 serde_json.workspace = true

crates/crypto/src/lib.rs

@@ -16,6 +16,7 @@
 mod encdec;
 mod error;
 mod jwt;
+pub mod license_token;
 
 pub use encdec::decrypt::decrypt_data;
 pub use encdec::encrypt::encrypt_data;
@@ -25,3 +26,4 @@ pub use encdec::stream_io::{decrypt_stream_io, encrypt_stream_io};
 pub use error::Error;
 pub use jwt::decode::decode as jwt_decode;
 pub use jwt::encode::encode as jwt_encode;
+pub use license_token::{Token, parse_license_with_public_key, parse_signed_license_token, sign_license_token};

crates/crypto/src/license_token.rs

@@ -115,12 +115,20 @@ mod tests {
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
 
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600, // 1 hour from now
         };
 
         let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
@@ -139,12 +147,20 @@ mod tests {
         let private_key = RsaPrivateKey::new(&mut rng, bits).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
 
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600,
         };
 
         let encoded = gencode(&token, &public_key_pem).expect("Failed to encode token");
@@ -159,11 +175,19 @@ mod tests {
         let mut rng = rand::rng();
         let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
-        let private_key_pem = private_key.to_pkcs8_pem(LineEnding::LF).unwrap();
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let private_key_pem = private_key
+            .to_pkcs8_pem(LineEnding::LF)
+            .expect("failed to encode private key pem");
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600,
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600,
         };
 
         let encoded = sign_license_token(&token, &private_key_pem).expect("Failed to encode token");
@@ -181,7 +205,7 @@ mod tests {
 
     #[test]
     fn test_source_does_not_embed_private_key() {
-        let source = include_str!("token.rs");
+        let source = include_str!("license_token.rs");
         let forbidden = ["BEGIN", "PRIVATE KEY"].join(" ");
 
         assert!(!source.contains(&forbidden));
@@ -192,7 +216,9 @@ mod tests {
         let mut rng = rand::rng();
         let private_key = RsaPrivateKey::new(&mut rng, 2048).expect("Failed to generate private key");
         let public_key = RsaPublicKey::from(&private_key);
-        let public_key_pem = public_key.to_public_key_pem(LineEnding::LF).unwrap();
+        let public_key_pem = public_key
+            .to_public_key_pem(LineEnding::LF)
+            .expect("failed to encode public key pem");
 
         let invalid_token = "invalid_base64_token";
         let result = parse_signed_license_token(invalid_token, &public_key_pem);
@@ -204,7 +230,11 @@ mod tests {
     fn test_sign_license_token_with_invalid_signing_key() {
         let token = Token {
             name: "test_app".to_string(),
-            expired: SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 3600, // 1 hour from now
+            expired: SystemTime::now()
+                .duration_since(UNIX_EPOCH)
+                .expect("system clock before unix epoch")
+                .as_secs()
+                + 3600, // 1 hour from now
         };
 
         let invalid_key = "invalid_private_key";

crates/data-usage/Cargo.toml

@@ -0,0 +1,38 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[package]
+name = "rustfs-data-usage"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+repository.workspace = true
+rust-version.workspace = true
+homepage.workspace = true
+description = "Shared data usage models and algorithms for RustFS"
+keywords = ["rustfs", "data-usage", "cache", "histogram"]
+categories = ["data-structures", "filesystem"]
+
+[lints]
+workspace = true
+
+[dependencies]
+serde = { workspace = true }
+path-clean = { workspace = true }
+rmp-serde = { workspace = true }
+async-trait = { workspace = true }
+rustfs-filemeta = { workspace = true }
+
+[lib]
+doctest = false

crates/data-usage/src/data_usage.rs

@@ -312,6 +312,12 @@ impl SizeHistogram {
         }
         res
     }
+
+    pub fn merge_from(&mut self, other: &Self) {
+        for (dst, src) in self.0.iter_mut().zip(other.0.iter()) {
+            *dst += src;
+        }
+    }
 }
 
 /// Versions histogram for version count distribution
@@ -361,6 +367,12 @@ impl VersionsHistogram {
         }
         res
     }
+
+    pub fn merge_from(&mut self, other: &Self) {
+        for (dst, src) in self.0.iter_mut().zip(other.0.iter()) {
+            *dst += src;
+        }
+    }
 }
 
 /// Replication statistics for a single target
@@ -419,6 +431,9 @@ pub struct DataUsageEntry {
     pub obj_versions: VersionsHistogram,
     pub replication_stats: Option<ReplicationAllStats>,
     pub compacted: bool,
+    /// Number of objects that failed to scan (e.g., IO errors)
+    #[serde(default)]
+    pub failed_objects: usize,
 }
 
 impl DataUsageEntry {
@@ -456,6 +471,7 @@ impl DataUsageEntry {
         self.versions += other.versions;
         self.delete_markers += other.delete_markers;
         self.size += other.size;
+        self.failed_objects += other.failed_objects;
 
         if let Some(o_rep) = &other.replication_stats {
             let s_rep = self.replication_stats.get_or_insert_with(ReplicationAllStats::default);
@@ -490,9 +506,11 @@ impl DataUsageEntry {
 #[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageCacheInfo {
     pub name: String,
-    pub next_cycle: u32,
+    pub next_cycle: u64,
     pub last_update: Option<SystemTime>,
     pub skip_healing: bool,
+    #[serde(default)]
+    pub failed_objects: HashMap<String, u64>,
 }
 
 /// Data usage cache
@@ -670,15 +688,14 @@ impl DataUsageCache {
             return;
         }
 
-        let mut leaves = Vec::new();
+        let mut candidates = Vec::new();
         let mut remove = total - limit;
-        add(self, path, &mut leaves);
-        leaves.sort_by_key(|a| a.objects);
+        add(self, path, &mut candidates);
+        candidates.sort_by_key(|a| a.objects);
 
-        while remove > 0 && !leaves.is_empty() {
-            let Some(e) = leaves.first() else {
-                break;
-            };
+        let mut candidate_index = 0;
+        while remove > 0 && candidate_index < candidates.len() {
+            let e = &candidates[candidate_index];
             let candidate = e.path.clone();
             if candidate == *path && !compact_self {
                 break;
@@ -687,7 +704,7 @@ impl DataUsageCache {
             let mut flat = match self.size_recursive(&candidate.key()) {
                 Some(flat) => flat,
                 None => {
-                    leaves.remove(0);
+                    candidate_index += 1;
                     continue;
                 }
             };
@@ -696,8 +713,8 @@ impl DataUsageCache {
             self.delete_recursive(&candidate);
             self.replace_hashed(&candidate, &None, &flat);
 
-            remove -= removing;
-            leaves.remove(0);
+            remove = remove.saturating_sub(removing);
+            candidate_index += 1;
         }
     }
 
@@ -851,23 +868,24 @@ struct Inner {
     path: DataUsageHash,
 }
 
-fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, leaves: &mut Vec<Inner>) {
+fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, candidates: &mut Vec<Inner>) -> usize {
     let e = match data_usage_cache.cache.get(&path.key()) {
         Some(e) => e,
-        None => return,
+        None => return 0,
     };
-    if !e.children.is_empty() {
-        return;
-    }
-
-    let sz = data_usage_cache.size_recursive(&path.key()).unwrap_or_default();
-    leaves.push(Inner {
-        objects: sz.objects,
-        path: path.clone(),
-    });
+    let mut objects = e.objects;
     for ch in e.children.iter() {
-        add(data_usage_cache, &DataUsageHash(ch.clone()), leaves);
+        objects += add(data_usage_cache, &DataUsageHash(ch.clone()), candidates);
     }
+    // Collect internal nodes (with children) as compaction candidates.
+    // Leaf nodes have no children to remove, so compacting them is a no-op.
+    if !e.children.is_empty() {
+        candidates.push(Inner {
+            objects,
+            path: path.clone(),
+        });
+    }
+    objects
 }
 
 fn mark(duc: &DataUsageCache, entry: &DataUsageEntry, found: &mut HashSet<String>) {
@@ -1343,4 +1361,161 @@ mod tests {
         assert_eq!(child_entry.size, 30);
         assert_eq!(child_entry.objects, 3);
     }
+
+    // --- Tests for `add` and `reduce_children_of` (bug fixes) ---
+
+    /// Build a small tree: root -> child1 (leaf), child2 -> grandchild (leaf).
+    fn build_test_tree() -> (DataUsageCache, DataUsageHash) {
+        let root = hash_path("bucket");
+        let c1 = hash_path("bucket/a");
+        let c2 = hash_path("bucket/b");
+        let gc = hash_path("bucket/b/c");
+
+        let mut cache = DataUsageCache::default();
+        cache.replace_hashed(&root, &None, &DataUsageEntry::default());
+        cache.replace_hashed(
+            &c1,
+            &Some(root.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                size: 10,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &c2,
+            &Some(root.clone()),
+            &DataUsageEntry {
+                objects: 2,
+                size: 20,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &gc,
+            &Some(c2.clone()),
+            &DataUsageEntry {
+                objects: 3,
+                size: 30,
+                ..Default::default()
+            },
+        );
+        (cache, root)
+    }
+
+    fn build_underflow_test_tree() -> (DataUsageCache, DataUsageHash) {
+        let root = hash_path("bucket");
+        let small = hash_path("bucket/small");
+        let small_a = hash_path("bucket/small/a");
+        let small_b = hash_path("bucket/small/b");
+        let large = hash_path("bucket/large");
+        let large_a = hash_path("bucket/large/a");
+        let large_b = hash_path("bucket/large/b");
+
+        let mut cache = DataUsageCache::default();
+        cache.replace_hashed(
+            &root,
+            &None,
+            &DataUsageEntry {
+                objects: 100,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(&small, &Some(root.clone()), &DataUsageEntry::default());
+        cache.replace_hashed(
+            &small_a,
+            &Some(small.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &small_b,
+            &Some(small.clone()),
+            &DataUsageEntry {
+                objects: 1,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(&large, &Some(root.clone()), &DataUsageEntry::default());
+        cache.replace_hashed(
+            &large_a,
+            &Some(large.clone()),
+            &DataUsageEntry {
+                objects: 10,
+                ..Default::default()
+            },
+        );
+        cache.replace_hashed(
+            &large_b,
+            &Some(large.clone()),
+            &DataUsageEntry {
+                objects: 10,
+                ..Default::default()
+            },
+        );
+        (cache, root)
+    }
+
+    #[test]
+    fn test_add_collects_internal_nodes_as_compaction_candidates() {
+        let (cache, root) = build_test_tree();
+        let mut candidates = Vec::new();
+        add(&cache, &root, &mut candidates);
+
+        let mut paths: Vec<String> = candidates.iter().map(|l| l.path.key()).collect();
+        paths.sort();
+        assert_eq!(paths.len(), 2, "add() should find internal nodes with children");
+        assert!(paths.contains(&hash_path("bucket").key()));
+        assert!(paths.contains(&hash_path("bucket/b").key()));
+    }
+
+    #[test]
+    fn test_add_skips_leaf_node() {
+        let mut cache = DataUsageCache::default();
+        let h = hash_path("single-leaf");
+        cache.replace_hashed(
+            &h,
+            &None,
+            &DataUsageEntry {
+                objects: 5,
+                size: 50,
+                ..Default::default()
+            },
+        );
+
+        let mut candidates = Vec::new();
+        add(&cache, &h, &mut candidates);
+        assert!(candidates.is_empty(), "leaf node should not be a compaction candidate");
+    }
+
+    #[test]
+    fn test_reduce_children_of_compacts_internal_node() {
+        let (mut cache, root) = build_test_tree();
+        cache.reduce_children_of(&root, 2, false);
+
+        let entry_c2 = cache.find("bucket/b").unwrap();
+        assert!(entry_c2.compacted, "internal node 'bucket/b' should be compacted");
+        let entry_c1 = cache.find("bucket/a").unwrap();
+        assert!(!entry_c1.compacted, "leaf 'bucket/a' should not be compacted");
+        assert!(cache.find("bucket/b/c").is_none(), "grandchild should be removed");
+    }
+
+    #[test]
+    fn test_reduce_children_of_usize_underflow_saturates() {
+        let (mut cache, root) = build_underflow_test_tree();
+
+        // total children=6, limit=5, remove=1. The smallest candidate removes
+        // two descendants, so plain subtraction would underflow and compact the
+        // next candidate too.
+        cache.reduce_children_of(&root, 5, false);
+
+        assert!(cache.find("bucket/small").is_some_and(|entry| entry.compacted));
+        assert!(cache.find("bucket/small/a").is_none());
+        assert!(cache.find("bucket/small/b").is_none());
+        assert!(cache.find("bucket/large").is_some_and(|entry| !entry.compacted));
+        assert!(cache.find("bucket/large/a").is_some());
+        assert!(cache.find("bucket/large/b").is_some());
+    }
 }

crates/data-usage/src/lib.rs

@@ -12,4 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-pub mod workers;
+pub mod data_usage;
+
+pub use data_usage::*;

crates/e2e_test/Cargo.toml

@@ -31,7 +31,7 @@ sftp = []
 [dependencies]
 rustfs-config = { workspace = true, features = ["constants"] }
 rustfs-ecstore.workspace = true
-rustfs-common.workspace = true
+rustfs-data-usage.workspace = true
 rustfs-rio.workspace = true
 flatbuffers.workspace = true
 futures.workspace = true

crates/e2e_test/src/admin_timeout_regression_test.rs

@@ -0,0 +1,208 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestClusterEnvironment, init_logging, local_http_client};
+use aws_sdk_s3::primitives::ByteStream;
+use http::header::HOST;
+use reqwest::StatusCode;
+use rustfs_madmin::{ITEM_OFFLINE, InfoMessage, StorageInfo};
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use serial_test::serial;
+use std::error::Error;
+use std::process::Command;
+use tokio::time::{Duration, sleep, timeout};
+use uuid::Uuid;
+
+const BUCKET: &str = "issue-2525-admin-timeout";
+
+async fn signed_admin_get(
+    url: &str,
+    access_key: &str,
+    secret_key: &str,
+) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+    let request = http::Request::builder()
+        .method(http::Method::GET)
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
+        .body(Body::empty())?;
+    let signed = sign_v4(request, 0, access_key, secret_key, "", "us-east-1");
+
+    let mut builder = local_http_client().get(url);
+    for (name, value) in signed.headers() {
+        builder = builder.header(name, value);
+    }
+
+    Ok(builder.send().await?)
+}
+
+async fn fetch_info(cluster: &RustFSTestClusterEnvironment) -> Result<InfoMessage, Box<dyn Error + Send + Sync>> {
+    let url = format!("{}/rustfs/admin/v3/info", cluster.nodes[0].url);
+    let response = signed_admin_get(&url, &cluster.access_key, &cluster.secret_key).await?;
+    parse_json_response(response, "cluster info").await
+}
+
+async fn fetch_storage_info(cluster: &RustFSTestClusterEnvironment) -> Result<StorageInfo, Box<dyn Error + Send + Sync>> {
+    let url = format!("{}/rustfs/admin/v3/storageinfo", cluster.nodes[0].url);
+    let response = signed_admin_get(&url, &cluster.access_key, &cluster.secret_key).await?;
+    parse_json_response(response, "storage info").await
+}
+
+async fn parse_json_response<T: serde::de::DeserializeOwned>(
+    response: reqwest::Response,
+    label: &str,
+) -> Result<T, Box<dyn Error + Send + Sync>> {
+    let status = response.status();
+    let body = response.bytes().await?;
+    if status != StatusCode::OK {
+        return Err(format!("{label} request failed: {status} {}", String::from_utf8_lossy(body.as_ref())).into());
+    }
+
+    Ok(serde_json::from_slice(&body)?)
+}
+
+fn signal_process(pid: u32, signal: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
+    let output = Command::new("kill").arg(format!("-{signal}")).arg(pid.to_string()).output()?;
+    if output.status.success() {
+        return Ok(());
+    }
+
+    Err(format!("kill -{signal} {pid} failed: {}", String::from_utf8_lossy(&output.stderr)).into())
+}
+
+fn offline_server_count(info: &InfoMessage) -> usize {
+    info.servers
+        .as_ref()
+        .map(|servers| servers.iter().filter(|server| server.state == ITEM_OFFLINE).count())
+        .unwrap_or_default()
+}
+
+#[tokio::test(flavor = "multi_thread")]
+#[serial]
+async fn test_single_admin_timeout_does_not_immediately_mark_peer_offline() -> Result<(), Box<dyn Error + Send + Sync>> {
+    init_logging();
+
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let warm_info = fetch_info(&cluster).await?;
+    assert_eq!(
+        warm_info
+            .backend
+            .as_ref()
+            .map(|backend| backend.offline_disks)
+            .unwrap_or_default(),
+        0,
+        "cluster should start with zero offline disks"
+    );
+
+    let warm_storage = fetch_storage_info(&cluster).await?;
+    assert_eq!(
+        warm_storage.backend.offline_disks.sum(),
+        0,
+        "storage info should start with zero offline disks"
+    );
+
+    let suspended_pid = cluster.nodes[1]
+        .process
+        .as_ref()
+        .ok_or("cluster node 1 process missing")?
+        .id();
+    signal_process(suspended_pid, "STOP")?;
+
+    let resume = tokio::spawn(async move {
+        sleep(Duration::from_secs(6)).await;
+        signal_process(suspended_pid, "CONT")
+    });
+
+    let client = cluster.create_s3_client(0)?;
+    let key = format!("issue-2525/{}", Uuid::new_v4().simple());
+    let payload = b"admin timeout regression payload".to_vec();
+
+    let during = timeout(Duration::from_secs(20), async {
+        tokio::join!(fetch_info(&cluster), fetch_storage_info(&cluster), async {
+            client
+                .put_object()
+                .bucket(BUCKET)
+                .key(&key)
+                .body(ByteStream::from(payload.clone()))
+                .send()
+                .await?;
+            Ok::<(), Box<dyn Error + Send + Sync>>(())
+        })
+    })
+    .await;
+
+    resume.await??;
+
+    let (info_during, storage_during, put_result) =
+        during.map_err(|_| "timed out while waiting for admin queries during suspended peer")?;
+    let info_during = info_during?;
+    let storage_during = storage_during?;
+    put_result?;
+
+    assert_eq!(
+        info_during
+            .backend
+            .as_ref()
+            .map(|backend| backend.offline_disks)
+            .unwrap_or_default(),
+        0,
+        "single admin timeout must not synthesize offline disks in /info"
+    );
+    assert_eq!(
+        offline_server_count(&info_during),
+        0,
+        "single admin timeout must not mark any server offline in /info"
+    );
+    assert_eq!(
+        storage_during.backend.offline_disks.sum(),
+        0,
+        "single admin timeout must not synthesize offline disks in /storageinfo"
+    );
+    assert!(
+        storage_during.disks.iter().all(|disk| disk.state != ITEM_OFFLINE),
+        "single admin timeout must not synthesize offline disk entries in /storageinfo"
+    );
+
+    let recovered_info = fetch_info(&cluster).await?;
+    assert_eq!(
+        recovered_info
+            .backend
+            .as_ref()
+            .map(|backend| backend.offline_disks)
+            .unwrap_or_default(),
+        0,
+        "offline disk count should stay at zero after the peer resumes"
+    );
+
+    let body = client
+        .get_object()
+        .bucket(BUCKET)
+        .key(&key)
+        .send()
+        .await?
+        .body
+        .collect()
+        .await?
+        .into_bytes();
+    assert_eq!(body.as_ref(), b"admin timeout regression payload");
+
+    Ok(())
+}

crates/e2e_test/src/data_usage_test.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use aws_sdk_s3::primitives::ByteStream;
-use rustfs_common::data_usage::DataUsageInfo;
+use rustfs_data_usage::DataUsageInfo;
 use serial_test::serial;
 
 use crate::common::{RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -0,0 +1,331 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Erasure-set healing regression tests.
+
+#[cfg(test)]
+mod tests {
+    use crate::common::{RustFSTestEnvironment, execute_awscurl, init_logging};
+    use aws_sdk_s3::primitives::ByteStream;
+    use serial_test::serial;
+    use std::collections::HashSet;
+    use std::path::{Path, PathBuf};
+    use tokio::time::{Duration, sleep};
+    use tracing::info;
+
+    fn has_file_under(path: &Path) -> bool {
+        let Ok(entries) = std::fs::read_dir(path) else {
+            return false;
+        };
+
+        for entry in entries.filter_map(Result::ok) {
+            let path = entry.path();
+            if path.is_dir() {
+                if has_file_under(&path) {
+                    return true;
+                }
+            } else {
+                return true;
+            }
+        }
+
+        false
+    }
+
+    fn object_metadata_exists_on_disk(disk: &Path, bucket: &str, key: &str) -> bool {
+        disk.join(bucket).join(key).join("xl.meta").is_file()
+    }
+
+    async fn assert_object_body(env: &RustFSTestEnvironment, bucket: &str, key: &str, expected: &[u8]) {
+        let client = env.create_s3_client();
+        let response = client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .expect("GET should succeed during/after heal");
+        let body = response.body.collect().await.expect("GET body should collect").into_bytes();
+        assert_eq!(body.as_ref(), expected, "object body changed for {key}");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_auto_heal_rebuilds_runtime_wiped_disk_without_restart() {
+        init_logging();
+        info!("Issue #1533: auto heal should rebuild a runtime-wiped disk in a 4-disk single-node erasure set without restart");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        let root = PathBuf::from(env.temp_dir.clone());
+        let disk0 = root.join("disk0");
+        let disk1 = root.join("disk1");
+        let disk2 = root.join("disk2");
+        let disk3 = root.join("disk3");
+        for disk in [&disk0, &disk1, &disk2, &disk3] {
+            std::fs::create_dir_all(disk).expect("disk directory should be created");
+        }
+
+        env.temp_dir = disk3.to_string_lossy().to_string();
+        let disk0_arg = disk0.to_string_lossy().to_string();
+        let disk1_arg = disk1.to_string_lossy().to_string();
+        let disk2_arg = disk2.to_string_lossy().to_string();
+        env.start_rustfs_server_with_env(
+            vec![disk0_arg.as_str(), disk1_arg.as_str(), disk2_arg.as_str()],
+            &[
+                ("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true"),
+                ("RUSTFS_HEAL_INTERVAL_SECS", "10"),
+            ],
+        )
+        .await
+        .expect("Failed to start 4-disk RustFS");
+        env.temp_dir = root.to_string_lossy().to_string();
+
+        let client = env.create_s3_client();
+        let bucket = "heal-runtime-wiped-disk";
+        let heal_timeout_secs = std::env::var("RUSTFS_AUTO_HEAL_RUNTIME_WIPE_TIMEOUT_SECS")
+            .ok()
+            .and_then(|value| value.parse::<u64>().ok())
+            .unwrap_or(45);
+
+        let objects: Vec<(String, Vec<u8>, &'static str)> = vec![
+            (
+                "runtime/alpha.txt".to_string(),
+                b"alpha payload for runtime wipe heal".to_vec(),
+                "text/plain; charset=utf-8",
+            ),
+            ("runtime/beta.bin".to_string(), (0..=127).collect::<Vec<u8>>(), "application/octet-stream"),
+            (
+                "runtime/dir/emoji-free-name.json".to_string(),
+                br#"{"status":"runtime-heal"}"#.to_vec(),
+                "application/json",
+            ),
+            (
+                "runtime/dir/gamma.txt".to_string(),
+                b"gamma payload for runtime wipe heal".to_vec(),
+                "text/plain; charset=utf-8",
+            ),
+        ];
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("bucket create should succeed");
+        for (key, body, content_type) in &objects {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .content_type(*content_type)
+                .body(ByteStream::from(body.clone()))
+                .send()
+                .await
+                .expect("PUT should succeed");
+        }
+
+        for (key, body, _) in &objects {
+            assert_object_body(&env, bucket, key, body).await;
+            assert!(
+                object_metadata_exists_on_disk(&disk0, bucket, key),
+                "disk0 should contain xl.meta before runtime wipe for {key}"
+            );
+        }
+
+        std::fs::remove_dir_all(&disk0).expect("disk0 wipe should succeed while server is running");
+        std::fs::create_dir_all(&disk0).expect("disk0 should be recreated empty while server is running");
+        assert!(!has_file_under(&disk0), "disk0 must be empty immediately after runtime wipe");
+
+        let mut remaining_rebuild_keys: HashSet<String> = objects.iter().map(|(key, _, _)| key.clone()).collect();
+        for _ in 0..heal_timeout_secs {
+            for (key, body, _) in &objects {
+                assert_object_body(&env, bucket, key, body).await;
+            }
+
+            if !remaining_rebuild_keys.is_empty() {
+                let mut rebuilt = Vec::new();
+                for key in &remaining_rebuild_keys {
+                    if object_metadata_exists_on_disk(&disk0, bucket, key) {
+                        rebuilt.push(key.clone());
+                    }
+                }
+                for key in rebuilt {
+                    let _ = remaining_rebuild_keys.remove(&key);
+                }
+            }
+
+            if remaining_rebuild_keys.is_empty() {
+                assert!(
+                    disk0.join(".rustfs.sys").join("format.json").is_file(),
+                    "runtime-wiped disk should have format.json restored by auto heal"
+                );
+                return;
+            }
+
+            sleep(Duration::from_secs(1)).await;
+        }
+
+        panic!("auto heal did not rebuild all files on the runtime-wiped disk within timeout");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn test_admin_deep_heal_rebuilds_cleared_disk_in_single_node_erasure_set() {
+        init_logging();
+        info!("Discussion #2964: admin deep heal should rebuild a wiped disk in a 4-disk single-node erasure set");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        let root = PathBuf::from(env.temp_dir.clone());
+        let disk0 = root.join("disk0");
+        let disk1 = root.join("disk1");
+        let disk2 = root.join("disk2");
+        let disk3 = root.join("disk3");
+        for disk in [&disk0, &disk1, &disk2, &disk3] {
+            std::fs::create_dir_all(disk).expect("disk directory should be created");
+        }
+
+        // The test helper always appends env.temp_dir as the final storage path.
+        // Point it at disk3 and pass the other three disks explicitly.
+        env.temp_dir = disk3.to_string_lossy().to_string();
+        let disk0_arg = disk0.to_string_lossy().to_string();
+        let disk1_arg = disk1.to_string_lossy().to_string();
+        let disk2_arg = disk2.to_string_lossy().to_string();
+        env.start_rustfs_server_with_env(
+            vec![disk0_arg.as_str(), disk1_arg.as_str(), disk2_arg.as_str()],
+            &[("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true")],
+        )
+        .await
+        .expect("Failed to start 4-disk RustFS");
+
+        let client = env.create_s3_client();
+        let bucket = "heal-cleared-disk-regression";
+        let target_object_count = std::env::var("RUSTFS_HEAL_REBUILD_OBJECT_COUNT")
+            .ok()
+            .and_then(|value| value.parse::<usize>().ok())
+            .unwrap_or(4)
+            .max(4);
+        let heal_timeout_secs = std::env::var("RUSTFS_HEAL_REBUILD_TIMEOUT_SECS")
+            .ok()
+            .and_then(|value| value.parse::<u64>().ok())
+            .unwrap_or(60);
+
+        let mut objects: Vec<(String, Vec<u8>, &'static str)> = vec![
+            (
+                "中文/报告-0001.json".to_string(),
+                "{\"message\":\"hello 中文\"}".as_bytes().to_vec(),
+                "application/json",
+            ),
+            (
+                "english/images/photo-0002.jpg".to_string(),
+                vec![0xff, 0xd8, 0xff, 0x00, 0x42, 0x24],
+                "image/jpeg",
+            ),
+            (
+                "mixed/空 格 + symbols @#%.txt".to_string(),
+                b"text object with spaces and symbols".to_vec(),
+                "text/plain; charset=utf-8",
+            ),
+            (
+                "bin/archive-0004.bin".to_string(),
+                (0..=255).collect::<Vec<u8>>(),
+                "application/octet-stream",
+            ),
+        ];
+        for index in objects.len()..target_object_count {
+            objects.push((
+                format!("bulk/prefix-{}/object-{index:04}.txt", index % 17),
+                format!("bulk object {index}: heal regression payload").into_bytes(),
+                "text/plain; charset=utf-8",
+            ));
+        }
+
+        let object_keys = objects.iter().map(|(key, _, _)| key.clone()).collect::<Vec<_>>();
+        let mut remaining_rebuild_keys: HashSet<String> = object_keys.iter().cloned().collect();
+
+        client
+            .create_bucket()
+            .bucket(bucket)
+            .send()
+            .await
+            .expect("bucket create should succeed");
+        for (key, body, content_type) in &objects {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .content_type(*content_type)
+                .body(ByteStream::from(body.clone()))
+                .send()
+                .await
+                .expect("PUT should succeed");
+        }
+
+        assert!(has_file_under(&disk0), "disk0 should contain object shards before wipe");
+        env.stop_server();
+
+        std::fs::remove_dir_all(&disk0).expect("disk0 wipe should succeed");
+        std::fs::create_dir_all(&disk0).expect("disk0 should be recreated empty");
+        assert!(!has_file_under(&disk0), "disk0 must be empty before restart");
+
+        env.start_rustfs_server_with_env(
+            vec![disk0_arg.as_str(), disk1_arg.as_str(), disk2_arg.as_str()],
+            &[("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true")],
+        )
+        .await
+        .expect("Failed to restart 4-disk RustFS after disk wipe");
+        // The helper's Drop cleanup removes env.temp_dir. Reset it to the parent
+        // directory after server startup so all four disk directories are cleaned
+        // without manually deleting a path Drop will also try to remove.
+        env.temp_dir = root.to_string_lossy().to_string();
+
+        let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
+        let heal_url = format!("{}/rustfs/admin/v3/heal/{}?forceStart=true", env.url, bucket);
+        execute_awscurl(&heal_url, "POST", Some(heal_body), &env.access_key, &env.secret_key)
+            .await
+            .expect("admin deep heal should be accepted");
+
+        for _ in 0..heal_timeout_secs {
+            if !remaining_rebuild_keys.is_empty() {
+                let mut rebuilt = Vec::new();
+                for key in &remaining_rebuild_keys {
+                    if object_metadata_exists_on_disk(&disk0, bucket, key) {
+                        rebuilt.push(key.clone());
+                    }
+                }
+                for key in rebuilt {
+                    let _ = remaining_rebuild_keys.remove(&key);
+                }
+            }
+
+            if remaining_rebuild_keys.is_empty() {
+                for (key, body, _) in &objects {
+                    assert_object_body(&env, bucket, key, body).await;
+                }
+
+                env.stop_server();
+                for key in &object_keys {
+                    assert!(
+                        object_metadata_exists_on_disk(&disk0, bucket, key),
+                        "wiped disk should contain rebuilt xl.meta for {key}"
+                    );
+                }
+                return;
+            }
+
+            sleep(Duration::from_secs(1)).await;
+        }
+
+        panic!("admin deep heal did not rebuild all files on the wiped disk within timeout");
+    }
+}

crates/e2e_test/src/kms/README.md

@@ -191,7 +191,7 @@ RUST_LOG=debug cargo test test_local_kms_end_to_end -- --nocapture
 4. **Monitor ports**
    ```bash
    netstat -an | grep 9050
-   curl http://127.0.0.1:9050/minio/health/ready
+   curl http://127.0.0.1:9050/health/ready
    ```
 
 ## 📊 Coverage
@@ -244,9 +244,9 @@ Designed to run inside CI/CD pipelines:
 
 ## 📚 References
 
-- [KMS configuration guide](../../../../docs/kms/README.md)
-- [Dynamic configuration API](../../../../docs/kms/http-api.md)
-- [Troubleshooting](../../../../docs/kms/troubleshooting.md)
+- [KMS configuration types](../../../kms/src/config.rs)
+- [Dynamic configuration API handlers](../../../../rustfs/src/admin/handlers/kms_dynamic.rs)
+- [KMS management API handlers](../../../../rustfs/src/admin/handlers/kms_management.rs)
 
 ---
 

crates/e2e_test/src/kms/bucket_default_encryption_test.rs

@@ -520,3 +520,75 @@ async fn test_explicit_encryption_overrides_bucket_default() -> Result<(), Box<d
 
     Ok(())
 }
+
+/// Test 5: Setting SSE-KMS without a specific key ID should auto-populate the
+/// default KMS key ID so that GetBucketEncryption returns it (issue #3039).
+#[tokio::test]
+#[serial]
+async fn test_sse_kms_without_key_id_populates_default() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Testing SSE-KMS without explicit key ID populates default key");
+
+    let mut kms_env = LocalKMSTestEnvironment::new().await?;
+    let default_key_id = kms_env.start_rustfs_for_local_kms().await?;
+    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
+
+    let s3_client = kms_env.base_env.create_s3_client();
+    kms_env.base_env.create_test_bucket(TEST_BUCKET).await?;
+
+    // Set bucket encryption to SSE-KMS WITHOUT specifying a key ID
+    info!("Setting bucket default encryption to SSE-KMS without key ID");
+    let encryption_config = ServerSideEncryptionConfiguration::builder()
+        .rules(
+            ServerSideEncryptionRule::builder()
+                .apply_server_side_encryption_by_default(
+                    ServerSideEncryptionByDefault::builder()
+                        .sse_algorithm(ServerSideEncryption::AwsKms)
+                        .kms_master_key_id("")
+                        .build()
+                        .unwrap(),
+                )
+                .build(),
+        )
+        .build()
+        .unwrap();
+
+    s3_client
+        .put_bucket_encryption()
+        .bucket(TEST_BUCKET)
+        .server_side_encryption_configuration(encryption_config)
+        .send()
+        .await
+        .expect("Failed to set bucket SSE-KMS encryption without key ID");
+
+    // GetBucketEncryption should return the default KMS key ID
+    info!("Verifying GetBucketEncryption returns default KMS key ID");
+    let get_response = s3_client
+        .get_bucket_encryption()
+        .bucket(TEST_BUCKET)
+        .send()
+        .await
+        .expect("Failed to get bucket encryption");
+
+    let config = get_response.server_side_encryption_configuration();
+    let rule = config
+        .and_then(|c| c.rules().first())
+        .expect("Should have at least one encryption rule");
+    let by_default = rule
+        .apply_server_side_encryption_by_default()
+        .expect("Should have ApplyServerSideEncryptionByDefault");
+
+    assert_eq!(by_default.sse_algorithm(), &ServerSideEncryption::AwsKms, "Algorithm should be aws:kms");
+    assert!(
+        by_default.kms_master_key_id().is_some(),
+        "KMS key ID should be populated with the default key (was None)"
+    );
+    assert_eq!(
+        by_default.kms_master_key_id().unwrap(),
+        &default_key_id,
+        "KMS key ID should match the configured default key"
+    );
+
+    info!("Test passed: SSE-KMS without key ID correctly populates default key '{}'", default_key_id);
+    Ok(())
+}

crates/e2e_test/src/lib.rs

@@ -63,6 +63,10 @@ mod archive_download_integrity_test;
 #[cfg(test)]
 mod list_objects_v2_pagination_test;
 
+// Regression test for Issue #3107: mc mirror small-bucket listing must not time out.
+#[cfg(test)]
+mod mc_mirror_small_bucket_test;
+
 // Policy variables tests
 #[cfg(test)]
 mod policy;
@@ -114,6 +118,9 @@ mod head_object_range_test;
 #[cfg(test)]
 mod head_object_consistency_test;
 
+#[cfg(test)]
+mod heal_erasure_disk_rebuild_test;
+
 #[cfg(test)]
 mod copy_object_metadata_test;
 
@@ -139,4 +146,10 @@ mod replication_extension_test;
 #[cfg(test)]
 mod snowball_auto_extract_test;
 
+#[cfg(test)]
+mod namespace_lock_quorum_test;
+
+#[cfg(test)]
+mod admin_timeout_regression_test;
+
 pub mod tls_gen;

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -612,4 +612,421 @@ mod tests {
 
         env.stop_server();
     }
+
+    /// Regression test: delimiter listing must not produce false-positive truncation
+    /// when many raw keys collapse into a small number of CommonPrefixes.
+    ///
+    /// Scenario: 30 objects across 3 directories + 2 direct files under prefix.
+    /// With max_keys=1000, all 5 visible results (3 prefixes + 2 objects) fit in one
+    /// page, so IsTruncated must be false even though raw entry count is much larger.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_delimiter_collapsed_prefix_no_false_truncation() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 delimiter collapsed-prefix no false truncation");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-collapsed-prefix";
+        let prefix = "data/";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create objects under 3 subdirectories (10 each = 30 raw keys)
+        let dirs = ["alpha/", "beta/", "gamma/"];
+        let mut expected_keys = Vec::new();
+        for dir in &dirs {
+            for i in 0..10 {
+                let key = format!("{prefix}{dir}file{i:02}.txt");
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                expected_keys.push(key);
+            }
+        }
+        // Add 2 direct objects under prefix
+        for name in ["readme.txt", "config.json"] {
+            let key = format!("{prefix}{name}");
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(&key)
+                .body(ByteStream::from_static(b"x"))
+                .send()
+                .await
+                .expect("Failed to put object");
+            expected_keys.push(key);
+        }
+
+        // List with delimiter="/", max_keys large enough to cover all visible results
+        // Visible: 3 common prefixes + 2 direct objects = 5
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .prefix(prefix)
+            .delimiter("/")
+            .max_keys(1000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        let listed_keys: Vec<String> = output
+            .contents()
+            .iter()
+            .filter_map(|obj| obj.key())
+            .map(|k| k.to_string())
+            .collect();
+        let listed_prefixes: Vec<String> = output
+            .common_prefixes()
+            .iter()
+            .filter_map(|cp| cp.prefix())
+            .map(|p| p.to_string())
+            .collect();
+
+        // KEY ASSERTION: IsTruncated must be false because all visible results fit within max_keys
+        let is_truncated = output.is_truncated().unwrap_or(false);
+        assert!(
+            !is_truncated,
+            "BUG: IsTruncated should be false when all visible results ({} objects + {} prefixes = {}) fit within max_keys (1000)",
+            listed_keys.len(),
+            listed_prefixes.len(),
+            listed_keys.len() + listed_prefixes.len()
+        );
+
+        assert!(
+            output.next_continuation_token().is_none(),
+            "NextContinuationToken should be None when IsTruncated is false"
+        );
+
+        // All 32 objects must be covered (listed_keys + keys under listed_prefixes)
+        let total_raw_count = expected_keys.len();
+        let keys_under_prefixes: usize = listed_prefixes
+            .iter()
+            .map(|p| {
+                let dir = p.trim_end_matches('/');
+                expected_keys.iter().filter(|k| k.starts_with(&format!("{dir}/"))).count()
+            })
+            .sum();
+        let covered = listed_keys.len() + keys_under_prefixes;
+
+        assert_eq!(
+            covered,
+            total_raw_count,
+            "Collapsed-prefix listing must cover all {} objects, got {} (keys={}, under_prefixes={})",
+            total_raw_count,
+            covered,
+            listed_keys.len(),
+            keys_under_prefixes
+        );
+
+        info!(
+            "Collapsed-prefix test passed: {} objects covered ({} keys + {} prefixes), IsTruncated=false",
+            covered,
+            listed_keys.len(),
+            listed_prefixes.len()
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: delimiter listing pagination must traverse all keys
+    /// even when the visible result count per page is much smaller than the
+    /// raw entry count (due to CommonPrefix collapse).
+    ///
+    /// Scenario: 1000 objects across 100 directories, paginated with max_keys=50.
+    /// Each page returns up to 50 CommonPrefixes. The server must correctly set
+    /// IsTruncated and provide a valid continuation token across all pages.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_delimiter_small_page_traverses_all() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 delimiter small page traverses all keys");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-delimiter-small-page";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create 1000 objects: 100 directories with 10 files each
+        let mut all_keys = Vec::new();
+        let dirs: Vec<String> = (0..100).map(|i| format!("dir-{:03}/", i)).collect();
+        for dir in &dirs {
+            for i in 0..10 {
+                let key = format!("{dir}file{:02}.txt", i);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                all_keys.push(key);
+            }
+        }
+
+        // Paginate with delimiter="/" and max_keys=50
+        // Visible per page: up to 50 CommonPrefixes
+        let mut listed_keys = Vec::new();
+        let mut listed_prefixes = Vec::new();
+        let mut continuation_token: Option<String> = None;
+        let mut page_count = 0;
+        let mut last_page_is_truncated: bool;
+
+        loop {
+            let mut request = client.list_objects_v2().bucket(bucket).delimiter("/").max_keys(50);
+
+            if let Some(token) = continuation_token.take() {
+                request = request.continuation_token(token);
+            }
+
+            let output = request.send().await.expect("Failed to list objects");
+            last_page_is_truncated = output.is_truncated().unwrap_or(false);
+
+            for obj in output.contents() {
+                if let Some(key) = obj.key() {
+                    listed_keys.push(key.to_string());
+                }
+            }
+            for cp in output.common_prefixes() {
+                if let Some(p) = cp.prefix() {
+                    listed_prefixes.push(p.to_string());
+                }
+            }
+
+            page_count += 1;
+
+            if last_page_is_truncated {
+                continuation_token = output.next_continuation_token().map(|s| s.to_string());
+                assert!(
+                    continuation_token.is_some(),
+                    "BUG: NextContinuationToken must be present when IsTruncated is true"
+                );
+            } else {
+                break;
+            }
+
+            if page_count > 20 {
+                panic!("Too many pages, possible infinite loop in delimiter pagination");
+            }
+        }
+
+        // Last page must have IsTruncated=false
+        assert!(
+            !last_page_is_truncated,
+            "BUG: Last page must have IsTruncated=false after all results returned"
+        );
+
+        // Verify all objects are covered via listed prefixes
+        let keys_under_prefixes: HashSet<String> = all_keys
+            .iter()
+            .filter(|k| {
+                listed_prefixes
+                    .iter()
+                    .any(|p| k.starts_with(&format!("{}/", p.trim_end_matches('/'))))
+            })
+            .cloned()
+            .collect();
+        let listed_set: HashSet<String> = listed_keys.iter().cloned().collect();
+        let covered: HashSet<String> = listed_set.union(&keys_under_prefixes).cloned().collect();
+        let expected_set: HashSet<String> = all_keys.iter().cloned().collect();
+
+        assert_eq!(
+            covered,
+            expected_set,
+            "Delimiter pagination must cover all {} objects, missing: {:?}",
+            expected_set.difference(&covered).count(),
+            expected_set.difference(&covered)
+        );
+
+        info!(
+            "Delimiter small-page test passed: {} objects covered in {} pages",
+            covered.len(),
+            page_count
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: raw entries exceed MaxKeys but all collapse into fewer
+    /// visible CommonPrefixes — IsTruncated must be false because there are no
+    /// more visible results beyond the collapsed prefixes.
+    ///
+    /// Scenario: 10 directories with 200 files each = 2000 raw keys.
+    /// With MaxKeys=1000, the raw entry count exceeds MaxKeys (disk_has_more=true),
+    /// but after delimiter collapse only 10 CommonPrefixes are visible (10 < 1000).
+    /// IsTruncated must be false since there are no additional visible results.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_raw_exceeds_maxkeys_but_visible_below() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 raw > MaxKeys but visible < MaxKeys after collapse");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-raw-exceeds-visible-below";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // Create 10 directories × 200 files = 2000 raw keys
+        let dir_count = 10;
+        let files_per_dir = 200;
+        let mut all_keys = Vec::new();
+        for d in 0..dir_count {
+            let dir = format!("dir-{:02}/", d);
+            for f in 0..files_per_dir {
+                let key = format!("{}file{:03}.txt", dir, f);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+                all_keys.push(key);
+            }
+        }
+
+        // List with delimiter="/", max_keys=1000
+        // Visible: 10 CommonPrefixes (dir-00/ .. dir-09/) = 10 visible << 1000 MaxKeys
+        // But raw entry count (2000+1001 requested) exceeds MaxKeys
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .delimiter("/")
+            .max_keys(1000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        let listed_keys: Vec<String> = output
+            .contents()
+            .iter()
+            .filter_map(|obj| obj.key())
+            .map(|k| k.to_string())
+            .collect();
+        let listed_prefixes: Vec<String> = output
+            .common_prefixes()
+            .iter()
+            .filter_map(|cp| cp.prefix())
+            .map(|p| p.to_string())
+            .collect();
+
+        // KEY ASSERTION: IsTruncated must be false because visible results (10)
+        // are far below MaxKeys (1000), even though raw entries exceeded MaxKeys.
+        let is_truncated = output.is_truncated().unwrap_or(false);
+        assert!(
+            !is_truncated,
+            "BUG: IsTruncated should be false when visible results ({} objects + {} prefixes = {}) < MaxKeys (1000), even if raw entries exceeded MaxKeys",
+            listed_keys.len(),
+            listed_prefixes.len(),
+            listed_keys.len() + listed_prefixes.len()
+        );
+
+        assert!(
+            output.next_continuation_token().is_none(),
+            "NextContinuationToken should be None when IsTruncated is false"
+        );
+
+        // All objects must be covered by listed prefixes
+        assert_eq!(
+            listed_prefixes.len(),
+            dir_count,
+            "Expected {} prefixes, got {}",
+            dir_count,
+            listed_prefixes.len()
+        );
+        for d in 0..dir_count {
+            let prefix = format!("dir-{:02}/", d);
+            assert!(listed_prefixes.contains(&prefix), "Missing prefix: {}", prefix);
+        }
+
+        info!(
+            "Raw-exceeds-visible test passed: {} raw keys → {} prefixes, IsTruncated=false",
+            all_keys.len(),
+            listed_prefixes.len()
+        );
+
+        env.stop_server();
+    }
+
+    /// Regression test: pagination with MaxKeys exceeding S3 limit (1000) and delimiter.
+    /// The server caps MaxKeys to 1000. With delimiter="/", many raw keys collapse into
+    /// few CommonPrefixes. Visible results (prefixes) < capped MaxKeys → IsTruncated=false.
+    ///
+    /// This complements test_list_objects_v2_max_keys_above_limit_returns_token which
+    /// tests the non-delimiter case.
+    #[tokio::test]
+    #[serial]
+    async fn test_list_objects_v2_maxkeys_above_limit_with_delimiter() {
+        init_logging();
+        info!("Starting test: ListObjectsV2 MaxKeys above limit with delimiter");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client(&env);
+        let bucket = "test-maxkeys-limit-delimiter";
+
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        // 12 dirs × 100 files = 1200 raw keys
+        let dir_count = 12;
+        let files_per_dir = 100;
+        for d in 0..dir_count {
+            for f in 0..files_per_dir {
+                let key = format!("dir-{:02}/file{:03}.txt", d, f);
+                client
+                    .put_object()
+                    .bucket(bucket)
+                    .key(&key)
+                    .body(ByteStream::from_static(b"x"))
+                    .send()
+                    .await
+                    .expect("Failed to put object");
+            }
+        }
+
+        // With delimiter: 12 CommonPrefixes visible, all fit within capped 1000
+        let output = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .delimiter("/")
+            .max_keys(2000)
+            .send()
+            .await
+            .expect("Failed to list objects");
+
+        assert_eq!(
+            output.common_prefixes().len(),
+            dir_count,
+            "Expected {} prefixes, got {}",
+            dir_count,
+            output.common_prefixes().len()
+        );
+        assert_eq!(output.max_keys(), Some(1000));
+        // 12 visible < 1000 capped MaxKeys → not truncated
+        assert!(
+            !output.is_truncated().unwrap_or(false),
+            "BUG: IsTruncated should be false when visible ({}) < capped MaxKeys (1000)",
+            output.common_prefixes().len()
+        );
+
+        info!("MaxKeys above limit with delimiter test passed");
+
+        env.stop_server();
+    }
 }

crates/e2e_test/src/mc_mirror_small_bucket_test.rs

@@ -0,0 +1,124 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, RustFSTestEnvironment};
+use serial_test::serial;
+use std::path::Path;
+use std::process::Command;
+use std::time::Duration;
+use tokio::fs;
+use tokio::time::timeout;
+use tracing::info;
+
+const BUCKET: &str = "ddd";
+const OBJECT_COUNT: usize = 484;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+async fn create_issue_3107_fixture(root: &Path) -> TestResult {
+    for idx in 0..OBJECT_COUNT {
+        let object_path = root.join("ddd").join("requirements").join(format!("file-{idx:04}.txt"));
+        fs::create_dir_all(object_path.parent().expect("object parent should exist")).await?;
+        fs::write(object_path, format!("issue-3107 payload {idx}\n").repeat(128)).await?;
+    }
+
+    for dir in ["aaa", "ccc", "ccc-package"] {
+        let object_path = root.join(dir).join("marker.txt");
+        fs::create_dir_all(object_path.parent().expect("object parent should exist")).await?;
+        fs::write(object_path, dir.as_bytes()).await?;
+    }
+
+    Ok(())
+}
+
+fn mc_available() -> bool {
+    Command::new("mc")
+        .arg("--version")
+        .output()
+        .is_ok_and(|output| output.status.success())
+}
+
+fn run_mc(args: &[&str]) -> TestResult {
+    let output = Command::new("mc").args(args).output()?;
+    if !output.status.success() {
+        return Err(format!(
+            "mc {:?} failed: status={:?}, stdout={}, stderr={}",
+            args,
+            output.status.code(),
+            String::from_utf8_lossy(&output.stdout),
+            String::from_utf8_lossy(&output.stderr)
+        )
+        .into());
+    }
+    Ok(())
+}
+
+fn count_files(root: &Path) -> usize {
+    walkdir::WalkDir::new(root)
+        .into_iter()
+        .filter_map(Result::ok)
+        .filter(|entry| entry.file_type().is_file())
+        .count()
+}
+
+#[tokio::test]
+#[serial]
+async fn test_mc_mirror_small_bucket_completes_without_list_timeout() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting issue #3107 mc mirror regression test");
+    if !mc_available() {
+        info!("Skipping issue #3107 mc mirror regression test because mc is not installed");
+        return Ok(());
+    }
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    env.create_test_bucket(BUCKET).await?;
+
+    let alias = format!("rustfs-issue-3107-{}", uuid::Uuid::new_v4());
+    run_mc(&["alias", "set", &alias, &env.url, DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY])?;
+
+    let fixture = Path::new(&env.temp_dir).join("issue-3107-fixture");
+    let backup = Path::new(&env.temp_dir).join("issue-3107-backup");
+    fs::create_dir_all(&fixture).await?;
+    fs::create_dir_all(&backup).await?;
+    create_issue_3107_fixture(&fixture).await?;
+
+    let bucket_target = format!("{alias}/{BUCKET}");
+    let source_path = fixture.to_string_lossy().to_string();
+    run_mc(&["mirror", "--overwrite", &source_path, &bucket_target])?;
+
+    let backup_path = backup.join("ddd-backup");
+    let backup_path_string = backup_path.to_string_lossy().to_string();
+    timeout(
+        Duration::from_secs(20),
+        tokio::task::spawn_blocking({
+            let bucket_target = bucket_target.clone();
+            let backup_path_string = backup_path_string.clone();
+            move || run_mc(&["mirror", "--overwrite", &bucket_target, &backup_path_string])
+        }),
+    )
+    .await
+    .map_err(|_| "mc mirror timed out while listing/comparing a small bucket")???;
+
+    let mirrored_count = count_files(&backup_path);
+    assert_eq!(
+        mirrored_count,
+        OBJECT_COUNT + 3,
+        "mc mirror should copy every object from the issue #3107 small bucket fixture"
+    );
+
+    run_mc(&["alias", "remove", &alias])?;
+    Ok(())
+}

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -0,0 +1,234 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::RustFSTestClusterEnvironment;
+use aws_sdk_s3::Client;
+use aws_sdk_s3::error::SdkError;
+use bytes::Bytes;
+use serial_test::serial;
+use std::sync::Arc;
+use tokio::sync::Barrier;
+use tracing::{info, warn};
+
+const BUCKET: &str = "namespace-lock-quorum-bucket";
+const KEY: &str = "thumb/79/concurrent-overwrite.jpg";
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
+    client
+        .put_object()
+        .bucket(BUCKET)
+        .key(KEY)
+        .body(Bytes::from(payload).into())
+        .send()
+        .await
+        .map(|_| ())
+        .map_err(|err| format_s3_error(err, writer_id))
+}
+
+fn format_s3_error(err: SdkError<aws_sdk_s3::operation::put_object::PutObjectError>, writer_id: usize) -> String {
+    match err {
+        SdkError::ServiceError(service_err) => {
+            let err = service_err.err();
+            let code = err.meta().code().unwrap_or("<unknown>");
+            let message = err.meta().message().unwrap_or("<empty>");
+            format!("writer {writer_id} returned service error {code}: {message}")
+        }
+        other => format!("writer {writer_id} returned SDK error: {other:?}"),
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting namespace lock quorum regression test with auto cluster");
+
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    // Keep the regression focused on false quorum-loss errors, not ordinary lock
+    // wait exhaustion under a heavily contended same-key overwrite workload.
+    cluster.set_env("RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT", "20");
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let clients = cluster.create_all_clients()?;
+    let first_payload = b"initial object contents".to_vec();
+    put_object(clients[0].clone(), first_payload, 0).await?;
+
+    let writer_count = clients.len() * 2;
+    let barrier = Arc::new(Barrier::new(writer_count));
+    let mut handles = Vec::with_capacity(writer_count);
+
+    for writer_id in 0..writer_count {
+        let client = clients[writer_id % clients.len()].clone();
+        let barrier = barrier.clone();
+        let payload = format!("replacement payload from writer {writer_id:02}").into_bytes();
+        handles.push(tokio::spawn(async move {
+            barrier.wait().await;
+            put_object(client, payload, writer_id).await
+        }));
+    }
+
+    let mut failures = Vec::new();
+    for handle in handles {
+        match handle.await {
+            Ok(Ok(())) => {}
+            Ok(Err(err)) => failures.push(err),
+            Err(err) => failures.push(format!("writer task join failed: {err}")),
+        }
+    }
+
+    if !failures.is_empty() {
+        for failure in &failures {
+            warn!("concurrent overwrite failure: {}", failure);
+        }
+    }
+
+    assert!(
+        failures.is_empty(),
+        "concurrent overwrites must not surface namespace lock quorum failures: {failures:#?}"
+    );
+
+    let body = clients[0]
+        .get_object()
+        .bucket(BUCKET)
+        .key(KEY)
+        .send()
+        .await?
+        .body
+        .collect()
+        .await?
+        .into_bytes();
+    let body = std::str::from_utf8(&body)?;
+    assert!(
+        body.starts_with("replacement payload from writer "),
+        "final object body should be one of the successful overwrite payloads, got {body:?}"
+    );
+
+    clients[0].delete_object().bucket(BUCKET).key(KEY).send().await?;
+    Ok(())
+}
+
+/// Regression test: concurrent PUTs to the same key must return 503 (ServiceUnavailable)
+/// on lock contention, never 500 (InternalError).
+///
+/// Before the fix, `map_namespace_lock_error` wrapped lock timeout/conflict errors as
+/// `StorageError::other(...)` → `StorageError::Io(...)`, which fell through to
+/// `S3ErrorCode::InternalError` (500) in the error mapping.
+#[tokio::test]
+#[serial]
+async fn test_concurrent_put_same_key_never_returns_500() -> TestResult {
+    crate::common::init_logging();
+    info!("Starting concurrent PUT 500 regression test");
+
+    let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+    // Short lock timeout to trigger contention errors quickly
+    cluster.set_env("RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT", "3");
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let clients = cluster.create_all_clients()?;
+    let writer_count = clients.len() * 4; // 16 writers for heavy contention
+    let barrier = Arc::new(Barrier::new(writer_count));
+
+    // Seed initial object
+    let first_payload = b"initial object for 500 regression".to_vec();
+    put_object(clients[0].clone(), first_payload, 0).await?;
+
+    let err_500_count = Arc::new(std::sync::atomic::AtomicU64::new(0));
+    let err_503_count = Arc::new(std::sync::atomic::AtomicU64::new(0));
+    let unexpected_err_count = Arc::new(std::sync::atomic::AtomicU64::new(0));
+    let ok_count = Arc::new(std::sync::atomic::AtomicU64::new(0));
+
+    let mut handles = Vec::with_capacity(writer_count);
+    for writer_id in 0..writer_count {
+        let client = clients[writer_id % clients.len()].clone();
+        let barrier = barrier.clone();
+        let payload = format!("payload from writer {writer_id:02}").into_bytes();
+        let err_500 = err_500_count.clone();
+        let err_503 = err_503_count.clone();
+        let unexpected_err = unexpected_err_count.clone();
+        let ok = ok_count.clone();
+
+        handles.push(tokio::spawn(async move {
+            barrier.wait().await;
+            match client
+                .put_object()
+                .bucket(BUCKET)
+                .key(KEY)
+                .body(Bytes::from(payload).into())
+                .send()
+                .await
+            {
+                Ok(_) => {
+                    ok.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                }
+                Err(err) => match &err {
+                    SdkError::ServiceError(service_err) => {
+                        let code = service_err.err().meta().code().unwrap_or("<unknown>");
+                        if code == "500" || code == "InternalError" {
+                            err_500.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                            warn!("writer {writer_id} returned 500: {code}");
+                        } else if code == "503" || code == "ServiceUnavailable" {
+                            err_503.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                        } else {
+                            unexpected_err.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                            warn!("writer {writer_id} returned unexpected error: {code}");
+                        }
+                    }
+                    other => {
+                        unexpected_err.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+                        warn!("writer {writer_id} returned SDK error: {other:?}");
+                    }
+                },
+            }
+        }));
+    }
+
+    for handle in handles {
+        if let Err(err) = handle.await {
+            unexpected_err_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+            warn!("writer task join failed: {err}");
+        }
+    }
+
+    let ok = ok_count.load(std::sync::atomic::Ordering::Relaxed);
+    let err_503 = err_503_count.load(std::sync::atomic::Ordering::Relaxed);
+    let err_500 = err_500_count.load(std::sync::atomic::Ordering::Relaxed);
+    let unexpected_err = unexpected_err_count.load(std::sync::atomic::Ordering::Relaxed);
+    let total = ok + err_503 + err_500 + unexpected_err;
+
+    info!("Concurrent PUT 500 regression: total={total}, ok={ok}, 503={err_503}, 500={err_500}, unexpected={unexpected_err}");
+
+    assert_eq!(
+        total, writer_count as u64,
+        "every concurrent PUT writer must be classified as success, 503, 500, or unexpected error"
+    );
+
+    assert_eq!(
+        unexpected_err, 0,
+        "Concurrent PUTs to the same key must only succeed or return 503. \
+         Got {unexpected_err} unexpected errors out of {total} requests. 503 count: {err_503}, ok: {ok}"
+    );
+
+    assert_eq!(
+        err_500, 0,
+        "Concurrent PUTs to the same key must NEVER return 500 InternalError. \
+         Got {err_500} out of {total} requests. 503 count: {err_503}, ok: {ok}"
+    );
+
+    clients[0].delete_object().bucket(BUCKET).key(KEY).send().await?;
+    Ok(())
+}

crates/e2e_test/src/protocols/sftp_compliance.rs

@@ -39,7 +39,7 @@
 use crate::protocols::sftp_compliance_tests::{
     cmptst_01, cmptst_02, cmptst_03, cmptst_04, cmptst_05, cmptst_06, cmptst_07, cmptst_08, cmptst_09, cmptst_10, cmptst_11,
     cmptst_12, cmptst_13, cmptst_14, cmptst_15, cmptst_16, cmptst_17, cmptst_18, cmptst_19, cmptst_20, cmptst_21, cmptst_22,
-    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, spawn_compliance_rustfs,
+    cmptst_23, cmptst_27, cmptst_28, cmptst_29, cmptst_32, cmptst_33, cmptst_34, spawn_compliance_rustfs,
 };
 #[cfg(target_os = "linux")]
 use crate::protocols::sftp_compliance_tests::{cmptst_24, cmptst_25, cmptst_26};
@@ -96,6 +96,15 @@ pub async fn test_sftp_compliance_suite() -> Result<()> {
         cmptst_13::run_implicit_dir_round_trip(&sftp).await?;
         cmptst_14::run_winscp_setstat_shape_on_handle(&sftp).await?;
 
+        // CMPTST-34 cross-checks the SFTP streaming-multipart write
+        // path against the S3 layer. The OPEN-time FileAttributes must
+        // reach the finalised object as x-amz-meta-* user metadata
+        // through the CreateMultipartUpload input field. The S3 client
+        // connects to the same rustfs process this suite already drives.
+        let s3 = build_test_s3_client(&format!("http://{COMPLIANCE_RW_S3_ADDRESS}"));
+        wait_for_s3_ready(&s3, 30).await?;
+        cmptst_34::run_open_attrs_round_trip_multipart(&sftp, &s3).await?;
+
         drop(sftp);
         session.disconnect(russh::Disconnect::ByApplication, "", "en").await?;
         info!("SFTP compliance suite passed");