chore(release): prepare 1.0.0-beta.10 (#4946)

Issue #4810's production incident showed the drive timeout knobs are
undiscoverable from symptoms: a listing that outruns the walk budget
gives operators no signal pointing at RUSTFS_DRIVE_* tuning. The knobs
existed only as code constants in crates/config/src/constants/drive.rs
with no operator-facing documentation.

Add docs/operations/drive-timeout-tuning.md covering the per-operation
drive timeout knobs, resolution precedence, the high_latency profile,
and a dedicated section on listing truncation and the walk stall
budget - including the correction that since the stall-budget rework
the foreground listing path is governed by
RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, not the total-timeout knob
the issue suggested. Link the guide from the README.

Ref #4810

.agents/skills/rustfs-release-publish/SKILL.md

@@ -0,0 +1,169 @@
+---
+name: rustfs-release-publish
+description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
+---
+# RustFS Release Publish (preview-validated pipeline)
+
+This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
+
+Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
+
+Pipeline shape:
+
+```
+bump version files to <target> (final version, ONE commit) -> merge
+  -> tag <preview-tag> at that commit -> CI green
+  -> verify release artifacts -> run binary locally + console checks
+  -> validate with latest rc client
+  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
+```
+
+On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
+
+## Required inputs
+
+- Final target version, for example `1.0.0-beta.10`.
+- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
+
+If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
+
+## Semver gate — confirm the target version before touching anything
+
+Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
+
+```
+1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
+```
+
+Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
+
+Rules:
+
+- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
+- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
+- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
+
+## Preview tag naming
+
+- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
+- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
+
+## Hard rules
+
+- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
+- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
+- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
+- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
+- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
+- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
+
+## Phase 0 — Preflight
+
+- `git status --short` clean; `git fetch origin main --tags`.
+- `gh auth status` works; confirm you can view `gh release list -L 3`.
+- Confirm the exact final target version with the user if not explicit.
+
+## Phase 1 — Version bump to the final target (once)
+
+- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
+- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
+- Get the PR merged into main. Record the resulting main commit:
+
+```bash
+git fetch origin main
+PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
+```
+
+`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
+
+## Phase 2 — Publish the preview tag
+
+```bash
+git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
+git push origin "<preview-tag>"
+```
+
+Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
+
+On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
+
+## Phase 3 — CI and artifact verification
+
+- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
+- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
+  - `isPrerelease` must be `true`.
+  - Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
+- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
+- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
+
+## Phase 4 — Run the artifact locally, verify the console
+
+Work inside the session scratchpad directory; never leave stray data dirs.
+
+```bash
+gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
+cd "$SCRATCH" && unzip -o rustfs-*.zip
+./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
+mkdir -p data
+RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
+```
+
+Defaults: S3 endpoint `:9000`, embedded console `:9001`.
+
+Checks (all must pass):
+
+- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
+- `curl -fsS http://localhost:9000/health/ready` returns ready.
+- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
+- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
+
+## Phase 5 — Validate with the latest rc client
+
+`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
+
+- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
+- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
+
+```bash
+rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
+rc ls preview/
+rc mb preview/rel-check
+rc cp <local-file> preview/rel-check/
+rc stat preview/rel-check/<file>
+rc cat preview/rel-check/<file>          # matches source
+rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
+rc cp -r <local-dir>/ preview/rel-check/dir/
+rc find preview/rel-check --name "*"
+rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
+rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
+rc rb preview/rel-check
+rc admin user list preview/
+rc admin user add preview/ relcheckuser relchecksecret12
+rc admin user remove preview/ relcheckuser
+rc alias remove preview
+```
+
+- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
+
+## Phase 6 — Publish the final tag on the validated commit
+
+No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
+
+```bash
+git fetch origin --tags
+git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
+git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
+git push origin "<target>"
+```
+
+- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
+- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
+- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
+
+## Output contract
+
+Always report:
+
+- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
+- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
+- Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -18,6 +18,8 @@ Validated baseline: release pattern used in PR `#2957`.
 
 If target version is missing or ambiguous, stop and ask before editing.
 
+Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
+
 ## Read before editing
 
 - `AGENTS.md` (root and nearest path-specific files).

.config/nextest.toml

@@ -138,6 +138,10 @@ test-group = 'ecstore-serial-flaky'
 # Each e2e test spawns its own rustfs server on a random port with an isolated
 # temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
 #
+# Replication failure harness (backlog#1147 repl-8): the first clause admits
+# its four in-process fake-target self-tests. They bind random loopback ports,
+# use no external service, and finish in under a second.
+#
 # Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
 # FAST bucket-replication tests from replication_extension_test — the
 # target-registration / replication-check / list / remove / delete admin paths
@@ -188,7 +192,7 @@ test-group = 'ecstore-serial-flaky'
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud)_test::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud)_test::|^fake_s3_target::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
@@ -259,8 +263,8 @@ path = "junit.xml"
 # The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
 #   * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed
 #     ports; they join a scheduled lane once ci-6 randomises the ports (ci-7).
-#   * the 6 cluster suites that spin up a RustFSTestClusterEnvironment
-#     (cluster_concurrency, stale_multipart_cleanup_cluster,
+#   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
+#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
 #     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
 #     object_lambda) — too heavy for the merge budget; they run in ci-7's
 #     nightly 4-node lane.
@@ -297,7 +301,7 @@ path = "junit.xml"
 default-filter = """
     package(e2e_test)
     & !test(/^protocols::/)
-    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
+    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
     & !test(/^replication_extension_test::/)
     & !test(/^multipart_auth_test::test_signed_put_object_extract_(accepts_compat_header|expands_tar_entries_with_prefix_headers|expands_tar_gz_archive|expands_tbz2_archive|expands_tgz_archive|expands_txz_archive|expands_tzst_archive|normalizes_prefix_header_value|preserves_directory_markers_by_default|preserves_object_lock_legal_hold|preserves_object_lock_retention|preserves_pax_metadata_and_version_id|preserves_request_metadata_on_extracted_objects|preserves_sse_c|preserves_sse_s3_and_redirect|preserves_storage_class|uses_bucket_default_sse_s3)$/)
     & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(prefers_exact_minio_prefix_over_suffix_fallback|supports_minio_prefix_and_directory_markers)$/)

.github/workflows/build.yml

@@ -190,7 +190,6 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"sm-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"sm-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-26-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -290,6 +289,7 @@ jobs:
             local console_url
             local console_sha256
             local curl_auth_args=()
+            console_tag=""
 
             if [[ "${{ matrix.platform }}" == "windows" ]]; then
               curl_bin="curl.exe"
@@ -312,7 +312,7 @@ jobs:
             "$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
               -o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
 
-            read -r console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
+            read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
           import json
           import re
           import sys
@@ -320,6 +320,8 @@ jobs:
           with open(sys.argv[1], encoding="utf-8") as handle:
               release = json.load(handle)
 
+          tag = release.get("tag_name", "") or "unknown"
+
           for asset in release.get("assets", []):
               name = asset.get("name", "")
               digest = asset.get("digest", "")
@@ -328,7 +330,7 @@ jobs:
                   sha256 = digest.split(":", 1)[1]
                   if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
                       raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
-                  sys.stdout.buffer.write(f"{url} {sha256}\n".encode("utf-8"))
+                  sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
                   break
           else:
               raise SystemExit("no console zip asset with sha256 digest found")
@@ -340,6 +342,8 @@ jobs:
               return 1
             fi
 
+            echo "Console release: ${console_tag}"
+            echo "Downloading console asset: ${console_url}"
             "$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
             verify_sha256 "$console_sha256" console.zip || return 2
             unzip -o console.zip -d ./rustfs/static || return 2
@@ -352,12 +356,19 @@ jobs:
             rm -f console.zip console-release.json
             if [[ "$status" -eq 2 ]]; then
               echo "Console asset integrity verification failed" >&2
-              exit 1
             fi
-            echo "Warning: Failed to download verified console assets, continuing without them"
-            echo "// Static assets not available" > ./rustfs/static/empty.txt
+            echo "Failed to download verified console assets" >&2
+            exit 1
           fi
 
+          if [[ ! -s ./rustfs/static/index.html ]]; then
+            echo "Console asset archive is missing static/index.html" >&2
+            exit 1
+          fi
+
+          asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
+          echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
+
       - name: Build RustFS
         shell: bash
         run: |
@@ -559,6 +570,60 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
+      - name: Verify packaged console
+        if: matrix.target == 'x86_64-unknown-linux-gnu'
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          find_free_port() {
+            python3 - <<'PY'
+          import socket
+
+          with socket.socket() as sock:
+              sock.bind(("127.0.0.1", 0))
+              print(sock.getsockname()[1])
+          PY
+          }
+
+          package_dir="$(mktemp -d)"
+          data_dir="$(mktemp -d)"
+          log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
+          server_pid=""
+          trap '
+            if [[ -n "${server_pid:-}" ]]; then
+              kill "$server_pid" 2>/dev/null || true
+              wait "$server_pid" 2>/dev/null || true
+            fi
+            rm -rf "$package_dir" "$data_dir"
+          ' EXIT
+
+          unzip -q "${{ steps.package.outputs.package_file }}" -d "$package_dir"
+          api_port="$(find_free_port)"
+          console_port="$(find_free_port)"
+          console_url="http://127.0.0.1:${console_port}/rustfs/console/"
+
+          "$package_dir/rustfs" server \
+            --address "127.0.0.1:${api_port}" \
+            --console-enable \
+            --console-address "127.0.0.1:${console_port}" \
+            --access-key console-smoke \
+            --secret-key console-smoke-secret \
+            "$data_dir" >"$log_file" 2>&1 &
+          server_pid=$!
+
+          for _ in {1..40}; do
+            response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
+            if [[ "$response" == 200\ text/html* ]]; then
+              exit 0
+            fi
+            sleep 0.25
+          done
+
+          echo "Console endpoint did not return 200 text/html: $response" >&2
+          cat "$log_file"
+          exit 1
+
       - name: Upload to GitHub artifacts
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
@@ -849,12 +914,14 @@ jobs:
 
           echo "✅ All assets uploaded successfully"
 
-  # Update latest.json for stable releases only: prerelease tags (alpha/beta/
-  # rc) must never overwrite the stable version pointer.
+  # Update latest.json for every release tag (stable and prerelease): the
+  # project currently ships prerelease tags only, so gating this to stable
+  # left the version pointer permanently stale. release_type records whether
+  # the pointed-to version is a prerelease.
   update-latest-version:
     name: Update Latest Version
     needs: [ build-check, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type == 'release'
+    if: startsWith(github.ref, 'refs/tags/')
     runs-on: ubuntu-latest
     steps:
       - name: Update latest.json
@@ -873,6 +940,12 @@ jobs:
           VERSION="${{ needs.build-check.outputs.version }}"
           TAG="${{ needs.build-check.outputs.version }}"
 
+          if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
+            RELEASE_TYPE="prerelease"
+          else
+            RELEASE_TYPE="stable"
+          fi
+
           # Install ossutil
           OSSUTIL_VERSION="2.1.1"
           OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
@@ -893,7 +966,7 @@ jobs:
             "version": "${VERSION}",
             "tag": "${TAG}",
             "release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
-            "release_type": "stable",
+            "release_type": "${RELEASE_TYPE}",
             "download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
           }
           EOF
@@ -901,7 +974,7 @@ jobs:
           # Upload to OSS
           "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
 
-          echo "✅ Updated latest.json for stable release $VERSION"
+          echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
 
   # Publish release (remove draft status)
   publish-release:

.github/workflows/ci.yml

@@ -221,15 +221,29 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      # Three scanner tests fail on main independently of this lane (restore of
-      # a transitioned multipart object, noncurrent transition/expiry after an
-      # immediate compensation transition); they keep #[ignore] with a backlog
-      # reference and are excluded here by name until fixed (rustfs/backlog#1148).
+      # The #4877 restore self-deadlock is fixed in this PR, which re-enabled
+      # test_multipart_restore_preserves_parts_and_etag. The remaining exclusions
+      # each hit a DIFFERENT, independent issue (all tracked under
+      # rustfs/backlog#1148; they keep #[ignore] with a backlog reference):
+      #   - test_noncurrent_{expiry,transition}_still_works_after_immediate_compensation_transition:
+      #     noncurrent transition/expiry after an immediate compensation transition.
+      #   - test_transition_and_restore_flows: transition metadata is missing on
+      #     one drive (assert_transition_meta_consistent: "missing xl.meta ... on
+      #     disk2") - an EC metadata-distribution issue, not the restore lock.
+      #   - test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore:
+      #     DeleteRestoredAction sets opts.transition.expire_restored, but no
+      #     delete path reads that flag, so cleanup deletes the whole object
+      #     instead of only the local restored copy (ObjectNotFound afterwards).
+      #     The expire_restored delete semantics are unimplemented.
+      #   - restore_object_usecase_reports_ongoing_conflict_and_completion: asserts
+      #     a concurrent get_object_info observes ongoing-request=true mid-restore,
+      #     which #4877's read-vs-restore serialization rules out (see backlog#1148
+      #     ilm-8 criterion 1 - an API-semantics decision, not a bug).
       - name: Run ignored ILM integration tests serially
         run: |
           cargo nextest run -j1 --run-ignored ignored-only \
             -p rustfs-scanner -p rustfs \
-            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
+            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition) or test(restore_object_usecase_reports_ongoing_conflict_and_completion) or test(test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore))'
 
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)

Cargo.lock

@@ -3589,7 +3589,7 @@ dependencies = [
  "libc",
  "option-ext",
  "redox_users 0.5.2",
- "windows-sys 0.59.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]
@@ -3644,7 +3644,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
 
 [[package]]
 name = "e2e_test"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "anyhow",
  "astral-tokio-tar",
@@ -3926,7 +3926,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
 dependencies = [
  "libc",
- "windows-sys 0.59.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -5335,7 +5335,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
 dependencies = [
  "hermit-abi",
  "libc",
- "windows-sys 0.59.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -6520,7 +6520,7 @@ version = "0.50.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
 dependencies = [
- "windows-sys 0.59.0",
+ "windows-sys 0.61.2",
 ]
 
 [[package]]
@@ -8050,7 +8050,7 @@ dependencies = [
  "once_cell",
  "socket2",
  "tracing",
- "windows-sys 0.59.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -8804,7 +8804,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "aes-gcm",
  "anyhow",
@@ -8935,7 +8935,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-audit"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "chrono",
@@ -8957,7 +8957,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-checksums"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "base64-simd",
  "bytes",
@@ -8972,7 +8972,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-common"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "chrono",
  "metrics",
@@ -8987,7 +8987,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-concurrency"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "insta",
  "rustfs-io-core",
@@ -8999,7 +8999,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-config"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "const-str",
  "serde",
@@ -9008,7 +9008,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-credentials"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "base64-simd",
  "hmac 0.13.0",
@@ -9021,7 +9021,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-crypto"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "aes-gcm",
  "argon2",
@@ -9041,7 +9041,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-data-usage"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "path-clean",
@@ -9052,7 +9052,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-ecstore"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "aes-gcm",
  "async-channel",
@@ -9186,7 +9186,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-extension-schema"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "serde",
  "serde_json",
@@ -9195,7 +9195,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-filemeta"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "arc-swap",
  "byteorder",
@@ -9221,7 +9221,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-heal"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "base64 0.22.1",
@@ -9251,7 +9251,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-iam"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "arc-swap",
  "async-trait",
@@ -9288,7 +9288,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-io-core"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "bytes",
  "memmap2",
@@ -9300,7 +9300,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-io-metrics"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "criterion",
  "metrics",
@@ -9363,7 +9363,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-keystone"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "bytes",
  "futures",
@@ -9389,7 +9389,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-kms"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "aes-gcm",
  "arc-swap",
@@ -9420,7 +9420,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-lifecycle"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "proptest",
@@ -9440,7 +9440,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-lock"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "crossbeam-queue",
@@ -9461,7 +9461,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-madmin"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "chrono",
  "humantime",
@@ -9475,7 +9475,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-notify"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "arc-swap",
  "async-trait",
@@ -9507,7 +9507,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-object-capacity"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "criterion",
  "futures",
@@ -9525,7 +9525,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-object-data-cache"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "bytes",
  "criterion",
@@ -9541,7 +9541,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-obs"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "chrono",
  "crossbeam-channel",
@@ -9593,7 +9593,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-policy"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "base64-simd",
@@ -9622,7 +9622,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-protocols"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "astral-tokio-tar",
  "async-compression",
@@ -9681,7 +9681,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-protos"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "flatbuffers",
  "prost 0.14.4",
@@ -9699,7 +9699,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-replication"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "byteorder",
  "bytes",
@@ -9716,7 +9716,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-rio"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "aes-gcm",
  "arc-swap",
@@ -9754,7 +9754,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-rio-v2"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "aes-gcm",
  "bytes",
@@ -9776,14 +9776,14 @@ dependencies = [
 
 [[package]]
 name = "rustfs-s3-ops"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "rustfs-s3-types",
 ]
 
 [[package]]
 name = "rustfs-s3-types"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "serde",
  "serde_json",
@@ -9791,7 +9791,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-s3select-api"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "bytes",
@@ -9818,7 +9818,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-s3select-query"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-recursion",
  "async-trait",
@@ -9834,7 +9834,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-scanner"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "chrono",
@@ -9866,14 +9866,14 @@ dependencies = [
 
 [[package]]
 name = "rustfs-security-governance"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "thiserror 2.0.18",
 ]
 
 [[package]]
 name = "rustfs-signer"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "base64-simd",
  "bytes",
@@ -9889,7 +9889,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-storage-api"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "insta",
@@ -9903,7 +9903,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-targets"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "arc-swap",
  "async-nats",
@@ -9953,7 +9953,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-test-utils"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "rustfs-ecstore",
  "rustfs-storage-api",
@@ -9965,7 +9965,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-tls-runtime"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "arc-swap",
  "metrics",
@@ -9985,7 +9985,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-trusted-proxies"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "async-trait",
  "axum",
@@ -10021,7 +10021,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-utils"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "base64-simd",
  "blake2",
@@ -10060,7 +10060,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-zip"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 dependencies = [
  "astral-tokio-tar",
  "async-compression",
@@ -10125,7 +10125,7 @@ dependencies = [
  "errno",
  "libc",
  "linux-raw-sys 0.4.15",
- "windows-sys 0.59.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -10138,7 +10138,7 @@ dependencies = [
  "errno",
  "libc",
  "linux-raw-sys 0.12.1",
- "windows-sys 0.59.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -10212,7 +10212,7 @@ dependencies = [
  "security-framework",
  "security-framework-sys",
  "webpki-root-certs",
- "windows-sys 0.59.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -11375,7 +11375,7 @@ dependencies = [
  "getrandom 0.4.3",
  "once_cell",
  "rustix 1.1.4",
- "windows-sys 0.59.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -12475,7 +12475,7 @@ version = "0.1.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
 dependencies = [
- "windows-sys 0.59.0",
+ "windows-sys 0.52.0",
 ]
 
 [[package]]
@@ -12605,15 +12605,6 @@ dependencies = [
  "windows-targets",
 ]
 
-[[package]]
-name = "windows-sys"
-version = "0.59.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
-dependencies = [
- "windows-targets",
-]
-
 [[package]]
 name = "windows-sys"
 version = "0.61.2"

Cargo.toml

@@ -68,7 +68,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.96.0"
-version = "1.0.0-beta.10-preview.1"
+version = "1.0.0-beta.10"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -85,51 +85,51 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.10-preview.1" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.10-preview.1" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.10-preview.1" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.10-preview.1" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.10-preview.1" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.10-preview.1" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.10-preview.1" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.10-preview.1" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.10-preview.1" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.10-preview.1" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.10-preview.1" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.10-preview.1" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.10-preview.1" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.10-preview.1" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.10-preview.1" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.10-preview.1" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.10-preview.1" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.10-preview.1" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.10-preview.1" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.10-preview.1" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.10-preview.1" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.10-preview.1" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.10-preview.1" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.10-preview.1" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.10-preview.1" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.10-preview.1" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.10-preview.1" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.10-preview.1" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.10-preview.1" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.10-preview.1" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.10-preview.1" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.10-preview.1" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.10-preview.1" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.10-preview.1" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.10-preview.1" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.10-preview.1" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.10-preview.1" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.10-preview.1" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.10-preview.1" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.10-preview.1" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.10-preview.1" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.10-preview.1" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.10-preview.1" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.10-preview.1" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.10-preview.1" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.10" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.10" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.10" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.10" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.10" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.10" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.10" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.10" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.10" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.10" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.10" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.10" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.10" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.10" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.10" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.10" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.10" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.10" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.10" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.10" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.10" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.10" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.10" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.10" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.10" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.10" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.10" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.10" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.10" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.10" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.10" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.10" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.10" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.10" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.10" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.10" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.10" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.10" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.10" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.10" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.10" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.10" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.10" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.10" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.10" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
@@ -277,7 +277,6 @@ rand = { version = "0.10.2" }
 ratelimit = "0.10.1"
 rayon = "1.12.0"
 reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.0" }
-#reed-solomon-erasure = { version = "6.0", features = ["simd-accel"], git = "https://github.com/houseme/reed-solomon-erasure",rev = "main" }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.33.2" }

Dockerfile

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM alpine:3.23.4 AS build
+FROM alpine:3.24.1 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -70,7 +70,7 @@ RUN set -eux; \
     rm -rf rustfs.zip /build/.tmp || true
 
 
-FROM alpine:3.23.4
+FROM alpine:3.24.1
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -88,7 +88,9 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-RUN apk update && \
+# Upgrade base-image packages so published images pick up security fixes
+# (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
+RUN apk upgrade --no-cache && \
     apk add --no-cache ca-certificates coreutils curl
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

Dockerfile.glibc

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:24.04 AS build
+FROM ubuntu:26.04 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -76,7 +76,7 @@ RUN set -eux; \
     chmod +x /build/rustfs; \
     rm -rf rustfs.zip /build/.tmp || true
 
-FROM ubuntu:24.04
+FROM ubuntu:26.04
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -93,7 +93,10 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-RUN apt-get update && apt-get install -y --no-install-recommends \
+# Upgrade base-image packages so published images pick up security fixes
+# (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
+RUN apt-get update && apt-get upgrade -y \
+    && apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \
     && rm -rf /var/lib/apt/lists/*

Dockerfile.source

@@ -208,7 +208,7 @@ CMD ["cargo", "run", "--bin", "rustfs", "--"]
 # -----------------------------
 # Runtime stage (Ubuntu minimal)
 # -----------------------------
-FROM ubuntu:24.04
+FROM ubuntu:26.04
 
 ARG BUILD_DATE
 ARG VCS_REF
@@ -223,6 +223,7 @@ LABEL name="RustFS (dev-local)" \
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \
     apt-get update; \
+    apt-get upgrade -y; \
     apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \

README.md

@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10-preview.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -218,7 +218,10 @@ For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status,
 and single-node single-disk idle CPU tuning, see
 [Scanner Runtime Controls](docs/operations/scanner-runtime-controls.md). For
 repeatable scanner-pressure validation, see
-[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md).
+[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md). For
+drive timeout knobs on slow storage — including the walk stall budget that
+governs `ListObjects` on large prefixes — see
+[Drive Timeout Tuning](docs/operations/drive-timeout-tuning.md).
 
 ### 5\. Nix Flake (Option 5)
 

README_ZH.md

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10-preview.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：

crates/config/src/constants/object.rs

@@ -73,15 +73,36 @@ pub const DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS: usize = 64;
 /// - Example: `export RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=5`
 pub const ENV_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT";
 
-/// Maximum time a GET request waits for a disk read permit (seconds).
+/// Maximum time a GET request waits for a primary disk read permit (seconds).
 ///
 /// Permits are held for the whole response body transfer, so slow clients can
 /// occupy all of them while the disks sit idle. Instead of stalling until the
-/// request-level timeout fires, a GET that waits longer than this proceeds
-/// without a permit (degraded pass-through) and the bypass is counted in
-/// metrics/logs. Set to 0 to wait indefinitely (previous behavior).
+/// request-level timeout fires, a GET that waits longer than this falls through
+/// to a bounded degraded admission lane; if that lane is also full the request
+/// is rejected with `SlowDown`/503 rather than proceeding without any permit.
+/// Set to 0 to wait on the primary lane indefinitely (never degrade or reject).
 pub const DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: u64 = 5;
 
+/// Environment variable for the bounded degraded disk-read admission lane size.
+/// - Purpose: Cap how many GETs may proceed after the primary disk-read permit
+///   pool is saturated, giving a hard upper bound on concurrent disk-active
+///   reads (primary cap + degraded cap) instead of an unbounded pass-through.
+/// - Unit: request count (usize). `0` means "mirror the primary cap", so the
+///   absolute hard cap defaults to twice the primary disk-read cap.
+/// - Example: `export RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP=16`
+pub const ENV_OBJECT_DISK_DEGRADED_READ_CAP: &str = "RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP";
+
+/// Size of the bounded degraded disk-read admission lane.
+///
+/// When the primary disk-read permit pool is saturated and a GET exceeds
+/// [`DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT`], it may take one permit from this
+/// bounded overflow lane instead of reading without any admission token. The
+/// total number of GETs performing disk-active reads is therefore hard-capped at
+/// `primary_cap + degraded_cap`; beyond that a GET is rejected with `SlowDown`.
+/// The default `0` mirrors the primary cap, so the hard cap is twice the primary
+/// disk-read concurrency.
+pub const DEFAULT_OBJECT_DISK_DEGRADED_READ_CAP: usize = 0;
+
 /// Skip bitrot hash verification on GetObject reads.
 ///
 /// When enabled, GetObject reads skip the per-shard hash
@@ -128,6 +149,39 @@ pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT"
 /// Default disk read timeout in seconds.
 pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;
 
+/// Environment variable for the per-shard erasure write stall timeout (seconds).
+///
+/// A single shard write (or shard-writer shutdown) that makes no forward
+/// progress for longer than this budget is failed and its disk is dropped
+/// before commit, so a black-hole peer that accepts the connection but never
+/// drains the body cannot pin an otherwise-healthy write quorum forever
+/// (see `MultiWriter` in `erasure/coding/encode.rs`). The budget is re-armed on
+/// every shard write, so it bounds a *stall* rather than the total transfer
+/// time of a large object.
+///
+/// Unit: seconds (u64). `0` disables the stall deadline (previous behavior:
+/// wait indefinitely). Default: 30 seconds.
+pub const ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT";
+
+/// Default per-shard erasure write stall timeout in seconds.
+pub const DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT: u64 = 30;
+
+/// Environment variable for the absolute per-object erasure write cap (seconds).
+///
+/// Optional administrator backstop against a "slow-drip" peer that produces
+/// just enough forward progress to reset the per-shard stall timeout on every
+/// block while never converging. When set, the shard writers for one object are
+/// engaged for at most this long in aggregate before a stalled writer is failed
+/// and dropped. It is disabled by default because a legitimate large upload
+/// over a slow-but-honest link must not be killed on total time alone; the
+/// per-shard stall timeout is the primary guarantee.
+///
+/// Unit: seconds (u64). `0` (default) disables the absolute cap.
+pub const ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP: &str = "RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP";
+
+/// Default absolute per-object erasure write cap in seconds (`0` = disabled).
+pub const DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP: u64 = 0;
+
 /// Environment variable for minimum GetObject timeout in seconds.
 ///
 /// When dynamic timeout calculation is enabled, this is the minimum timeout

crates/e2e_test/src/cluster_multidrive_pool_test.rs

@@ -0,0 +1,108 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Smoke tests for the multi-drive / multi-pool cluster harness.
+//!
+//! These belong to the nightly 4-node cluster lane (they spin up real RustFS
+//! processes, so they are excluded from the merge-gate `e2e-full` profile in
+//! `.config/nextest.toml`). They assert that:
+//!
+//! * `drivesPerNode > 1` produces a bootable single-pool cluster whose
+//!   `RUSTFS_VOLUMES` string enumerates every drive, and that a PUT/GET
+//!   round-trips through the multi-drive erasure layout.
+//! * A two-pool topology (one node per pool, `drivesPerNode` drives each) boots
+//!   with the ellipses `RUSTFS_VOLUMES` form and round-trips a PUT/GET.
+//!
+//! Readiness is established by the harness's `start()` handshake (TCP reachability
+//! plus an S3 `ListBuckets` poll) — there are no fixed sleeps.
+//!
+//! Out of scope for this block (tracked separately): network fault injection
+//! (toxiproxy / socket proxy) and 5GiB large-object budgets.
+
+use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
+use serial_test::serial;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+const BUCKET: &str = "cluster-multidrive-pool-bucket";
+
+/// PUT an object then GET it back and assert the bytes round-trip.
+async fn put_get_roundtrip(cluster: &RustFSTestClusterEnvironment, key: &str, payload: &[u8]) -> TestResult {
+    let writer = cluster.create_s3_client(0)?;
+    writer
+        .put_object()
+        .bucket(BUCKET)
+        .key(key)
+        .body(bytes::Bytes::copy_from_slice(payload).into())
+        .send()
+        .await?;
+
+    // Read back from the last node to exercise the cross-node/cross-pool path.
+    let reader = cluster.create_s3_client(cluster.nodes.len() - 1)?;
+    let got = reader.get_object().bucket(BUCKET).key(key).send().await?;
+    let body = got.body.collect().await?.into_bytes();
+    assert_eq!(body.as_ref(), payload, "round-tripped object bytes must match");
+    Ok(())
+}
+
+/// 4 nodes x 2 drives, single pool: the multi-drive layout boots and round-trips.
+#[tokio::test]
+#[serial]
+async fn cluster_multidrive_single_pool_smoke() -> TestResult {
+    crate::common::init_logging();
+
+    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 2)).await?;
+
+    // The single-pool multi-drive layout must list every (node, drive) endpoint
+    // explicitly (8 endpoints, no ellipses) so the server keeps one legacy pool.
+    let volumes = cluster.rustfs_volumes_arg();
+    assert_eq!(volumes.split(' ').count(), 8, "expected 8 explicit endpoints, got: {volumes}");
+    assert!(!volumes.contains('{'), "single-pool layout must not use ellipses: {volumes}");
+
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let payload = vec![0xA5u8; 512 * 1024];
+    put_get_roundtrip(&cluster, "multidrive/object", &payload).await?;
+    Ok(())
+}
+
+/// Two single-node pools, 2 drives each: the multi-pool layout boots and
+/// round-trips. Every pool is a distinct erasure pool (`pool_idx` 0 and 1).
+#[tokio::test]
+#[serial]
+async fn cluster_two_pool_smoke() -> TestResult {
+    crate::common::init_logging();
+
+    let mut cluster =
+        RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]])).await?;
+
+    // The two-pool layout must emit one ellipses argument per pool.
+    let volumes = cluster.rustfs_volumes_arg();
+    let args: Vec<&str> = volumes.split(' ').collect();
+    assert_eq!(args.len(), 2, "expected two pool arguments, got: {volumes}");
+    assert!(
+        args.iter().all(|a| a.contains("/drive{0...1}")),
+        "each pool arg must use the drive ellipses form: {volumes}"
+    );
+    assert_eq!(cluster.nodes[0].pool_idx, 0);
+    assert_eq!(cluster.nodes[1].pool_idx, 1);
+
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let payload = vec![0x5Au8; 256 * 1024];
+    put_get_roundtrip(&cluster, "twopool/object", &payload).await?;
+    Ok(())
+}

crates/e2e_test/src/common.rs

@@ -717,15 +717,166 @@ pub async fn awscurl_delete(
     execute_awscurl(url, "DELETE", None, access_key, secret_key).await
 }
 
+/// Cluster topology declaration for a `RustFSTestClusterEnvironment`.
+///
+/// Describes how many nodes to launch, how many data drives each node exposes,
+/// and how those nodes are grouped into erasure pools. The topology drives both
+/// on-disk directory creation and the `RUSTFS_VOLUMES` string assembled for the
+/// node processes.
+///
+/// # Single-host expressibility constraints
+///
+/// The harness runs every node on `127.0.0.1` with a distinct port, so the
+/// `RUSTFS_VOLUMES` string can only use the two forms the server parser accepts
+/// on a single machine (verified against `ecstore` `DisksLayout::from_volumes`):
+///
+/// * **Single pool** — every `(node, drive)` endpoint listed explicitly, no
+///   ellipses. This is the legacy form and yields exactly one pool spanning all
+///   nodes and drives (`DistErasure`). Any `drives_per_node >= 1` works.
+/// * **Multiple pools** — one ellipses arg per pool. A pool argument is a single
+///   URL template, so it can enumerate several drives on *one* host but cannot
+///   enumerate multiple distinct-port hosts. A pool that spanned two localhost
+///   nodes would need a host ellipses (`127.0.0.1:900{0...1}`), which forces the
+///   *same* on-disk path across those ports and collides physically. Therefore
+///   every pool in a multi-pool topology owns exactly one node. In addition, the
+///   parser rejects a single-drive ellipses pool (`drive{0...0}`), so a
+///   multi-pool topology requires `drives_per_node >= 2`.
+///
+/// Genuine multi-node pools (a pool striped across several hosts) require real
+/// multi-host infrastructure and are deferred to the nightly cluster CI lane
+/// (backlog #1313 / #1314); they are intentionally not expressible here.
+#[derive(Clone, Debug)]
+pub struct ClusterTopology {
+    /// Number of node processes to launch.
+    pub node_count: usize,
+    /// Number of independent data drives (directories) each node exposes.
+    pub drives_per_node: usize,
+    /// Pool membership as a list of node-index groups. An empty vector means a
+    /// single pool spanning every node.
+    pub pools: Vec<Vec<usize>>,
+}
+
+impl ClusterTopology {
+    /// Single pool, one drive per node — identical to the historical
+    /// `RustFSTestClusterEnvironment::new` layout.
+    pub fn single_pool(node_count: usize) -> Self {
+        Self {
+            node_count,
+            drives_per_node: 1,
+            pools: Vec::new(),
+        }
+    }
+
+    /// Single pool with `drives_per_node` drives on every node. The pool spans
+    /// all nodes and all drives (one `DistErasure` pool).
+    pub fn single_pool_multidrive(node_count: usize, drives_per_node: usize) -> Self {
+        Self {
+            node_count,
+            drives_per_node,
+            pools: Vec::new(),
+        }
+    }
+
+    /// Multi-pool topology where every pool owns exactly one node. `pools` lists
+    /// the node index backing each pool (e.g. `vec![vec![0], vec![1]]` for a
+    /// two-pool cluster). Requires `drives_per_node >= 2` (see the type-level
+    /// note on single-host expressibility).
+    pub fn per_node_pools(drives_per_node: usize, pools: Vec<Vec<usize>>) -> Self {
+        let node_count = pools.iter().flatten().copied().max().map(|m| m + 1).unwrap_or(0);
+        Self {
+            node_count,
+            drives_per_node,
+            pools,
+        }
+    }
+
+    /// Normalized pool membership: an empty `pools` becomes a single pool over
+    /// all node indices.
+    fn normalized_pools(&self) -> Vec<Vec<usize>> {
+        if self.pools.is_empty() {
+            vec![(0..self.node_count).collect()]
+        } else {
+            self.pools.clone()
+        }
+    }
+
+    /// Validate that the topology is expressible on a single localhost machine.
+    fn validate(&self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        if self.node_count == 0 {
+            return Err("Node count must be greater than zero".into());
+        }
+        if self.drives_per_node == 0 {
+            return Err("drives_per_node must be greater than zero".into());
+        }
+
+        let pools = self.normalized_pools();
+
+        // Every referenced node index must be in range.
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            if nodes.is_empty() {
+                return Err(format!("pool {pool_idx} has no nodes").into());
+            }
+            for &n in nodes {
+                if n >= self.node_count {
+                    return Err(format!("pool {pool_idx} references node {n} but node_count is {}", self.node_count).into());
+                }
+            }
+        }
+
+        // Each node must belong to exactly one pool.
+        let mut seen = vec![0usize; self.node_count];
+        for nodes in &pools {
+            for &n in nodes {
+                seen[n] += 1;
+            }
+        }
+        for (n, count) in seen.iter().enumerate() {
+            match count {
+                0 => return Err(format!("node {n} is not assigned to any pool").into()),
+                1 => {}
+                _ => return Err(format!("node {n} is assigned to more than one pool").into()),
+            }
+        }
+
+        // Multi-pool constraints imposed by the single-host RUSTFS_VOLUMES syntax.
+        if pools.len() > 1 {
+            if self.drives_per_node < 2 {
+                return Err(
+                    "multi-pool topology requires drives_per_node >= 2 (the server parser rejects a single-drive ellipses pool)"
+                        .into(),
+                );
+            }
+            for (pool_idx, nodes) in pools.iter().enumerate() {
+                if nodes.len() != 1 {
+                    return Err(format!(
+                        "pool {pool_idx} spans {} nodes; a pool striped across multiple localhost nodes is not expressible (needs host ellipses, which collides on shared paths). Use one node per pool, or the nightly multi-host lane (backlog #1313/#1314).",
+                        nodes.len()
+                    )
+                    .into());
+                }
+            }
+        }
+
+        Ok(())
+    }
+}
+
 /// Represents a single RustFS server instance in a test cluster.
 ///
 /// Each `ClusterNode` tracks the node's network address, base URL for
-/// S3-compatible requests, on-disk data directory, and the underlying
+/// S3-compatible requests, on-disk data directories, and the underlying
 /// child process handle when the node is running.
 pub struct ClusterNode {
     pub address: String,
     pub url: String,
+    /// Primary data directory for the node. For single-drive nodes this is the
+    /// node's only drive; for multi-drive nodes it is the first drive. Kept as a
+    /// stable field so existing single-drive tests continue to compile.
     pub data_dir: String,
+    /// All data drives exposed by this node (`data_dirs[0] == data_dir`).
+    pub data_dirs: Vec<String>,
+    /// Index of the pool this node belongs to.
+    pub pool_idx: usize,
     pub process: Option<Child>,
 }
 
@@ -741,6 +892,7 @@ pub struct RustFSTestClusterEnvironment {
     pub access_key: String,
     pub secret_key: String,
     pub extra_env: Vec<(String, String)>,
+    pub topology: ClusterTopology,
 }
 
 impl RustFSTestClusterEnvironment {
@@ -762,34 +914,82 @@ impl RustFSTestClusterEnvironment {
     /// * `Err(Box<dyn Error + Send + Sync>)` - An error if any step fails, such as temporary
     ///   directory creation failure or available port lookup failure.
     pub async fn new(node_count: usize) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
-        if node_count == 0 {
-            return Err("Node count must be greater than zero".into());
-        }
+        Self::with_topology(ClusterTopology::single_pool(node_count)).await
+    }
+
+    /// Create a RustFS test cluster environment from an explicit topology.
+    ///
+    /// Allocates a unique temporary root directory, an available TCP port per
+    /// node, and one data directory per drive. The topology is validated up front
+    /// (node/pool assignment, single-host multi-pool constraints); node processes
+    /// are not started at this stage.
+    ///
+    /// When the topology exposes more than one local drive on a node (multi-drive
+    /// or multi-pool layouts), `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` is added to
+    /// every node's environment: those drives live on the same temp filesystem, so
+    /// the server's distinct-physical-disk safety check would otherwise reject
+    /// startup. Single-drive single-pool clusters keep the historical environment
+    /// unchanged.
+    pub async fn with_topology(topology: ClusterTopology) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
+        topology.validate()?;
+
         let temp_dir = format!("/tmp/rustfs_cluster_test_{}", Uuid::new_v4());
         fs::create_dir_all(&temp_dir).await?;
 
-        let mut nodes = Vec::with_capacity(node_count);
-        for i in 0..node_count {
+        // Map every node to its owning pool index.
+        let pools = topology.normalized_pools();
+        let mut pool_of_node = vec![0usize; topology.node_count];
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            for &n in nodes {
+                pool_of_node[n] = pool_idx;
+            }
+        }
+
+        let multidrive = topology.drives_per_node > 1;
+
+        let mut nodes = Vec::with_capacity(topology.node_count);
+        for (i, &pool_idx) in pool_of_node.iter().enumerate() {
             let port = RustFSTestEnvironment::find_available_port().await?;
             let address = format!("127.0.0.1:{}", port);
             let url = format!("http://{}", address);
-            let data_dir = format!("{}/node{}", temp_dir, i);
-            fs::create_dir_all(&data_dir).await?;
+
+            // Single-drive nodes keep the historical `{temp}/node{i}` path so
+            // existing tests (and their on-disk assertions) are unaffected.
+            // Multi-drive nodes nest one `drive{d}` directory per drive so the
+            // ellipses form `drive{0...N-1}` can address them.
+            let data_dirs: Vec<String> = if multidrive {
+                (0..topology.drives_per_node)
+                    .map(|d| format!("{}/node{}/drive{}", temp_dir, i, d))
+                    .collect()
+            } else {
+                vec![format!("{}/node{}", temp_dir, i)]
+            };
+            for dir in &data_dirs {
+                fs::create_dir_all(dir).await?;
+            }
 
             nodes.push(ClusterNode {
                 address,
                 url,
-                data_dir,
+                data_dir: data_dirs[0].clone(),
+                data_dirs,
+                pool_idx,
                 process: None,
             });
         }
 
+        let mut extra_env = Vec::new();
+        if multidrive {
+            extra_env.push(("RUSTFS_UNSAFE_BYPASS_DISK_CHECK".to_string(), "true".to_string()));
+        }
+
         Ok(Self {
             nodes,
             temp_dir,
             access_key: DEFAULT_ACCESS_KEY.to_string(),
             secret_key: DEFAULT_SECRET_KEY.to_string(),
-            extra_env: Vec::new(),
+            extra_env,
+            topology,
         })
     }
 
@@ -809,14 +1009,51 @@ impl RustFSTestClusterEnvironment {
         Ok(())
     }
 
-    /// Build the volumes argument string for RustFS binary (internal helper method).
+    /// Build the `RUSTFS_VOLUMES` argument string for the cluster's topology.
     ///
-    /// Concatenates the address and data directory of all cluster nodes into a single string
-    /// used as the `RUSTFS_VOLUMES` environment variable for RustFS node processes.
+    /// * **Single pool** — every `(node, drive)` endpoint is listed explicitly and
+    ///   space-joined. With no ellipses the server parser collapses them into one
+    ///   legacy pool spanning all nodes and drives.
+    /// * **Multiple pools** — each pool (exactly one node) contributes one ellipses
+    ///   argument `http://<addr><node-base>/drive{0...N-1}`; the space-separated
+    ///   arguments become one pool each.
+    ///
+    /// The server splits `RUSTFS_VOLUMES` on spaces (`value_delimiter = ' '`), which
+    /// this assembly matches, and the resulting layout is verified against the
+    /// `ecstore` parser in the unit tests below.
+    /// Public view of the assembled `RUSTFS_VOLUMES` string, so tests can assert
+    /// the pool/drive layout without starting node processes.
+    pub fn rustfs_volumes_arg(&self) -> String {
+        self.build_volumes_arg()
+    }
+
     fn build_volumes_arg(&self) -> String {
-        self.nodes
+        let pools = self.topology.normalized_pools();
+
+        if pools.len() <= 1 {
+            // Single pool: explicit enumeration of every drive on every node.
+            return self
+                .nodes
+                .iter()
+                .flat_map(|n| n.data_dirs.iter().map(move |dir| format!("http://{}{}", n.address, dir)))
+                .collect::<Vec<_>>()
+                .join(" ");
+        }
+
+        // Multi-pool: one ellipses argument per single-node pool. The drive
+        // directories are `<temp>/node{i}/drive{0..N-1}`, so the ellipses base is
+        // the shared parent of the node's drives.
+        pools
             .iter()
-            .map(|n| format!("http://{}{}", n.address, n.data_dir))
+            .map(|nodes| {
+                let node = &self.nodes[nodes[0]];
+                let base = node
+                    .data_dirs
+                    .first()
+                    .and_then(|d| d.rsplit_once('/').map(|(parent, _)| parent))
+                    .unwrap_or(&node.data_dir);
+                format!("http://{}{}/drive{{0...{}}}", node.address, base, self.topology.drives_per_node - 1)
+            })
             .collect::<Vec<_>>()
             .join(" ")
     }
@@ -1091,4 +1328,130 @@ mod tests {
 
         stdfs::remove_file(stamp_path).ok();
     }
+
+    /// Build a cluster environment struct in-memory (no ports, no processes) so
+    /// that `build_volumes_arg` can be exercised as a pure string builder. Node
+    /// directories mirror what `with_topology` would create for the topology.
+    fn fake_cluster(topology: ClusterTopology) -> RustFSTestClusterEnvironment {
+        let temp_dir = "/tmp/rustfs_cluster_test_FAKE".to_string();
+        let pools = topology.normalized_pools();
+        let mut pool_of_node = vec![0usize; topology.node_count];
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            for &n in nodes {
+                pool_of_node[n] = pool_idx;
+            }
+        }
+        let multidrive = topology.drives_per_node > 1;
+
+        let nodes = (0..topology.node_count)
+            .map(|i| {
+                let address = format!("127.0.0.1:{}", 9000 + i);
+                let data_dirs: Vec<String> = if multidrive {
+                    (0..topology.drives_per_node)
+                        .map(|d| format!("{}/node{}/drive{}", temp_dir, i, d))
+                        .collect()
+                } else {
+                    vec![format!("{}/node{}", temp_dir, i)]
+                };
+                ClusterNode {
+                    url: format!("http://{}", address),
+                    address,
+                    data_dir: data_dirs[0].clone(),
+                    data_dirs,
+                    pool_idx: pool_of_node[i],
+                    process: None,
+                }
+            })
+            .collect();
+
+        RustFSTestClusterEnvironment {
+            nodes,
+            temp_dir,
+            access_key: DEFAULT_ACCESS_KEY.to_string(),
+            secret_key: DEFAULT_SECRET_KEY.to_string(),
+            extra_env: Vec::new(),
+            topology,
+        }
+    }
+
+    #[test]
+    fn volumes_single_pool_single_drive_is_backward_compatible() {
+        // The historical layout: one explicit endpoint per node, space-joined,
+        // no ellipses, no `drive` sub-directory.
+        let env = fake_cluster(ClusterTopology::single_pool(4));
+        assert_eq!(
+            env.build_volumes_arg(),
+            "http://127.0.0.1:9000/tmp/rustfs_cluster_test_FAKE/node0 \
+             http://127.0.0.1:9001/tmp/rustfs_cluster_test_FAKE/node1 \
+             http://127.0.0.1:9002/tmp/rustfs_cluster_test_FAKE/node2 \
+             http://127.0.0.1:9003/tmp/rustfs_cluster_test_FAKE/node3"
+        );
+    }
+
+    #[test]
+    fn volumes_single_pool_multidrive_enumerates_every_drive() {
+        // 4 nodes x 2 drives -> 8 explicit endpoints, one legacy pool. No
+        // ellipses, so the server parser keeps this as a single DistErasure pool.
+        let env = fake_cluster(ClusterTopology::single_pool_multidrive(4, 2));
+        let expected = (0..4)
+            .flat_map(|i| {
+                (0..2).map(move |d| format!("http://127.0.0.1:{}/tmp/rustfs_cluster_test_FAKE/node{}/drive{}", 9000 + i, i, d))
+            })
+            .collect::<Vec<_>>()
+            .join(" ");
+        assert_eq!(env.build_volumes_arg(), expected);
+        assert_eq!(env.build_volumes_arg().split(' ').count(), 8);
+    }
+
+    #[test]
+    fn volumes_two_pool_uses_one_ellipses_arg_per_pool() {
+        // Two single-node pools, 2 drives each -> two ellipses arguments. The
+        // server parser treats each space-separated ellipses arg as its own pool.
+        let env = fake_cluster(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]));
+        assert_eq!(
+            env.build_volumes_arg(),
+            "http://127.0.0.1:9000/tmp/rustfs_cluster_test_FAKE/node0/drive{0...1} \
+             http://127.0.0.1:9001/tmp/rustfs_cluster_test_FAKE/node1/drive{0...1}"
+        );
+    }
+
+    #[test]
+    fn topology_rejects_multi_node_pool() {
+        // A pool striped across two localhost nodes is not expressible.
+        let err = ClusterTopology::per_node_pools(2, vec![vec![0, 1], vec![2, 3]])
+            .validate()
+            .unwrap_err()
+            .to_string();
+        assert!(err.contains("not expressible"), "unexpected error: {err}");
+    }
+
+    #[test]
+    fn topology_rejects_single_drive_multi_pool() {
+        // The server parser rejects a single-drive ellipses pool (`drive{0...0}`).
+        let err = ClusterTopology::per_node_pools(1, vec![vec![0], vec![1]])
+            .validate()
+            .unwrap_err()
+            .to_string();
+        assert!(err.contains("drives_per_node >= 2"), "unexpected error: {err}");
+    }
+
+    #[test]
+    fn topology_rejects_unassigned_and_duplicated_nodes() {
+        // Node 2 is never assigned to a pool.
+        let mut t = ClusterTopology::single_pool_multidrive(3, 2);
+        t.pools = vec![vec![0], vec![1]];
+        assert!(t.validate().unwrap_err().to_string().contains("not assigned"));
+
+        // Node 0 assigned twice.
+        let mut t = ClusterTopology::single_pool_multidrive(2, 2);
+        t.pools = vec![vec![0], vec![0]];
+        assert!(t.validate().unwrap_err().to_string().contains("more than one pool"));
+    }
+
+    #[test]
+    fn topology_single_pool_accepts_any_drive_count() {
+        assert!(ClusterTopology::single_pool(4).validate().is_ok());
+        assert!(ClusterTopology::single_pool_multidrive(4, 4).validate().is_ok());
+        assert!(ClusterTopology::single_pool_multidrive(1, 1).validate().is_ok());
+    }
 }

crates/e2e_test/src/fake_s3_target/README.md

@@ -0,0 +1,11 @@
+# Programmable fake S3 target
+
+This module is the shared failure-injection boundary for replication end-to-end tests. It runs an in-process, path-style S3 endpoint backed by `s3s`; no production crate depends on it.
+
+`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
+
+Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
+
+Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
+
+The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.

crates/e2e_test/src/fault_proxy.rs

@@ -0,0 +1,498 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! In-process, socket-level network fault-injection proxy for black-box
+//! cluster E2E tests (backlog#1325 network fault-injection block).
+//!
+//! `FaultProxy` binds a TCP listener on a random local port, forwards every
+//! accepted connection to a fixed target address, and lets a test flip the
+//! forwarding behaviour at runtime. It is the black-box counterpart to the
+//! in-process white-box hooks (rename barrier, disk call counters): those
+//! `#[cfg(test)]` primitives cannot reach across the process boundary into a
+//! spawned RustFS server, whereas this proxy sits on the wire between two
+//! nodes and needs no cooperation from the peer.
+//!
+//! Service targets:
+//! - **#1312 / #1319** lock-plane one-way partition and accept-then-blackhole
+//!   peer: run a node's lock/RPC port through the proxy, then
+//!   [`FaultMode::Partition`] one direction (data plane stays reachable via a
+//!   separate direct port) or [`FaultMode::Blackhole`] the whole connection.
+//! - **#1327** cross-process replay / tamper harness: the proxy is the wire
+//!   seam a replay/tamper filter can later hook into.
+//!
+//! Supported fault modes:
+//! - [`FaultMode::Pass`]: transparent byte forwarding (round-trip identical).
+//! - [`FaultMode::Latency`]: delay every forwarded chunk by a fixed duration.
+//! - [`FaultMode::Blackhole`]: accept the connection but forward nothing in
+//!   either direction and never respond (models an accepted-then-dead peer);
+//!   the client observes a read timeout, the proxy never panics.
+//! - [`FaultMode::Partition`]: block exactly one direction while the other
+//!   keeps flowing (models a lock-RPC one-way partition with a live data
+//!   plane).
+//!
+//! Wiring this into the cluster harness ("expose node port N through a proxy")
+//! is intentionally left as a follow-up: the harness multi-drive / 2-pool
+//! changes land separately (rustfs#4937), so this block ships the proxy tool
+//! plus its local self-tests against a loopback echo server and stays
+//! independent of that harness work.
+//!
+//! # Example
+//!
+//! ```ignore
+//! let proxy = FaultProxy::start(target_addr).await?;
+//! let via = proxy.local_addr(); // point the client here instead of `target_addr`
+//! proxy.set_mode(FaultMode::Latency(Duration::from_millis(50)));
+//! // ... exercise the client ...
+//! proxy.set_mode(FaultMode::Partition(Direction::ClientToServer));
+//! // ... assert the blocked direction is dead ...
+//! proxy.shutdown().await; // releases the listener port
+//! ```
+
+use std::io;
+use std::net::{Ipv4Addr, SocketAddr};
+use std::time::Duration;
+
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::net::{TcpListener, TcpStream};
+use tokio::sync::watch;
+use tokio::task::JoinHandle;
+
+/// A single logical direction of a proxied connection.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Direction {
+    /// Bytes travelling from the connecting client towards the target server
+    /// (the forward / request path).
+    ClientToServer,
+    /// Bytes travelling from the target server back to the client (the return
+    /// / response path).
+    ServerToClient,
+}
+
+/// Runtime-switchable forwarding behaviour of a [`FaultProxy`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FaultMode {
+    /// Forward bytes transparently in both directions.
+    Pass,
+    /// Forward bytes in both directions, but delay every chunk by this
+    /// duration before it is written onward.
+    Latency(Duration),
+    /// Accept connections but forward nothing in either direction and never
+    /// respond. Bytes read from either side are drained and discarded so the
+    /// proxy never blocks or panics; the peer simply never hears back.
+    Blackhole,
+    /// Block exactly one direction (drop its bytes) while the other direction
+    /// keeps forwarding normally.
+    Partition(Direction),
+}
+
+/// An async TCP proxy that forwards a random local port to a fixed target and
+/// can inject network faults at runtime.
+///
+/// The proxy owns a background accept loop; each accepted connection is
+/// handled by its own task pair (one per direction). [`FaultProxy::shutdown`]
+/// stops the accept loop and drops the listener, releasing the bound port.
+pub struct FaultProxy {
+    listen_addr: SocketAddr,
+    target: SocketAddr,
+    mode_tx: watch::Sender<FaultMode>,
+    shutdown_tx: watch::Sender<bool>,
+    accept_task: JoinHandle<()>,
+}
+
+impl FaultProxy {
+    /// Bind a listener on `127.0.0.1:0` and start forwarding accepted
+    /// connections to `target`. Starts in [`FaultMode::Pass`].
+    pub async fn start(target: SocketAddr) -> io::Result<Self> {
+        let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
+        let listen_addr = listener.local_addr()?;
+
+        let (mode_tx, mode_rx) = watch::channel(FaultMode::Pass);
+        let (shutdown_tx, shutdown_rx) = watch::channel(false);
+
+        let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx));
+
+        Ok(Self {
+            listen_addr,
+            target,
+            mode_tx,
+            shutdown_tx,
+            accept_task,
+        })
+    }
+
+    /// Local address the proxy is listening on. Point clients here instead of
+    /// the real target to route their traffic through the proxy.
+    pub fn local_addr(&self) -> SocketAddr {
+        self.listen_addr
+    }
+
+    /// The fixed target address every connection is forwarded to.
+    pub fn target_addr(&self) -> SocketAddr {
+        self.target
+    }
+
+    /// The mode currently in effect.
+    pub fn mode(&self) -> FaultMode {
+        *self.mode_tx.borrow()
+    }
+
+    /// Switch the fault mode at runtime. Takes effect on the next chunk read by
+    /// each direction of every live and future connection.
+    pub fn set_mode(&self, mode: FaultMode) {
+        // A send only fails if every receiver has dropped, i.e. the accept loop
+        // and all connections have already ended; there is nothing to steer.
+        let _ = self.mode_tx.send(mode);
+    }
+
+    /// Stop the accept loop, signal live connections to wind down, and drop the
+    /// listener so the bound port is released. Awaits the accept loop so the
+    /// port is free once this returns.
+    pub async fn shutdown(self) {
+        let _ = self.shutdown_tx.send(true);
+        let _ = self.accept_task.await;
+    }
+}
+
+/// Accept loop: takes new connections until shutdown is signalled, then returns
+/// (dropping `listener`, which releases the port).
+async fn accept_loop(
+    listener: TcpListener,
+    target: SocketAddr,
+    mode_rx: watch::Receiver<FaultMode>,
+    mut shutdown_rx: watch::Receiver<bool>,
+) {
+    loop {
+        tokio::select! {
+            biased;
+            _ = shutdown_rx.changed() => {
+                if *shutdown_rx.borrow() {
+                    break;
+                }
+            }
+            accepted = listener.accept() => {
+                match accepted {
+                    Ok((client, _peer)) => {
+                        tokio::spawn(handle_connection(
+                            client,
+                            target,
+                            mode_rx.clone(),
+                            shutdown_rx.clone(),
+                        ));
+                    }
+                    // A transient accept error should not tear the proxy down.
+                    Err(_) => continue,
+                }
+            }
+        }
+    }
+}
+
+/// Bridge one accepted client connection to a fresh connection to the target,
+/// running an independent pump per direction.
+async fn handle_connection(
+    client: TcpStream,
+    target: SocketAddr,
+    mode_rx: watch::Receiver<FaultMode>,
+    shutdown_rx: watch::Receiver<bool>,
+) {
+    let server = match TcpStream::connect(target).await {
+        Ok(s) => s,
+        // Target unreachable: nothing to bridge; drop the client.
+        Err(_) => return,
+    };
+
+    let (client_rd, client_wr) = client.into_split();
+    let (server_rd, server_wr) = server.into_split();
+
+    let up = tokio::spawn(pump(
+        Direction::ClientToServer,
+        client_rd,
+        server_wr,
+        mode_rx.clone(),
+        shutdown_rx.clone(),
+    ));
+    let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx));
+
+    let _ = up.await;
+    let _ = down.await;
+}
+
+/// Copy bytes from `from` to `to` for a single direction, applying the current
+/// [`FaultMode`]. Returns when the source hits EOF/error, a write fails, or
+/// shutdown is signalled.
+async fn pump<R, W>(
+    dir: Direction,
+    mut from: R,
+    mut to: W,
+    mut mode_rx: watch::Receiver<FaultMode>,
+    mut shutdown_rx: watch::Receiver<bool>,
+) where
+    R: AsyncReadExt + Unpin,
+    W: AsyncWriteExt + Unpin,
+{
+    let mut buf = vec![0u8; 16 * 1024];
+    loop {
+        let n = tokio::select! {
+            biased;
+            _ = shutdown_rx.changed() => break,
+            read = from.read(&mut buf) => match read {
+                Ok(0) => break,        // clean EOF
+                Ok(n) => n,
+                Err(_) => break,       // reset / error: end this direction
+            },
+        };
+
+        // Snapshot the mode without holding the borrow across an await.
+        let mode = *mode_rx.borrow_and_update();
+        match mode {
+            // Whole connection is a black hole: drain and drop.
+            FaultMode::Blackhole => continue,
+            // This direction is partitioned off: drop its bytes; the peer keeps
+            // reading nothing while the other direction may still flow.
+            FaultMode::Partition(blocked) if blocked == dir => continue,
+            FaultMode::Latency(delay) => {
+                tokio::time::sleep(delay).await;
+                if to.write_all(&buf[..n]).await.is_err() {
+                    break;
+                }
+                if to.flush().await.is_err() {
+                    break;
+                }
+            }
+            FaultMode::Pass | FaultMode::Partition(_) => {
+                if to.write_all(&buf[..n]).await.is_err() {
+                    break;
+                }
+                if to.flush().await.is_err() {
+                    break;
+                }
+            }
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::Arc;
+    use std::time::Instant;
+    use tokio::sync::Mutex;
+
+    /// A loopback echo server that records every byte it receives, so tests can
+    /// distinguish "the server never got it" (client->server blocked) from "the
+    /// server got it but the reply never came back" (server->client blocked).
+    struct EchoServer {
+        addr: SocketAddr,
+        received: Arc<Mutex<Vec<u8>>>,
+        _task: JoinHandle<()>,
+    }
+
+    impl EchoServer {
+        async fn start() -> io::Result<Self> {
+            let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
+            let addr = listener.local_addr()?;
+            let received = Arc::new(Mutex::new(Vec::new()));
+            let received_task = received.clone();
+            let task = tokio::spawn(async move {
+                loop {
+                    let Ok((mut sock, _)) = listener.accept().await else {
+                        break;
+                    };
+                    let received_conn = received_task.clone();
+                    tokio::spawn(async move {
+                        let mut buf = vec![0u8; 16 * 1024];
+                        loop {
+                            match sock.read(&mut buf).await {
+                                Ok(0) | Err(_) => break,
+                                Ok(n) => {
+                                    received_conn.lock().await.extend_from_slice(&buf[..n]);
+                                    if sock.write_all(&buf[..n]).await.is_err() {
+                                        break;
+                                    }
+                                    let _ = sock.flush().await;
+                                }
+                            }
+                        }
+                    });
+                }
+            });
+            Ok(Self {
+                addr,
+                received,
+                _task: task,
+            })
+        }
+
+        async fn received(&self) -> Vec<u8> {
+            self.received.lock().await.clone()
+        }
+    }
+
+    /// Write `payload`, then try to read up to `want` bytes with a deadline.
+    /// Returns the bytes actually received before the timeout (possibly empty).
+    async fn write_then_read(stream: &mut TcpStream, payload: &[u8], want: usize, timeout: Duration) -> Vec<u8> {
+        stream.write_all(payload).await.expect("client write");
+        stream.flush().await.expect("client flush");
+
+        let mut out = Vec::new();
+        let mut buf = vec![0u8; want.max(1)];
+        let deadline = tokio::time::Instant::now() + timeout;
+        while out.len() < want {
+            let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
+            if remaining.is_zero() {
+                break;
+            }
+            match tokio::time::timeout(remaining, stream.read(&mut buf)).await {
+                Ok(Ok(0)) => break, // peer closed
+                Ok(Ok(n)) => out.extend_from_slice(&buf[..n]),
+                Ok(Err(_)) => break, // connection error
+                Err(_) => break,     // read timed out
+            }
+        }
+        out
+    }
+
+    #[tokio::test]
+    async fn pass_mode_round_trips_bytes() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        assert_eq!(proxy.mode(), FaultMode::Pass);
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"hello-fault-proxy";
+        let got = write_then_read(&mut client, payload, payload.len(), Duration::from_secs(5)).await;
+
+        assert_eq!(got, payload, "pass mode must forward bytes unchanged");
+        assert_eq!(echo.received().await, payload, "server must have received the bytes");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn latency_mode_delays_forwarding() {
+        // Loose lower bound only: sleep() guarantees *at least* the delay, so
+        // the assertion cannot flake on a slow machine, and we never assert an
+        // upper bound.
+        let delay = Duration::from_millis(200);
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Latency(delay));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"delayed";
+        let start = Instant::now();
+        let got = write_then_read(&mut client, payload, payload.len(), Duration::from_secs(5)).await;
+        let elapsed = start.elapsed();
+
+        assert_eq!(got, payload, "latency mode must still deliver the bytes");
+        // One delay on the request path plus one on the echo return path: the
+        // round trip must exceed at least a single configured delay.
+        assert!(elapsed >= delay, "round trip {elapsed:?} shorter than configured delay {delay:?}");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn blackhole_mode_yields_no_response_and_no_panic() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Blackhole);
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let got = write_then_read(&mut client, b"into-the-void", 1, Duration::from_millis(300)).await;
+
+        assert!(got.is_empty(), "blackhole mode must not return any bytes, got {got:?}");
+        assert!(echo.received().await.is_empty(), "blackhole must not forward to the server");
+
+        // Proxy is still alive and steerable after a blackholed connection.
+        proxy.set_mode(FaultMode::Pass);
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn partition_client_to_server_blocks_request_path() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Partition(Direction::ClientToServer));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let got = write_then_read(&mut client, b"blocked-request", 1, Duration::from_millis(300)).await;
+
+        assert!(got.is_empty(), "client->server partition must yield no echo");
+        assert!(
+            echo.received().await.is_empty(),
+            "client->server partition must stop bytes from reaching the server"
+        );
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn partition_server_to_client_blocks_only_return_path() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Partition(Direction::ServerToClient));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"one-way";
+        let got = write_then_read(&mut client, payload, 1, Duration::from_millis(300)).await;
+
+        // Client hears nothing back...
+        assert!(got.is_empty(), "server->client partition must block the reply");
+        // ...but the request path is live, so the server did receive the bytes.
+        // Poll briefly to avoid racing the forward direction.
+        let mut server_saw = Vec::new();
+        for _ in 0..30 {
+            server_saw = echo.received().await;
+            if !server_saw.is_empty() {
+                break;
+            }
+            tokio::time::sleep(Duration::from_millis(10)).await;
+        }
+        assert_eq!(server_saw, payload, "server->client partition must leave the request path intact");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn set_mode_switches_behaviour_at_runtime() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+
+        // Start passing: first exchange round-trips.
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let first = write_then_read(&mut client, b"first", 5, Duration::from_secs(5)).await;
+        assert_eq!(first, b"first", "initial pass exchange must round-trip");
+
+        // Flip to blackhole on the same live connection: the next write gets no reply.
+        proxy.set_mode(FaultMode::Blackhole);
+        let second = write_then_read(&mut client, b"second", 1, Duration::from_millis(300)).await;
+        assert!(second.is_empty(), "after switching to blackhole the live connection must go silent");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn shutdown_releases_the_listener_port() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        let addr = proxy.local_addr();
+
+        proxy.shutdown().await;
+
+        // The port must be free to bind again once shutdown returns.
+        let rebind = TcpListener::bind(addr).await;
+        assert!(rebind.is_ok(), "shutdown must release the listener port {addr}");
+    }
+}

crates/e2e_test/src/lib.rs

@@ -23,6 +23,18 @@ pub mod common;
 #[cfg(test)]
 pub mod chaos;
 
+// Programmable S3 target for replication failure-path tests (backlog#1147 repl-8).
+#[cfg(test)]
+pub mod fake_s3_target;
+
+// Socket-level network fault-injection proxy for black-box cluster tests
+// (backlog#1325 network fault-injection block): latency / blackhole / one-way
+// partition on the wire between nodes. Serves #1312/#1319 (lock-plane one-way
+// partition, accept-then-blackhole peer) and #1327 (cross-process replay/tamper
+// seam); cluster-harness wiring is a follow-up (rustfs#4937).
+#[cfg(test)]
+pub mod fault_proxy;
+
 // Reliability tests built on the fault-injection harness
 #[cfg(test)]
 mod reliability_disk_fault_test;
@@ -147,6 +159,10 @@ mod object_lock;
 #[cfg(test)]
 mod cluster_concurrency_test;
 
+// Multi-drive (drivesPerNode) and 2-pool cluster harness smoke tests
+#[cfg(test)]
+mod cluster_multidrive_pool_test;
+
 // PutObject / MultipartUpload with checksum (Content-MD5, x-amz-checksum-*)
 #[cfg(test)]
 mod checksum_upload_test;

crates/ecstore/src/api/mod.rs

@@ -134,7 +134,7 @@ pub mod bucket {
     }
 
     pub mod quota {
-        pub use crate::bucket::quota::{BucketQuota, QuotaError, QuotaOperation};
+        pub use crate::bucket::quota::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
 
         pub mod checker {
             pub use crate::bucket::quota::checker::QuotaChecker;

crates/ecstore/src/disk/disk_store.rs

@@ -217,6 +217,27 @@ pub fn get_object_disk_read_timeout() -> Duration {
     )
 }
 
+/// Per-shard erasure write stall budget: a shard write (or shutdown) that makes
+/// no forward progress for this long is failed and its disk dropped before
+/// commit. Re-armed on every shard write, so it bounds a stall rather than the
+/// whole transfer. `0` disables the deadline (wait indefinitely).
+pub fn get_object_disk_write_stall_timeout() -> Duration {
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT,
+        rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT,
+    ))
+}
+
+/// Optional absolute per-object erasure write cap (administrator slow-drip
+/// backstop). `0` (default) disables the cap; the per-shard stall timeout is the
+/// primary guarantee.
+pub fn get_object_disk_write_absolute_cap() -> Duration {
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP,
+        rustfs_config::DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP,
+    ))
+}
+
 pub fn get_drive_active_check_interval() -> Duration {
     Duration::from_secs(rustfs_utils::get_env_u64(
         rustfs_config::ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS,
@@ -1653,6 +1674,32 @@ mod tests {
         });
     }
 
+    #[test]
+    fn object_disk_write_stall_timeout_default_and_override() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, || {
+            assert_eq!(
+                get_object_disk_write_stall_timeout(),
+                Duration::from_secs(rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT)
+            );
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("9"), || {
+            assert_eq!(get_object_disk_write_stall_timeout(), Duration::from_secs(9));
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("0"), || {
+            assert!(get_object_disk_write_stall_timeout().is_zero(), "0 disables the stall deadline");
+        });
+    }
+
+    #[test]
+    fn object_disk_write_absolute_cap_defaults_disabled() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, || {
+            assert!(get_object_disk_write_absolute_cap().is_zero(), "absolute cap is disabled by default");
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, Some("120"), || {
+            assert_eq!(get_object_disk_write_absolute_cap(), Duration::from_secs(120));
+        });
+    }
+
     #[test]
     fn object_disk_read_timeout_uses_default_when_unset() {
         temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_READ_TIMEOUT, || {

crates/ecstore/src/disk/local.rs

@@ -26,10 +26,7 @@ use crate::disk::{
     error::{DiskError, Error, FileAccessDeniedWithContext, Result},
     error_conv::{to_access_error, to_file_error, to_unformatted_disk_error, to_volume_error},
     format::FormatV3,
-    fs::{
-        O_APPEND, O_CREATE, O_RDONLY, O_TRUNC, O_WRONLY, access, access_std, lstat, lstat_std, remove, remove_all_std,
-        remove_std, rename,
-    },
+    fs::{O_APPEND, O_CREATE, O_RDONLY, O_TRUNC, O_WRONLY, access, lstat, lstat_std, remove, remove_all_std, remove_std, rename},
     os,
     os::{check_path_length, is_empty_dir, is_root_disk, rename_all, rename_all_ignore_missing_source},
 };
@@ -101,7 +98,7 @@ fn inline_metadata_rollback_dir(version_id: Uuid, meta: &FileMeta) -> Uuid {
 fn remove_file_if_exists(path: &Path) -> std::io::Result<()> {
     match std::fs::remove_file(path) {
         Ok(()) => Ok(()),
-        Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
+        Err(err) if err.kind() == ErrorKind::NotFound => Ok(()),
         Err(err) => Err(err),
     }
 }
@@ -109,7 +106,7 @@ fn remove_file_if_exists(path: &Path) -> std::io::Result<()> {
 fn remove_dir_all_if_exists(path: &Path) -> std::io::Result<()> {
     match std::fs::remove_dir_all(path) {
         Ok(()) => Ok(()),
-        Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
+        Err(err) if err.kind() == ErrorKind::NotFound => Ok(()),
         Err(err) => Err(err),
     }
 }
@@ -121,7 +118,7 @@ fn rollback_committed_rename_std(
 ) -> std::io::Result<()> {
     if let Some(old_data_dir) = rollback_data_dir {
         let Some(dst_parent) = dst_file_path.parent() else {
-            return Err(std::io::Error::new(std::io::ErrorKind::InvalidInput, "missing object metadata parent"));
+            return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing object metadata parent"));
         };
         let backup_path = dst_parent.join(old_data_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP);
         std::fs::rename(backup_path, dst_file_path)?;
@@ -591,7 +588,7 @@ impl DurabilityMode {
             "strict" => Some(Self::Strict),
             "relaxed" => Some(Self::Relaxed),
             "none" => Some(Self::None),
-            _ => Option::None,
+            _ => None,
         }
     }
 
@@ -799,7 +796,7 @@ pub(crate) mod bucket_durability {
 /// configured durability mode: their contents commit into user buckets and
 /// are exactly the writes the relaxed tiers exist for.
 fn is_scratch_volume(volume: &str) -> bool {
-    for scratch in [super::RUSTFS_META_TMP_BUCKET, super::RUSTFS_META_MULTIPART_BUCKET] {
+    for scratch in [RUSTFS_META_TMP_BUCKET, super::RUSTFS_META_MULTIPART_BUCKET] {
         if volume == scratch || volume.strip_prefix(scratch).is_some_and(|rest| rest.starts_with('/')) {
             return true;
         }
@@ -816,7 +813,7 @@ fn is_system_critical_volume(volume: &str) -> bool {
     if is_scratch_volume(volume) {
         return false;
     }
-    for meta in [super::RUSTFS_META_BUCKET, super::MIGRATING_META_BUCKET] {
+    for meta in [RUSTFS_META_BUCKET, super::MIGRATING_META_BUCKET] {
         if volume == meta || volume.strip_prefix(meta).is_some_and(|rest| rest.starts_with('/')) {
             return true;
         }
@@ -935,12 +932,12 @@ struct AlignedBuf {
 impl AlignedBuf {
     fn new(len: usize, align: usize) -> std::io::Result<Self> {
         debug_assert!(len > 0, "AlignedBuf must not be zero-sized");
-        let layout = std::alloc::Layout::from_size_align(len, align)
-            .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidInput, e))?;
+        let layout =
+            std::alloc::Layout::from_size_align(len, align).map_err(|e| std::io::Error::new(ErrorKind::InvalidInput, e))?;
         // SAFETY: `layout` has non-zero size (callers guarantee len > 0) and a
         // valid power-of-two alignment enforced by Layout::from_size_align.
         let ptr = unsafe { std::alloc::alloc_zeroed(layout) };
-        let ptr = std::ptr::NonNull::new(ptr).ok_or(std::io::ErrorKind::OutOfMemory)?;
+        let ptr = std::ptr::NonNull::new(ptr).ok_or(ErrorKind::OutOfMemory)?;
         Ok(Self { ptr, len, layout })
     }
 
@@ -994,10 +991,9 @@ fn pread_direct_aligned(file_path: &Path, offset: u64, length: usize, state: &Di
     let align_u64 = align as u64;
 
     let aligned_offset = offset - (offset % align_u64);
-    let logical_start =
-        usize::try_from(offset - aligned_offset).map_err(|_| std::io::Error::from(std::io::ErrorKind::InvalidInput))?;
-    let logical_end = logical_start.checked_add(length).ok_or(std::io::ErrorKind::InvalidInput)?;
-    let aligned_len = logical_end.checked_add(align - 1).ok_or(std::io::ErrorKind::InvalidInput)? / align * align;
+    let logical_start = usize::try_from(offset - aligned_offset).map_err(|_| std::io::Error::from(ErrorKind::InvalidInput))?;
+    let logical_end = logical_start.checked_add(length).ok_or(ErrorKind::InvalidInput)?;
+    let aligned_len = logical_end.checked_add(align - 1).ok_or(ErrorKind::InvalidInput)? / align * align;
 
     let mut buf = AlignedBuf::new(aligned_len, align)?;
 
@@ -1012,7 +1008,7 @@ fn pread_direct_aligned(file_path: &Path, offset: u64, length: usize, state: &Di
         filled += n;
     }
     if filled < logical_end {
-        return Err(std::io::Error::new(std::io::ErrorKind::UnexpectedEof, "short O_DIRECT read"));
+        return Err(std::io::Error::new(ErrorKind::UnexpectedEof, "short O_DIRECT read"));
     }
 
     Ok(Bytes::copy_from_slice(&buf.as_slice()[logical_start..logical_end]))
@@ -1094,10 +1090,7 @@ fn pwrite_all(file: &std::fs::File, mut buf: &[u8], mut offset: u64) -> std::io:
     while !buf.is_empty() {
         let n = file.write_at(buf, offset)?;
         if n == 0 {
-            return Err(std::io::Error::new(
-                std::io::ErrorKind::WriteZero,
-                "O_DIRECT positioned write wrote 0 bytes",
-            ));
+            return Err(std::io::Error::new(ErrorKind::WriteZero, "O_DIRECT positioned write wrote 0 bytes"));
         }
         buf = &buf[n..];
         offset += n as u64;
@@ -2128,7 +2121,8 @@ impl LocalIoBackend for StdBackend {
                 let access_check_start = metrics_enabled.then(StdInstant::now);
                 let volume_dir = local_disk_bucket_path(&root, &volume_owned)?;
                 if !skip_access_checks(&volume_owned) {
-                    access_std(&volume_dir).map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
+                    crate::disk::fs::access_std(&volume_dir)
+                        .map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
                 }
                 let access_check_duration = access_check_start.map_or(StdDuration::ZERO, |started_at| started_at.elapsed());
 
@@ -2948,7 +2942,8 @@ fn is_io_uring_unsupported(err: &std::io::Error) -> bool {
 fn resolve_uring_object_path(root: &Path, volume: &str, path: &str) -> Result<PathBuf> {
     let volume_dir = local_disk_bucket_path(root, volume)?;
     if !skip_access_checks(volume) {
-        access_std(&volume_dir).map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
+        crate::disk::fs::access_std(&volume_dir)
+            .map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
     }
     let file_path = local_disk_object_path(root, volume, path)?;
     check_path_length(file_path.to_string_lossy().as_ref())?;
@@ -3797,10 +3792,9 @@ impl LocalDisk {
                         // rebuilding (see rustfs-heal); surface it so scanner
                         // coordination, lock selection and admin/metrics see
                         // the rebuild. Refreshed with this cache (~1s).
-                        let healing =
-                            tokio::fs::try_exists(root.join(super::RUSTFS_META_BUCKET).join(super::HEALING_MARKER_PATH))
-                                .await
-                                .unwrap_or(false);
+                        let healing = tokio::fs::try_exists(root.join(RUSTFS_META_BUCKET).join(super::HEALING_MARKER_PATH))
+                            .await
+                            .unwrap_or(false);
                         let disk_info = DiskInfo {
                             total: info.total,
                             free: info.free,
@@ -4780,8 +4774,8 @@ impl LocalDisk {
         let file_path = self.get_object_path(volume, path)?;
         check_path_length(file_path.to_string_lossy().as_ref())?;
 
-        let tmp_volume_dir = self.get_bucket_path(super::RUSTFS_META_TMP_BUCKET)?;
-        let tmp_file_path = self.get_object_path(super::RUSTFS_META_TMP_BUCKET, Uuid::new_v4().to_string().as_str())?;
+        let tmp_volume_dir = self.get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
+        let tmp_file_path = self.get_object_path(RUSTFS_META_TMP_BUCKET, Uuid::new_v4().to_string().as_str())?;
 
         let durability = effective_durability(volume);
 
@@ -5641,7 +5635,7 @@ async fn read_file_metadata(p: impl AsRef<Path>) -> Result<Metadata> {
 fn skip_access_checks(p: impl AsRef<str>) -> bool {
     let vols = [
         RUSTFS_META_TMP_DELETED_BUCKET,
-        super::RUSTFS_META_TMP_BUCKET,
+        RUSTFS_META_TMP_BUCKET,
         super::RUSTFS_META_MULTIPART_BUCKET,
         RUSTFS_META_BUCKET,
     ];
@@ -5952,12 +5946,11 @@ impl DiskAPI for LocalDisk {
         );
         for (i, part) in fi.parts.iter().enumerate() {
             let checksum_info = erasure.get_checksum_info(part.number);
-            let checksum_algo =
-                if fi.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
-                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
-                } else {
-                    checksum_info.algorithm
-                };
+            let checksum_algo = if fi.uses_legacy_checksum && checksum_info.algorithm == HashAlgorithm::HighwayHash256S {
+                HashAlgorithm::HighwayHash256SLegacy
+            } else {
+                checksum_info.algorithm
+            };
             let part_path = self.get_object_path(
                 volume,
                 path_join_buf(&[
@@ -6985,7 +6978,7 @@ impl DiskAPI for LocalDisk {
                 // Read existing xl.meta
                 let has_dst_buf = match std::fs::read(&dst) {
                     Ok(buf) => Some(Bytes::from(buf)),
-                    Err(e) if e.kind() == std::io::ErrorKind::NotFound => None,
+                    Err(e) if e.kind() == ErrorKind::NotFound => None,
                     Err(e) => return Err(to_file_error(e)),
                 };
 
@@ -7077,8 +7070,8 @@ impl DiskAPI for LocalDisk {
 
                 match std::fs::rename(&src, &dst) {
                     Ok(()) => Ok(()),
-                    Err(err) if err.kind() == std::io::ErrorKind::NotFound && !src.exists() => Ok(()),
-                    Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
+                    Err(err) if err.kind() == ErrorKind::NotFound && !src.exists() => Ok(()),
+                    Err(err) if err.kind() == ErrorKind::NotFound => {
                         if let Some(parent) = dst.parent() {
                             std::fs::create_dir_all(parent)?;
                         }
@@ -7126,7 +7119,7 @@ impl DiskAPI for LocalDisk {
                     }
                 }
 
-                Ok::<(Option<uuid::Uuid>, Option<Vec<u8>>, Option<OldCurrentSize>), std::io::Error>((
+                Ok::<(Option<Uuid>, Option<Vec<u8>>, Option<OldCurrentSize>), std::io::Error>((
                     rollback_data_dir,
                     version_signature,
                     old_current_size,
@@ -8110,7 +8103,7 @@ mod test {
 
     /// Crash-consistency harness for the rename_data commit sequence
     /// (rustfs/backlog#935 HP-14, test plan rustfs/backlog#896; hard rule from
-    /// rustfs/backlog#878: "partial commit 后对象只能是旧版本或新版本,不能混合").
+    /// rustfs/backlog#878: "After partial commit, the object can only be an old or new version; it cannot be mixed").
     ///
     /// For every pre-commit crash point × durability tier, it seeds a committed
     /// object, stages a replacement, injects a hard power loss (no in-process
@@ -8449,7 +8442,7 @@ mod test {
     /// Backdate a path's mtime so zero-expiry cleanup tests classify it as
     /// stale deterministically, instead of sleeping and hoping the filesystem
     /// timestamp granularity (or a backward wall-clock step) cooperates.
-    fn backdate_mtime(path: &std::path::Path, age: Duration) {
+    fn backdate_mtime(path: &Path, age: Duration) {
         use std::fs::{File, FileTimes};
         let mtime = std::time::SystemTime::now() - age;
         File::open(path)
@@ -8858,17 +8851,13 @@ mod test {
     async fn blocking_scan_writer_keeps_flush_and_shutdown_pending() {
         let mut flush_writer = BlockingScanWriter { entered_tx: None };
         assert!(
-            tokio::time::timeout(Duration::from_millis(10), flush_writer.flush())
-                .await
-                .is_err(),
+            timeout(Duration::from_millis(10), flush_writer.flush()).await.is_err(),
             "blocking scan writer flush should stay pending"
         );
 
         let mut shutdown_writer = BlockingScanWriter { entered_tx: None };
         assert!(
-            tokio::time::timeout(Duration::from_millis(10), shutdown_writer.shutdown())
-                .await
-                .is_err(),
+            timeout(Duration::from_millis(10), shutdown_writer.shutdown()).await.is_err(),
             "blocking scan writer shutdown should stay pending"
         );
     }
@@ -8877,15 +8866,11 @@ mod test {
     /// consumer (quorum merge, a lagging peer drive).
     struct SlowWriter {
         delay: Duration,
-        sleep: Option<std::pin::Pin<Box<Sleep>>>,
+        sleep: Option<Pin<Box<Sleep>>>,
     }
 
     impl AsyncWrite for SlowWriter {
-        fn poll_write(
-            mut self: std::pin::Pin<&mut Self>,
-            cx: &mut std::task::Context<'_>,
-            buf: &[u8],
-        ) -> std::task::Poll<std::io::Result<usize>> {
+        fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8]) -> Poll<io::Result<usize>> {
             if self.sleep.is_none() {
                 let delay = self.delay;
                 self.sleep = Some(Box::pin(tokio::time::sleep(delay)));
@@ -8893,23 +8878,20 @@ mod test {
 
             let sleep = self.sleep.as_mut().expect("sleep was just installed");
             match sleep.as_mut().poll(cx) {
-                std::task::Poll::Ready(()) => {
+                Poll::Ready(()) => {
                     self.sleep = None;
-                    std::task::Poll::Ready(Ok(buf.len()))
+                    Poll::Ready(Ok(buf.len()))
                 }
-                std::task::Poll::Pending => std::task::Poll::Pending,
+                Poll::Pending => Poll::Pending,
             }
         }
 
-        fn poll_flush(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll<std::io::Result<()>> {
-            std::task::Poll::Ready(Ok(()))
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
         }
 
-        fn poll_shutdown(
-            self: std::pin::Pin<&mut Self>,
-            _cx: &mut std::task::Context<'_>,
-        ) -> std::task::Poll<std::io::Result<()>> {
-            std::task::Poll::Ready(Ok(()))
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
         }
     }
 
@@ -9605,14 +9587,14 @@ mod test {
         bucket_durability::set(super::super::RUSTFS_META_MULTIPART_BUCKET, Some(DurabilityMode::Relaxed));
         bucket_durability::set("", Some(DurabilityMode::Relaxed));
 
-        assert_eq!(bucket_durability::lookup(RUSTFS_META_BUCKET), Option::None);
-        assert_eq!(bucket_durability::lookup(RUSTFS_META_TMP_BUCKET), Option::None);
+        assert_eq!(bucket_durability::lookup(RUSTFS_META_BUCKET), None);
+        assert_eq!(bucket_durability::lookup(RUSTFS_META_TMP_BUCKET), None);
         assert_eq!(effective_durability(RUSTFS_META_BUCKET), DurabilityMode::Strict);
 
         // The legacy full-off tier is process-wide only: registering it per
         // bucket is dropped, not stored.
         bucket_durability::set("hp5b-legacy-refused", Some(DurabilityMode::LegacyOff));
-        assert_eq!(bucket_durability::lookup("hp5b-legacy-refused"), Option::None);
+        assert_eq!(bucket_durability::lookup("hp5b-legacy-refused"), None);
     }
 
     #[test]
@@ -11071,7 +11053,7 @@ mod test {
 
         let vols = [
             RUSTFS_META_TMP_DELETED_BUCKET,
-            super::super::RUSTFS_META_TMP_BUCKET,
+            RUSTFS_META_TMP_BUCKET,
             super::super::RUSTFS_META_MULTIPART_BUCKET,
             RUSTFS_META_BUCKET,
         ];
@@ -11110,7 +11092,7 @@ mod test {
         let mut reader = StallTimeoutReader::new(PendingTestReader, Duration::ZERO);
         let mut buf = [0; 1];
 
-        let result = tokio::time::timeout(Duration::from_millis(10), reader.read(&mut buf)).await;
+        let result = timeout(Duration::from_millis(10), reader.read(&mut buf)).await;
 
         assert!(result.is_err(), "zero timeout must leave stalled reads pending instead of failing");
     }
@@ -11240,10 +11222,10 @@ mod test {
 
     #[tokio::test(start_paused = true)]
     async fn cleanup_loop_interval_does_not_tick_immediately() {
-        let start_at = tokio::time::Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
+        let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
         let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL);
 
-        assert!(tokio::time::timeout(Duration::from_secs(1), interval.tick()).await.is_err());
+        assert!(timeout(Duration::from_secs(1), interval.tick()).await.is_err());
 
         tokio::time::advance(DELETED_OBJECTS_CLEANUP_INTERVAL).await;
         interval.tick().await;
@@ -11291,26 +11273,16 @@ mod test {
         struct BrokenPipeWriter;
 
         impl AsyncWrite for BrokenPipeWriter {
-            fn poll_write(
-                self: std::pin::Pin<&mut Self>,
-                _cx: &mut std::task::Context<'_>,
-                _buf: &[u8],
-            ) -> std::task::Poll<std::io::Result<usize>> {
-                std::task::Poll::Ready(Err(std::io::Error::new(std::io::ErrorKind::BrokenPipe, "closed")))
+            fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<io::Result<usize>> {
+                Poll::Ready(Err(io::Error::new(ErrorKind::BrokenPipe, "closed")))
             }
 
-            fn poll_flush(
-                self: std::pin::Pin<&mut Self>,
-                _cx: &mut std::task::Context<'_>,
-            ) -> std::task::Poll<std::io::Result<()>> {
-                std::task::Poll::Ready(Ok(()))
+            fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+                Poll::Ready(Ok(()))
             }
 
-            fn poll_shutdown(
-                self: std::pin::Pin<&mut Self>,
-                _cx: &mut std::task::Context<'_>,
-            ) -> std::task::Poll<std::io::Result<()>> {
-                std::task::Poll::Ready(Ok(()))
+            fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+                Poll::Ready(Ok(()))
             }
         }
 
@@ -12442,7 +12414,7 @@ mod test {
         fs::set_permissions(&meta_path, Permissions::from_mode(0o000))
             .await
             .expect("operation should succeed");
-        if fs::File::open(&meta_path).await.is_ok() {
+        if File::open(&meta_path).await.is_ok() {
             fs::set_permissions(&meta_path, original_permissions)
                 .await
                 .expect("operation should succeed");
@@ -12977,7 +12949,7 @@ mod test {
         // It only checks length and platform-specific special characters
         // System volume names are valid according to the current implementation
         assert!(LocalDisk::is_valid_volname(RUSTFS_META_BUCKET));
-        assert!(LocalDisk::is_valid_volname(super::super::RUSTFS_META_TMP_BUCKET));
+        assert!(LocalDisk::is_valid_volname(RUSTFS_META_TMP_BUCKET));
 
         // Testing platform-specific behavior for special characters
         #[cfg(windows)]
@@ -13321,24 +13293,20 @@ mod test {
 
         #[cfg(unix)]
         {
-            for (kind, expected) in [("minor", 1), ("major", 2)] {
+            for (kind, expected) in [("minor", 1), ("major", 2)].iter().copied() {
+                let labels = [("path", "local_test"), ("stage", "mmap_map"), ("kind", kind)];
                 assert_eq!(
-                    recorder.counter_value(
-                        METRIC_GET_OBJECT_MMAP_PAGE_FAULTS_TOTAL,
-                        &[("path", "local_test"), ("stage", "mmap_map"), ("kind", kind)]
-                    ),
+                    recorder.counter_value(METRIC_GET_OBJECT_MMAP_PAGE_FAULTS_TOTAL, &labels),
                     expected,
-                    "zero deltas must not emit and positive {kind} deltas must accumulate exactly"
+                    "zero deltas must not emit and positive mmap page fault deltas must accumulate exactly"
                 );
             }
-            for (kind, expected) in [("minor", 3), ("major", 4)] {
+            for (kind, expected) in [("minor", 3), ("major", 4)].iter().copied() {
+                let labels = [("path", "local_test"), ("stage", "direct_read_copy"), ("kind", kind)];
                 assert_eq!(
-                    recorder.counter_value(
-                        METRIC_GET_OBJECT_DIRECT_READ_PAGE_FAULTS_TOTAL,
-                        &[("path", "local_test"), ("stage", "direct_read_copy"), ("kind", kind)]
-                    ),
+                    recorder.counter_value(METRIC_GET_OBJECT_DIRECT_READ_PAGE_FAULTS_TOTAL, &labels),
                     expected,
-                    "zero deltas must not emit and positive {kind} deltas must accumulate exactly"
+                    "zero deltas must not emit and positive direct-read page fault deltas must accumulate exactly"
                 );
             }
         }
@@ -13553,15 +13521,15 @@ mod test {
 
     #[test]
     fn test_is_bitrot_size_mismatch_error_only_matches_target_message() {
-        assert!(is_bitrot_size_mismatch_error(&std::io::Error::other("bitrot shard file size mismatch")));
-        assert!(!is_bitrot_size_mismatch_error(&std::io::Error::other("bitrot hash mismatch")));
+        assert!(is_bitrot_size_mismatch_error(&io::Error::other("bitrot shard file size mismatch")));
+        assert!(!is_bitrot_size_mismatch_error(&io::Error::other("bitrot hash mismatch")));
     }
 
     #[test]
     fn test_is_bitrot_verification_error_matches_hash_and_size_mismatch() {
-        assert!(is_bitrot_verification_error(&std::io::Error::other("bitrot shard file size mismatch")));
-        assert!(is_bitrot_verification_error(&std::io::Error::other("bitrot hash mismatch")));
-        assert!(!is_bitrot_verification_error(&std::io::Error::other("unrelated io failure")));
+        assert!(is_bitrot_verification_error(&io::Error::other("bitrot shard file size mismatch")));
+        assert!(is_bitrot_verification_error(&io::Error::other("bitrot hash mismatch")));
+        assert!(!is_bitrot_verification_error(&io::Error::other("unrelated io failure")));
     }
 
     #[tokio::test]
@@ -13600,7 +13568,7 @@ mod test {
             ..Default::default()
         }];
 
-        let mut writer = BitrotWriter::new(std::io::Cursor::new(Vec::new()), file_info.erasure.shard_size(), checksum_algo);
+        let mut writer = BitrotWriter::new(io::Cursor::new(Vec::new()), file_info.erasure.shard_size(), checksum_algo);
         writer
             .write(&payload)
             .await
@@ -13667,7 +13635,7 @@ mod test {
             } else {
                 HashAlgorithm::HighwayHash256S
             };
-            let mut writer = BitrotWriter::new(std::io::Cursor::new(Vec::new()), codec_erasure.shard_size(), checksum_algo);
+            let mut writer = BitrotWriter::new(io::Cursor::new(Vec::new()), codec_erasure.shard_size(), checksum_algo);
             writer.write(&payload[..8]).await.expect("first shard block should encode");
             writer.write(&payload[8..]).await.expect("final shard block should encode");
             writer.shutdown().await.expect("bitrot writer should flush test payload");
@@ -14524,7 +14492,7 @@ mod test {
         // SAFETY: `map.as_ptr()` is page-aligned and `len` bytes long; `vec` has
         // one byte per page of that range, which is what mincore writes.
         let rc = unsafe { libc::mincore(map.as_ptr() as *mut libc::c_void, len, vec.as_mut_ptr()) };
-        assert_eq!(rc, 0, "mincore failed: {}", std::io::Error::last_os_error());
+        assert_eq!(rc, 0, "mincore failed: {}", io::Error::last_os_error());
         vec.iter().filter(|b| *b & 1 == 1).count()
     }
 

crates/ecstore/src/erasure/coding/decode.rs

@@ -40,6 +40,10 @@ use tracing::{error, warn};
 
 type ShardReadFuture<'a> = Pin<Box<dyn Future<Output = (usize, ShardReadCost, Result<Vec<u8>, Error>, bool)> + Send + 'a>>;
 
+/// One stripe's worth of shard buffers plus the per-shard read errors, as
+/// returned by `ParallelReader::read` / `read_stripe_timed`.
+type StripeReadOutput = (Vec<Option<Vec<u8>>>, Vec<Option<Error>>);
+
 const ENV_RUSTFS_SHARD_LOCALITY_SCHEDULING: &str = "RUSTFS_SHARD_LOCALITY_SCHEDULING";
 const ENV_RUSTFS_GET_SHARD_LOCALITY_PREFERENCE_ENABLE: &str = "RUSTFS_GET_SHARD_LOCALITY_PREFERENCE_ENABLE";
 const SHARD_LOCALITY_SCHEDULING_OFF: &str = "off";
@@ -1770,29 +1774,78 @@ impl Erasure {
 
                     if idx + 1 < blocks.len() {
                         // Overlap: read stripe idx+1 while reconstructing/emitting idx.
-                        let read_fut = read_stripe_timed(&mut reader, stage_metrics_enabled);
-                        let emit_fut = self.emit_decoded_stripe(
-                            writer,
-                            &mut shards,
-                            &errs,
-                            block_offset,
-                            block_length,
-                            &mut written,
-                            &mut ret_err,
-                            stage_metrics_enabled,
-                        );
-                        let (next, flow) = tokio::join!(read_fut, emit_fut);
+                        //
+                        // Cancel-safety (https://github.com/rustfs/backlog/issues/1310):
+                        // the prefetch read and the emit run concurrently, but if emit
+                        // terminates the loop — client disconnect or any emit error —
+                        // we must NOT keep waiting for the speculative next-stripe read.
+                        // A plain `tokio::join!` did exactly that: it drove both futures
+                        // to completion, so a `Stop` still blocked on the prefetch read,
+                        // which could stall for a whole shard-read deadline on a slow or
+                        // wedged remote shard before the GET could fail. Instead, drive
+                        // both with a biased `select!` and, the moment emit reports
+                        // `Stop`, drop the in-flight read future.
+                        //
+                        // Dropping the future is a real cancellation, not a detach: the
+                        // entire read pipeline is structured async (a `FuturesUnordered`
+                        // of `read_shard` futures inside `ParallelReader::read`, with no
+                        // `tokio::spawn`), so dropping `read_fut` drops every in-flight
+                        // shard read and propagates cancellation down to the
+                        // RemoteDisk / HTTP reader — no background read is left behind.
+                        //
+                        // The default path (`legacy_stripe_prefetch_enabled() == false`)
+                        // never reaches this branch, so this changes nothing for the
+                        // strictly-serial read → reconstruct → emit default.
+                        //
+                        // The `select!` is scoped so that both pinned futures are
+                        // dropped at the end of this inner block — before `reader` and
+                        // `shards` are borrowed again below. In the `Stop` case that
+                        // drop is what cancels the still-in-flight prefetch read.
+                        let (flow, next): (Option<StripeFlow>, Option<StripeReadOutput>) = {
+                            let read_fut = read_stripe_timed(&mut reader, stage_metrics_enabled);
+                            let emit_fut = self.emit_decoded_stripe(
+                                writer,
+                                &mut shards,
+                                &errs,
+                                block_offset,
+                                block_length,
+                                &mut written,
+                                &mut ret_err,
+                                stage_metrics_enabled,
+                            );
+                            tokio::pin!(read_fut);
+                            tokio::pin!(emit_fut);
+
+                            let mut next: Option<StripeReadOutput> = None;
+                            let mut flow: Option<StripeFlow> = None;
+                            // Poll until emit finishes. Once it has, only keep waiting for
+                            // the prefetch read when emit said `Continue` (that stripe is
+                            // still needed). On `Stop` the loop condition is false
+                            // immediately and we break; the pinned read future is then
+                            // dropped at the end of this block, cancelling the in-flight
+                            // read. `biased` polls emit first so a ready `Stop` cancels the
+                            // read at the earliest opportunity.
+                            while flow.is_none() || (matches!(flow, Some(StripeFlow::Continue)) && next.is_none()) {
+                                tokio::select! {
+                                    biased;
+                                    f = &mut emit_fut, if flow.is_none() => flow = Some(f),
+                                    n = &mut read_fut, if next.is_none() => next = Some(n),
+                                }
+                            }
+                            (flow, next)
+                        };
+
                         match flow {
-                            StripeFlow::Continue => {
+                            // `next` is guaranteed `Some` here: the loop only exits with
+                            // `Continue` once the read has also completed.
+                            Some(StripeFlow::Continue) => {
                                 reader.recycle_shards(&mut shards);
-                                current = Some(next);
-                            }
-                            StripeFlow::Stop => {
-                                // `next` is speculative; drop it without surfacing
-                                // its errors against the stripe that stopped.
-                                drop(next);
-                                break;
+                                current = next;
                             }
+                            // Emit stopped the loop. The speculative read is cancelled by
+                            // dropping `read_fut` (it goes out of scope on `break`); its
+                            // errors never surface against the stripe that stopped.
+                            _ => break,
                         }
                     } else {
                         // Final stripe: nothing left to prefetch.
@@ -1974,6 +2027,16 @@ mod tests {
             emitted: bool,
         },
         TimedOut,
+        /// Serves `cursor` (typically the first stripe's bytes) normally, then
+        /// once it is exhausted parks on a long `sleep` instead of returning EOF —
+        /// modelling a shard whose *next*-stripe read never completes (a wedged or
+        /// very slow remote shard). Used by the prefetch cancel-safety test: the
+        /// speculative next-stripe read must be cancelled, not waited out.
+        PrefixThenSlow {
+            cursor: Cursor<Vec<u8>>,
+            stall: Duration,
+            sleep: Option<Pin<Box<Sleep>>>,
+        },
     }
 
     impl crate::erasure::coding::ShardSource for TestShardReader {}
@@ -2007,6 +2070,24 @@ mod tests {
                     Poll::Ready(Ok(()))
                 }
                 TestShardReader::TimedOut => Poll::Ready(Err(io::Error::new(ErrorKind::TimedOut, "test shard read timed out"))),
+                TestShardReader::PrefixThenSlow { cursor, stall, sleep } => {
+                    let before = buf.filled().len();
+                    match Pin::new(cursor).poll_read(cx, buf) {
+                        // Cursor still has bytes for the current stripe: serve them.
+                        Poll::Ready(Ok(())) if buf.filled().len() != before => Poll::Ready(Ok(())),
+                        // Cursor exhausted (would-be EOF): stall on a long sleep rather
+                        // than signalling EOF, so the next-stripe read hangs. This parks
+                        // the task cleanly (no busy `wake_by_ref` spin), letting the
+                        // `#[tokio::test(start_paused = true)]` clock auto-advance.
+                        Poll::Ready(Ok(())) => {
+                            let stall = *stall;
+                            let sleeper = sleep.get_or_insert_with(|| Box::pin(tokio::time::sleep(stall)));
+                            let _ = sleeper.as_mut().poll(cx);
+                            Poll::Pending
+                        }
+                        other => other,
+                    }
+                }
             }
         }
     }
@@ -2788,6 +2869,77 @@ mod tests {
         });
     }
 
+    /// Cancel-safety (https://github.com/rustfs/backlog/issues/1310): when the
+    /// current stripe's emit fails (client disconnect / broken pipe), the
+    /// speculatively prefetched next-stripe read must be *cancelled*, not waited
+    /// out. Each shard serves the first stripe and then stalls for `STALL` on the
+    /// next-stripe read (a wedged remote shard); the per-shard read timeout is
+    /// pinned far above the assertion window. Under the paused clock the correct
+    /// path returns at virtual t≈0 (read future dropped); a `tokio::join!`
+    /// regression would block until the shard read timeout, so this test fails
+    /// closed if cancel-safety is reverted.
+    #[tokio::test(start_paused = true)]
+    #[serial_test::serial]
+    async fn test_legacy_prefetch_cancels_next_read_on_emit_failure() {
+        const DATA_SHARDS: usize = 4;
+        const PARITY_SHARDS: usize = 2;
+        const BLOCK_SIZE: usize = 64;
+        // Two full stripes: stripe 0 emits (and fails), stripe 1 is the prefetch
+        // that must be cancelled.
+        let total_data: Vec<u8> = (0..(2 * BLOCK_SIZE) as u32).map(|i| i as u8).collect();
+        let total_len = total_data.len();
+        let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+        let shard_size = erasure.shard_size();
+        let hash_algo = HashAlgorithm::HighwayHash256;
+        let hash_size = hash_algo.size();
+        let shard_bufs = encode_prefetch_object(&erasure, &total_data, &hash_algo).await;
+        let first_block_len = hash_size + shard_size;
+
+        // Shard read timeout and next-stripe stall both far exceed the assertion
+        // window, so a regression cannot "pass" by timing out inside the window.
+        const READ_TIMEOUT_SECS: &str = "600";
+        const STALL: Duration = Duration::from_secs(3600);
+        const ASSERT_WINDOW: Duration = Duration::from_secs(5);
+
+        // Overlap on (bitrot-decode overlap switch) so the prefetch branch runs.
+        let vars = [
+            (ENV_RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE, Some("true")),
+            (rustfs_config::ENV_OBJECT_DISK_READ_TIMEOUT, Some(READ_TIMEOUT_SECS)),
+        ];
+        temp_env::async_with_vars(vars, async {
+            let readers: Vec<Option<BitrotReader<TestShardReader>>> = shard_bufs
+                .iter()
+                .map(|buf| {
+                    // Serve only the first stripe's bytes, then stall the next read.
+                    let prefix = buf[..first_block_len.min(buf.len())].to_vec();
+                    let reader = TestShardReader::PrefixThenSlow {
+                        cursor: Cursor::new(prefix),
+                        stall: STALL,
+                        sleep: None,
+                    };
+                    Some(BitrotReader::new(reader, shard_size, hash_algo.clone(), false))
+                })
+                .collect();
+
+            let mut writer = FailingEmitWriter;
+            let start = TokioInstant::now();
+            let (written, err) = erasure.decode(&mut writer, readers, 0, total_len, total_len).await;
+            let elapsed = start.elapsed();
+
+            // Emit failed on stripe 0, so the GET fails with no bytes emitted.
+            assert!(err.is_some(), "emit failure must surface as an error");
+            assert_eq!(written, 0, "the failing writer accepts no bytes");
+            // The decisive assertion: the prefetch read was cancelled rather than
+            // awaited. Without cancel-safety this would take READ_TIMEOUT_SECS.
+            assert!(
+                elapsed < ASSERT_WINDOW,
+                "prefetch next-stripe read was not cancelled on emit failure: took {elapsed:?} \
+                 (>= {ASSERT_WINDOW:?}); a tokio::join! regression waits out the shard read timeout"
+            );
+        })
+        .await;
+    }
+
     /// Decode a healthy 2+2 multi-stripe object through the lockstep GET path
     /// and return the raw bytes pulled from each shard stream.
     async fn healthy_lockstep_shard_bytes() -> (usize, Vec<usize>) {

crates/ecstore/src/erasure/coding/encode.rs

@@ -171,19 +171,90 @@ fn quorum_dominant_error_metric_label(summary: &WriteQuorumFailureSummary) -> &'
     dominant_error_summary_label(summary)
 }
 
+/// Progress deadlines that bound how long shard writers may stall.
+///
+/// A single shard write (or writer shutdown) is a unit of forward progress: for
+/// a remote shard the underlying `HttpWriter` buffers the bytes and hands them
+/// to a background HTTP task, so a peer that accepts the connection but never
+/// drains the body eventually wedges the write once the bounded buffers fill.
+/// Without a deadline that stalled writer is awaited forever and pins an
+/// otherwise-healthy write quorum (rustfs/backlog#1319).
+///
+/// * `stall_timeout` is re-armed on every shard write, so it bounds a stall, not
+///   the total transfer time of a large object — a slow-but-honest writer that
+///   keeps completing shards is never killed.
+/// * `absolute_cap` is an optional administrator backstop: the shard writers for
+///   one object are engaged for at most this long in aggregate. It defends
+///   against a "slow-drip" peer that produces just enough progress to reset the
+///   per-shard timeout on every block while never converging. Disabled by
+///   default so a legitimate large upload over a slow link is not killed on
+///   total time alone.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct WriteProgressPolicy {
+    stall_timeout: Option<std::time::Duration>,
+    absolute_cap: Option<std::time::Duration>,
+}
+
+impl WriteProgressPolicy {
+    /// Build a policy, mapping a zero duration to "disabled" for both knobs.
+    pub(crate) fn new(stall_timeout: std::time::Duration, absolute_cap: std::time::Duration) -> Self {
+        Self {
+            stall_timeout: (!stall_timeout.is_zero()).then_some(stall_timeout),
+            absolute_cap: (!absolute_cap.is_zero()).then_some(absolute_cap),
+        }
+    }
+}
+
+impl Default for WriteProgressPolicy {
+    fn default() -> Self {
+        Self::new(
+            crate::disk::disk_store::get_object_disk_write_stall_timeout(),
+            crate::disk::disk_store::get_object_disk_write_absolute_cap(),
+        )
+    }
+}
+
 pub(crate) struct MultiWriter<'a> {
     writers: &'a mut [Option<BitrotWriterWrapper>],
     write_quorum: usize,
     errs: Vec<Option<Error>>,
+    policy: WriteProgressPolicy,
+    /// Absolute deadline for the whole object, armed lazily on the first
+    /// progress-bounded operation when `policy.absolute_cap` is set.
+    absolute_deadline: Option<tokio::time::Instant>,
 }
 
 impl<'a> MultiWriter<'a> {
     pub fn new(writers: &'a mut [Option<BitrotWriterWrapper>], write_quorum: usize) -> Self {
+        Self::with_policy(writers, write_quorum, WriteProgressPolicy::default())
+    }
+
+    pub fn with_policy(writers: &'a mut [Option<BitrotWriterWrapper>], write_quorum: usize, policy: WriteProgressPolicy) -> Self {
         let length = writers.len();
         MultiWriter {
             writers,
             write_quorum,
             errs: vec![None; length],
+            policy,
+            absolute_deadline: None,
+        }
+    }
+
+    /// Effective budget for one shard operation: the smaller of the per-shard
+    /// stall timeout and the time remaining until the object's absolute cap.
+    /// Returns `None` when neither deadline is configured (wait indefinitely).
+    fn next_progress_budget(&mut self) -> Option<std::time::Duration> {
+        if let Some(cap) = self.policy.absolute_cap {
+            let deadline = *self
+                .absolute_deadline
+                .get_or_insert_with(|| tokio::time::Instant::now() + cap);
+            let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
+            match self.policy.stall_timeout {
+                Some(stall) => Some(stall.min(remaining)),
+                None => Some(remaining),
+            }
+        } else {
+            self.policy.stall_timeout
         }
     }
 
@@ -214,13 +285,30 @@ impl<'a> MultiWriter<'a> {
     pub async fn write(&mut self, data: Vec<Bytes>) -> std::io::Result<()> {
         assert_eq!(data.len(), self.writers.len());
 
+        let budget = self.next_progress_budget();
         {
             let mut futures = FuturesUnordered::new();
             for ((writer_opt, err), shard) in self.writers.iter_mut().zip(self.errs.iter_mut()).zip(data.iter()) {
                 if err.is_some() {
                     continue; // Skip if we already have an error for this writer
                 }
-                futures.push(Self::write_shard(writer_opt, err, shard));
+                // A shard write that makes no forward progress within `budget` is
+                // failed and its disk dropped, so a stalled peer cannot pin an
+                // otherwise-healthy write quorum (rustfs/backlog#1319). `budget`
+                // is recomputed per block, so it bounds a stall — not the total
+                // transfer time — and a slow-but-honest writer is never killed.
+                futures.push(async move {
+                    match budget {
+                        Some(budget) => match tokio::time::timeout(budget, Self::write_shard(writer_opt, err, shard)).await {
+                            Ok(()) => {}
+                            Err(_elapsed) => {
+                                *err = Some(Error::Timeout);
+                                *writer_opt = None;
+                            }
+                        },
+                        None => Self::write_shard(writer_opt, err, shard).await,
+                    }
+                });
             }
             while let Some(()) = futures.next().await {}
         }
@@ -269,13 +357,30 @@ impl<'a> MultiWriter<'a> {
 
     pub async fn shutdown(&mut self) -> std::io::Result<()> {
         crate::hp_guard!("MultiWriter::shutdown");
+        let budget = self.next_progress_budget();
         {
             let mut futures = FuturesUnordered::new();
             for (writer_opt, err) in self.writers.iter_mut().zip(self.errs.iter_mut()) {
                 if err.is_some() {
                     continue;
                 }
-                futures.push(Self::shutdown_writer(writer_opt, err));
+                // Shutdown flushes the tail and waits for the remote to finish the
+                // HTTP request; a black-hole peer that never responds would hang
+                // here forever for a small object whose bytes were fully buffered
+                // (so `write` never blocked). Bound it with the same progress
+                // budget and drop the stalled writer before the quorum check.
+                futures.push(async move {
+                    match budget {
+                        Some(budget) => match tokio::time::timeout(budget, Self::shutdown_writer(writer_opt, err)).await {
+                            Ok(()) => {}
+                            Err(_elapsed) => {
+                                *err = Some(Error::Timeout);
+                                *writer_opt = None;
+                            }
+                        },
+                        None => Self::shutdown_writer(writer_opt, err).await,
+                    }
+                });
             }
             while let Some(()) = futures.next().await {}
         }
@@ -728,10 +833,12 @@ mod tests {
     use crate::erasure::coding::{BitrotWriterWrapper, CustomWriter};
     use rustfs_rio::HardLimitReader;
     use rustfs_utils::HashAlgorithm;
+    use std::future::Future;
     use std::io::Cursor;
     use std::pin::Pin;
     use std::sync::{Arc, Mutex};
     use std::task::{Context, Poll};
+    use std::time::Duration;
     use tokio::io::{AsyncWrite, AsyncWriteExt};
 
     #[derive(Clone, Default)]
@@ -821,6 +928,101 @@ mod tests {
         }
     }
 
+    /// A writer that models a "black-hole" peer: it accepts up to `stall_after`
+    /// bytes and then parks every subsequent `poll_write` forever (never
+    /// registering a waker), the way a remote `HttpWriter` wedges once its
+    /// bounded buffers fill and the peer never drains the body. With
+    /// `stall_after = 0` it stalls on the very first byte. `poll_shutdown`
+    /// completes only when `stall_shutdown` is false.
+    #[derive(Clone)]
+    struct StallingWriter {
+        accepted: Arc<std::sync::atomic::AtomicUsize>,
+        stall_after: usize,
+        stall_shutdown: bool,
+    }
+
+    impl StallingWriter {
+        fn stalls_on_write(stall_after: usize) -> Self {
+            Self {
+                accepted: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
+                stall_after,
+                stall_shutdown: false,
+            }
+        }
+
+        fn stalls_on_shutdown() -> Self {
+            Self {
+                accepted: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
+                stall_after: usize::MAX,
+                stall_shutdown: true,
+            }
+        }
+    }
+
+    impl AsyncWrite for StallingWriter {
+        fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &[u8]) -> Poll<std::io::Result<usize>> {
+            use std::sync::atomic::Ordering;
+            let already = self.accepted.load(Ordering::SeqCst);
+            if already >= self.stall_after {
+                // Black hole: no forward progress, no waker registered.
+                return Poll::Pending;
+            }
+            let take = buf.len().min(self.stall_after - already);
+            self.accepted.fetch_add(take, Ordering::SeqCst);
+            Poll::Ready(Ok(take))
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            if self.stall_shutdown {
+                Poll::Pending
+            } else {
+                Poll::Ready(Ok(()))
+            }
+        }
+    }
+
+    /// A writer that always makes forward progress but each `poll_write` first
+    /// waits `delay` (on the runtime timer). Under a paused virtual clock this
+    /// deterministically models a slow-but-honest peer: it must never be killed
+    /// by the per-shard stall deadline as long as `delay < stall_timeout`, yet an
+    /// absolute cap can still bound its aggregate engagement.
+    struct SlowWriter {
+        delay: std::time::Duration,
+        timer: Option<Pin<Box<tokio::time::Sleep>>>,
+    }
+
+    impl SlowWriter {
+        fn new(delay: std::time::Duration) -> Self {
+            Self { delay, timer: None }
+        }
+    }
+
+    impl AsyncWrite for SlowWriter {
+        fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8]) -> Poll<std::io::Result<usize>> {
+            let delay = self.delay;
+            let timer = self.timer.get_or_insert_with(|| Box::pin(tokio::time::sleep(delay)));
+            match timer.as_mut().poll(cx) {
+                Poll::Ready(()) => {
+                    self.timer = None;
+                    Poll::Ready(Ok(buf.len()))
+                }
+                Poll::Pending => Poll::Pending,
+            }
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
+
     fn bitrot_writer<W>(writer: W, shard_size: usize) -> BitrotWriterWrapper
     where
         W: AsyncWrite + Send + Sync + Unpin + 'static,
@@ -828,6 +1030,16 @@ mod tests {
         BitrotWriterWrapper::new(CustomWriter::new_tokio_writer(writer), shard_size, HashAlgorithm::HighwayHash256S)
     }
 
+    /// Like `bitrot_writer` but with no interleaved hash, so one shard write
+    /// issues a single `poll_write` to the underlying fake. Used by the paused
+    /// virtual-clock stall tests so each block maps to exactly one modeled delay.
+    fn bitrot_writer_plain<W>(writer: W, shard_size: usize) -> BitrotWriterWrapper
+    where
+        W: AsyncWrite + Send + Sync + Unpin + 'static,
+    {
+        BitrotWriterWrapper::new(CustomWriter::new_tokio_writer(writer), shard_size, HashAlgorithm::None)
+    }
+
     #[tokio::test]
     async fn helper_writers_cover_flush_and_shutdown_paths() {
         let mut failing_write = FailingWriteWriter;
@@ -922,6 +1134,194 @@ mod tests {
         assert!(shutdown_err.contains("erasure write quorum"));
     }
 
+    // The production wiring (`MultiWriter::new`) must arm a real deadline by
+    // default, and honor `0` as "disabled". This guards against the fix silently
+    // regressing to wait-forever behavior on the default PUT path.
+    #[test]
+    fn default_write_progress_policy_is_armed_and_configurable() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, || {
+            temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, || {
+                let policy = WriteProgressPolicy::default();
+                assert_eq!(
+                    policy.stall_timeout,
+                    Some(Duration::from_secs(rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT)),
+                    "the default PUT path must be protected by a stall deadline"
+                );
+                assert_eq!(policy.absolute_cap, None, "the absolute cap is disabled by default");
+            });
+        });
+
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("0"), || {
+            let policy = WriteProgressPolicy::default();
+            assert_eq!(policy.stall_timeout, None, "0 disables the stall deadline (wait indefinitely)");
+        });
+    }
+
+    fn four_shards() -> Vec<Bytes> {
+        vec![
+            Bytes::from_static(b"shard-0"),
+            Bytes::from_static(b"shard-1"),
+            Bytes::from_static(b"shard-2"),
+            Bytes::from_static(b"shard-3"),
+        ]
+    }
+
+    /// Four full-`shard_size` shards. A shard shorter than `shard_size` marks the
+    /// bitrot writer `finished`, so multi-block tests must send full shards to
+    /// keep writing across iterations.
+    fn four_full_shards(shard_size: usize) -> Vec<Bytes> {
+        (0..4).map(|i| Bytes::from(vec![0x40 + i as u8; shard_size])).collect()
+    }
+
+    // rustfs/backlog#1319: a single black-hole writer that never accepts a byte
+    // must be failed within the stall budget so the remaining 3/4 writers still
+    // meet a write quorum of 3 — the write must not wait forever.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_write_stall_fails_black_hole_but_meets_quorum() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..3).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_write(0), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[2].clone()), 64)),
+        ];
+
+        {
+            let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            mw.write(four_shards())
+                .await
+                .expect("one stalled writer must still satisfy a 3/4 write quorum without hanging");
+        }
+
+        assert!(writers[0].is_none(), "the black-hole writer must be failed and dropped before commit");
+        assert!(writers[1].is_some() && writers[2].is_some() && writers[3].is_some());
+    }
+
+    // Two black-hole writers drop the healthy count to 2/4, below the quorum of
+    // 3, so the write must fail cleanly (not hang).
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_write_stall_two_black_holes_fail_quorum() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..2).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_write(0), 64)),
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_write(0), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+        ];
+
+        let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+        let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+        let err = mw
+            .write(four_shards())
+            .await
+            .expect_err("two stalled writers must fail the write quorum instead of hanging");
+        assert!(err.to_string().contains("Failed to write data"));
+    }
+
+    // A small object whose bytes were fully buffered leaves `write` succeeding
+    // for a black-hole peer; the stall must then be caught during shutdown so it
+    // cannot pin the quorum there either.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_shutdown_stall_fails_black_hole_but_meets_quorum() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..3).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_shutdown(), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[2].clone()), 64)),
+        ];
+
+        {
+            let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            mw.write(four_shards()).await.expect("all writers accept the buffered shard");
+            mw.shutdown()
+                .await
+                .expect("a single shutdown stall must still satisfy a 3/4 quorum without hanging");
+        }
+
+        assert!(writers[0].is_none(), "the writer that stalled shutdown must be failed and dropped");
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_shutdown_stall_two_black_holes_fail_quorum() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..2).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_shutdown(), 64)),
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_shutdown(), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+        ];
+
+        let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+        let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+        mw.write(four_shards()).await.expect("all writers accept the buffered shard");
+        let err = mw
+            .shutdown()
+            .await
+            .expect_err("two shutdown stalls must fail the shutdown quorum instead of hanging");
+        assert!(err.to_string().contains("Failed to shutdown writers"));
+    }
+
+    // A slow-but-honest writer that keeps completing shards (delay < stall
+    // budget) must never be killed, even across many blocks, when no absolute cap
+    // is configured.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_slow_but_progressing_writer_is_not_killed() {
+        let mut writers = vec![
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(1)), 64)),
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(1)), 64)),
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(1)), 64)),
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(1)), 64)),
+        ];
+
+        {
+            let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            for _ in 0..8 {
+                mw.write(four_full_shards(64))
+                    .await
+                    .expect("a writer that keeps making progress within the stall budget must not be failed");
+            }
+        }
+
+        assert!(writers.iter().all(Option::is_some), "no slow-but-honest writer should be dropped");
+    }
+
+    // Slow-drip defense: the per-shard stall timer resets on every block, so a
+    // peer that dribbles just enough progress each block is not caught by the
+    // stall timeout alone. The optional absolute cap bounds its aggregate
+    // engagement and fails it within a finite budget, while the healthy writers
+    // still meet quorum.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_absolute_cap_bounds_slow_drip_writer() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..3).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(8)), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[2].clone()), 64)),
+        ];
+
+        {
+            // stall (10s) never fires for an 8s-per-block writer, but the 15s
+            // absolute cap does once its aggregate engagement crosses the cap.
+            let policy = WriteProgressPolicy::new(Duration::from_secs(10), Duration::from_secs(15));
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            for _ in 0..4 {
+                mw.write(four_full_shards(64))
+                    .await
+                    .expect("healthy writers keep the 3/4 quorum while the drip writer is bounded");
+            }
+        }
+
+        assert!(
+            writers[0].is_none(),
+            "the slow-drip writer must be failed within the absolute cap, not held forever"
+        );
+    }
+
     #[tokio::test]
     async fn drain_queued_inflight_bytes_consumes_pending_blocks() {
         let (tx, mut rx) = mpsc::channel(2);

crates/ecstore/src/set_disk/core/io_primitives.rs

@@ -2050,7 +2050,7 @@ impl SetDisks {
         let bucket = Arc::new(bucket.to_string());
         let object = Arc::new(object.to_string());
         let version_id = Arc::new(version_id.to_string());
-        let futures = disks.iter().map(|disk| {
+        let futures = disks.iter().enumerate().map(|(disk_index, disk)| {
             let disk = disk.clone();
             let opts = opts.clone();
             let org_bucket = org_bucket.clone();
@@ -2060,6 +2060,7 @@ impl SetDisks {
             tokio::spawn(async move {
                 let response_start = observe.then(Instant::now);
                 let result = if let Some(disk) = disk {
+                    Self::record_read_version_call(&object, disk_index);
                     disk.read_version(&org_bucket, &bucket, &object, &version_id, &opts).await
                 } else {
                     Err(DiskError::DiskNotFound)
@@ -2145,6 +2146,7 @@ impl SetDisks {
             join_set.spawn(async move {
                 let response_start = Instant::now();
                 let result = if let Some(disk) = disk {
+                    Self::record_read_version_call(&object, index);
                     disk.read_version(&org_bucket, &bucket, &object, &version_id, &opts).await
                 } else {
                     Err(DiskError::DiskNotFound)
@@ -2522,11 +2524,57 @@ enum OrphanDirScan {
     Missing,
 }
 
-fn rename_data_versions_key(versions: &[u8]) -> Option<[u8; 8]> {
-    let prefix = versions.get(..8)?;
-    let mut key = [0; 8];
-    key.copy_from_slice(prefix);
-    Some(key)
+/// Outcome of a *post-quorum* `rename_data` commit, classifying whether the
+/// committed replicas converged so the caller can decide heal admission
+/// WITHOUT conflating "a version signature exists" with "this write needs
+/// heal" (rustfs/backlog#1321).
+///
+/// `rename_data` reaches this classification only once write quorum is met — a
+/// sub-quorum commit returns `Err` and never produces a convergence — so every
+/// variant describes an already-durable, already-ACKable write.
+///
+/// # Extension point for #1312 (commit fencing)
+///
+/// #1312 layers a per-object fencing epoch onto the SAME `rename_data` path.
+/// An epoch rejection is a *commit* failure surfaced through the existing
+/// `Result::Err` channel (the write never lands), which is orthogonal to this
+/// post-commit convergence signal (the write landed; do the replicas need
+/// heal). The two therefore compose: fencing gates whether we reach a
+/// convergence at all, and this enum classifies the convergence once reached.
+/// A future fence-aware variant, if ever needed, is an additive change here.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(in crate::set_disk) enum RenameConvergence {
+    /// Every attempted disk committed and reported an identical, known version
+    /// signature. The committed replicas are already converged; no heal needed.
+    AllSuccessIdentical,
+    /// Write quorum was met but at least one attempted disk failed to commit
+    /// (error or offline). The committed set may be short a replica; the
+    /// laggard must be converged by heal.
+    PartialCommit,
+    /// Every attempted disk committed but their version signatures diverge (or
+    /// mix signed <=10-version disks with unsigned >10-version disks, itself a
+    /// version-count divergence). Heal must reconcile the committed replicas.
+    SignatureDivergent,
+    /// Every attempted disk committed but none produced a version signature
+    /// (all observed >10 versions, where the signature is deliberately
+    /// omitted). Divergence can neither be proven nor ruled out here, so any
+    /// latent divergence is left to the scanner backstop rather than enqueued
+    /// for heal.
+    Unknown,
+}
+
+impl RenameConvergence {
+    /// Whether this commit outcome must enqueue an object heal to converge the
+    /// committed replicas.
+    ///
+    /// `Unknown` and `AllSuccessIdentical` return `false`: the former is
+    /// scanner-backstopped (see the variant docs), the latter is already
+    /// converged. This is the single decision point that replaced the old
+    /// `Option<Vec<u8>>::is_some()` heuristic, under which a healthy MPU
+    /// (identical signatures on every disk) always looked like it needed heal.
+    pub(in crate::set_disk) fn needs_heal(&self) -> bool {
+        matches!(self, Self::PartialCommit | Self::SignatureDivergent)
+    }
 }
 
 impl SetDisks {
@@ -2555,7 +2603,7 @@ impl SetDisks {
         write_quorum: usize,
     ) -> disk::error::Result<(
         Vec<Option<DiskStore>>,
-        Option<Vec<u8>>,
+        RenameConvergence,
         Option<Uuid>,
         Vec<Option<DiskStore>>,
         Option<OldCurrentSize>,
@@ -2578,6 +2626,11 @@ impl SetDisks {
             let dst_bucket = dst_bucket.clone();
 
             futures.push(tokio::spawn(async move {
+                // Test-only introspection guard: counts this task as in-flight for
+                // the whole body. Compiles to `()` in production (no behavior).
+                #[allow(clippy::let_unit_value)]
+                let _fanout_task_guard = Self::rename_fanout_task_guard(&dst_object);
+
                 if file_info.erasure.index == 0 {
                     file_info.erasure.index = i + 1;
                 }
@@ -2586,6 +2639,10 @@ impl SetDisks {
                     return Err(DiskError::FileCorrupt);
                 }
 
+                // Test-only awaitable pause point right before the disk rename.
+                // A no-op immediately-ready future in production.
+                Self::rename_fanout_barrier(&dst_object, i, rename_fanout_barrier_phase::RENAME).await;
+
                 if let Some(disk) = disk {
                     disk.rename_data(&src_bucket, &src_object, file_info, &dst_bucket, &dst_object)
                         .await
@@ -2709,7 +2766,7 @@ impl SetDisks {
         }
 
         let data_dir = Self::reduce_common_data_dir(&data_dirs, write_quorum);
-        let versions = Self::select_rename_data_versions(&disk_versions, &errs, write_quorum);
+        let convergence = Self::classify_rename_convergence(&disk_versions, &errs);
         let old_current_size = Self::reduce_common_old_current_size(&old_current_sizes, write_quorum);
         let online_disks = Self::eval_disks(disks, &errs);
         let cleanup_disks = if let Some(data_dir) = data_dir {
@@ -2729,7 +2786,7 @@ impl SetDisks {
             vec![None; disks.len()]
         };
 
-        Ok((online_disks, versions, data_dir, cleanup_disks, old_current_size))
+        Ok((online_disks, convergence, data_dir, cleanup_disks, old_current_size))
     }
 
     /// rustfs/backlog#1009: reduce the per-disk observations of the
@@ -2760,61 +2817,62 @@ impl SetDisks {
         if max >= write_quorum { old_current_size } else { None }
     }
 
-    pub(in crate::set_disk) fn reduce_common_versions(disk_versions: &[Option<Vec<u8>>], write_quorum: usize) -> Option<Vec<u8>> {
-        let mut versions_count = HashMap::new();
-
-        for versions in disk_versions.iter().flatten() {
-            if let Some(key) = rename_data_versions_key(versions) {
-                *versions_count.entry(key).or_insert(0usize) += 1;
-            }
-        }
-
-        let (common_versions, max_count) = versions_count
-            .into_iter()
-            .max_by_key(|(_, count)| *count)
-            .unwrap_or(([0; 8], 0));
-
-        if max_count < write_quorum {
-            return None;
-        }
-
-        disk_versions
-            .iter()
-            .flatten()
-            .find(|versions| rename_data_versions_key(versions).is_some_and(|key| key == common_versions))
-            .cloned()
-    }
-
-    pub(in crate::set_disk) fn select_rename_data_versions(
+    /// Classify a *post-quorum* `rename_data` commit into an explicit
+    /// convergence outcome (rustfs/backlog#1321). This runs only after the
+    /// write-quorum gate has already passed, so every returned variant
+    /// describes an already-durable, already-ACKable write; the decision here
+    /// is purely "do the committed replicas need heal to converge".
+    ///
+    /// The per-disk version signature (`disk_versions[i]`) is used only as
+    /// comparison material — it is deliberately NOT overloaded to also mean
+    /// "needs heal". A `Some(bytes)` signature carries the concatenated
+    /// version-id bytes of a disk that observed <=10 versions; a `None`
+    /// signature is a disk that committed but deliberately produced no
+    /// signature (>10 versions). A `None` entry for a *failed* disk never
+    /// reaches this point as a signature because failures are handled first.
+    pub(in crate::set_disk) fn classify_rename_convergence(
         disk_versions: &[Option<Vec<u8>>],
         errs: &[Option<DiskError>],
-        write_quorum: usize,
-    ) -> Option<Vec<u8>> {
-        let mut versions = Self::reduce_common_versions(disk_versions, write_quorum);
-        for (dversions, err) in disk_versions.iter().zip(errs.iter()) {
-            if err.is_some() {
-                continue;
-            }
-            let Some(dversions) = dversions.as_ref().filter(|versions| !versions.is_empty()) else {
+    ) -> RenameConvergence {
+        // Any failed / offline disk that got past the write-quorum gate means a
+        // committed replica is missing or stale: converge it via heal,
+        // regardless of what the surviving disks' signatures say.
+        if errs.iter().any(|err| err.is_some()) {
+            return RenameConvergence::PartialCommit;
+        }
+
+        // Every disk committed. Compare their reported version signatures.
+        let mut seen: Option<&Vec<u8>> = None;
+        let mut signed_count = 0usize;
+        let mut divergent = false;
+        for signature in disk_versions.iter() {
+            let Some(sig) = signature.as_ref() else {
                 continue;
             };
-
-            match versions.as_ref() {
-                Some(current_versions) if dversions != current_versions => {
-                    if dversions.len() > current_versions.len() {
-                        versions = Some(dversions.clone());
-                    }
-                    break;
-                }
+            signed_count += 1;
+            match seen {
+                None => seen = Some(sig),
+                Some(prev) if prev != sig => divergent = true,
                 Some(_) => {}
-                None => {
-                    versions = Some(dversions.clone());
-                    break;
-                }
             }
         }
 
-        versions
+        if divergent {
+            return RenameConvergence::SignatureDivergent;
+        }
+        if signed_count == 0 {
+            // No disk produced a signature (all observed >10 versions): a
+            // latent divergence cannot be proven or ruled out here, so it is
+            // left to the scanner backstop rather than enqueued for heal.
+            return RenameConvergence::Unknown;
+        }
+        if signed_count < disk_versions.len() {
+            // A mix of signed (<=10 versions) and unsigned (>10 versions) disks
+            // is itself a version-count divergence between committed replicas:
+            // reconcile it via heal.
+            return RenameConvergence::SignatureDivergent;
+        }
+        RenameConvergence::AllSuccessIdentical
     }
 
     /// Reclaim the old (now dereferenced) `object/<old_data_dir>` on the disks
@@ -2874,6 +2932,12 @@ impl SetDisks {
             let disk = disk.clone();
             let object_for_fault = object_for_fault.clone();
             tokio::spawn(async move {
+                // Test-only introspection guard + awaitable pause point for the
+                // old-data-dir cleanup fan-out. Both compile away in production.
+                #[allow(clippy::let_unit_value)]
+                let _fanout_task_guard = Self::rename_fanout_task_guard(&object_for_fault);
+                Self::rename_fanout_barrier(&object_for_fault, idx, rename_fanout_barrier_phase::CLEANUP).await;
+
                 if let Some(err) = Self::cleanup_injected_error(&object_for_fault, idx) {
                     return Some(err);
                 }
@@ -2914,6 +2978,56 @@ impl SetDisks {
         None
     }
 
+    /// Test-only seam that records one per-disk `read_version` metadata RPC for
+    /// the call-counter registry (backlog#1325). In production this is inlined to
+    /// nothing and adds no behavior; only the `#[cfg(test)]` variant touches the
+    /// registry. Placed inside the fan-out spawn tasks so counts are observed
+    /// even though the increments run on arbitrary runtime worker threads.
+    #[cfg(test)]
+    #[inline]
+    fn record_read_version_call(object: &str, disk_index: usize) {
+        disk_call_counters::record(object, disk_call_counters::KIND_READ_VERSION, disk_index);
+    }
+
+    #[cfg(not(test))]
+    #[inline(always)]
+    fn record_read_version_call(_object: &str, _disk_index: usize) {}
+
+    /// Test-only awaitable pause point for the rename/commit fan-out (backlog#1325,
+    /// serving the barrier-style acceptances of #1312 / #1319 / #1313). `phase` is
+    /// [`rename_fanout_barrier::PHASE_RENAME`] or `PHASE_CLEANUP`. When a test has
+    /// armed a barrier for `object` at this `(disk_index, phase)`, the spawned
+    /// fan-out task blocks here until the test releases it, so the test can await
+    /// the pause point and then introspect in-flight background disk work. In
+    /// production this awaits an immediately-ready no-op future — no yield, no
+    /// registry access, no behavior change.
+    #[cfg(test)]
+    #[inline]
+    async fn rename_fanout_barrier(object: &str, disk_index: usize, phase: &'static str) {
+        rename_fanout_barrier::checkpoint(object, disk_index, phase).await;
+    }
+
+    #[cfg(not(test))]
+    #[inline(always)]
+    #[allow(clippy::unused_async)]
+    async fn rename_fanout_barrier(_object: &str, _disk_index: usize, _phase: &'static str) {}
+
+    /// Test-only RAII counter for one in-flight rename/commit fan-out task
+    /// (backlog#1325). Held for the whole spawned task body so a test observing
+    /// `object` can assert "background disk writes are still running" while the
+    /// fan-out is paused and "no background disk writes remain" once it drains —
+    /// the exact signal #1312 needs after a lock release. In production this
+    /// returns `()` and touches no registry.
+    #[cfg(test)]
+    #[inline]
+    fn rename_fanout_task_guard(object: &str) -> rename_fanout_barrier::TaskGuard {
+        rename_fanout_barrier::task_guard(object)
+    }
+
+    #[cfg(not(test))]
+    #[inline(always)]
+    fn rename_fanout_task_guard(_object: &str) {}
+
     /// Report a post-commit old-data-dir cleanup receipt: emit metrics, warn on
     /// residue/below-quorum, and — on residue — enqueue an object heal.
     ///
@@ -3909,6 +4023,340 @@ pub(in crate::set_disk) mod cleanup_fault_injection {
     }
 }
 
+/// Test-only per-disk call counters for the metadata fan-out (backlog#1325,
+/// serving the RPC-count assertions of #1309 / #1314 / #1315).
+///
+/// The metadata fan-out issues each per-disk `read_version` inside a
+/// `tokio::spawn` task, so the increments happen on arbitrary runtime worker
+/// threads. A thread-local `metrics::Recorder` (see `test_metrics.rs`) cannot
+/// observe those spawned increments; this registry is a process-global keyed by
+/// object name, so counts recorded inside spawned tasks are visible from the
+/// test thread that installed the observing scope.
+///
+/// The whole module is `#[cfg(test)]`, so it never compiles into production.
+/// Parallel tests stay isolated by observing distinct object names — an
+/// unobserved object records nothing, keeping the registry bounded, and each
+/// [`CallCounterScope`] clears only its own object's counts on drop.
+#[cfg(test)]
+pub(in crate::set_disk) mod disk_call_counters {
+    use std::collections::{HashMap, HashSet};
+    use std::sync::{Mutex, OnceLock};
+
+    /// Kind label for the per-disk `read_version` metadata RPC.
+    pub const KIND_READ_VERSION: &str = "read_version";
+
+    /// Registry key: (object, kind, disk_index).
+    type CountKey = (String, String, usize);
+
+    #[derive(Default)]
+    struct Registry {
+        /// Object names with an active observing scope. Only these accumulate,
+        /// so unrelated concurrent tests never inflate one another's counts.
+        observed: HashSet<String>,
+        /// (object, kind, disk_index) -> call count.
+        counts: HashMap<CountKey, u64>,
+    }
+
+    fn registry() -> &'static Mutex<Registry> {
+        static REG: OnceLock<Mutex<Registry>> = OnceLock::new();
+        REG.get_or_init(|| Mutex::new(Registry::default()))
+    }
+
+    /// Record one call of `kind` against `object`'s `disk_index`. A no-op unless
+    /// an observing scope for `object` is currently installed.
+    pub(super) fn record(object: &str, kind: &str, disk_index: usize) {
+        let mut reg = registry().lock().expect("disk call-counter registry poisoned");
+        if !reg.observed.contains(object) {
+            return;
+        }
+        *reg.counts
+            .entry((object.to_string(), kind.to_string(), disk_index))
+            .or_insert(0) += 1;
+    }
+
+    /// RAII scope that observes call counts for a single `object`. Counts recorded
+    /// while the scope is alive — including those recorded inside spawned tasks —
+    /// are queryable; the scope clears its own counts on drop.
+    #[must_use]
+    pub struct CallCounterScope {
+        object: String,
+    }
+
+    /// Begin observing per-disk call counts for `object`.
+    pub fn observe(object: &str) -> CallCounterScope {
+        let mut reg = registry().lock().expect("disk call-counter registry poisoned");
+        reg.observed.insert(object.to_string());
+        CallCounterScope {
+            object: object.to_string(),
+        }
+    }
+
+    impl CallCounterScope {
+        /// Total number of `kind` calls across all disks for the observed object.
+        pub fn total(&self, kind: &str) -> u64 {
+            let reg = registry().lock().expect("disk call-counter registry poisoned");
+            reg.counts
+                .iter()
+                .filter(|((obj, k, _), _)| obj == &self.object && k == kind)
+                .map(|(_, count)| *count)
+                .sum()
+        }
+
+        /// Number of `kind` calls recorded against a specific `disk_index`.
+        pub fn for_disk(&self, kind: &str, disk_index: usize) -> u64 {
+            let reg = registry().lock().expect("disk call-counter registry poisoned");
+            reg.counts
+                .get(&(self.object.clone(), kind.to_string(), disk_index))
+                .copied()
+                .unwrap_or(0)
+        }
+    }
+
+    impl Drop for CallCounterScope {
+        fn drop(&mut self) {
+            let mut reg = registry().lock().expect("disk call-counter registry poisoned");
+            reg.observed.remove(&self.object);
+            reg.counts.retain(|(obj, _, _), _| obj != &self.object);
+        }
+    }
+}
+
+/// Fan-out phase labels for the rename/commit barrier seam (backlog#1325).
+///
+/// These are referenced from the production fan-out call sites (as inert string
+/// literals passed to a no-op seam), so — unlike the `#[cfg(test)]` barrier
+/// registry itself — they are defined unconditionally. In production builds the
+/// seam ignores them entirely.
+pub(in crate::set_disk) mod rename_fanout_barrier_phase {
+    /// The per-disk `rename_data` phase of the write-commit fan-out.
+    pub const RENAME: &str = "rename";
+    /// The per-disk old-data-dir cleanup phase of the commit fan-out.
+    pub const CLEANUP: &str = "cleanup";
+}
+
+/// Test-only awaitable pause barrier + background-task introspection for the
+/// rename/commit fan-out (backlog#1325, the second facility block after the
+/// per-disk call counters). Serves the barrier-style white-box acceptances of
+/// #1312 (commit fencing: "abort at the first-disk rename barrier, assert the
+/// coordinator still holds the lock, assert no background disk write remains
+/// after release"), #1319, and #1313.
+///
+/// Two independent, object-keyed mechanisms share one process-global registry:
+///
+/// 1. **Awaitable pause barrier.** A test [`arm`]s a barrier for `(object,
+///    disk_index, phase)`. The matching spawned fan-out task blocks at its
+///    [`checkpoint`] until the test releases it. The test awaits the pause via
+///    [`BarrierHandle::wait_until_paused`] (a deterministic `Notify` handshake —
+///    no sleeps) and resumes it via [`BarrierHandle::release`]. At most one
+///    barrier is armed per object at a time, matching the single-scope style of
+///    `disk_call_counters`.
+///
+/// 2. **Background-task introspection.** A test [`observe_tasks`] for `object`;
+///    each instrumented fan-out task then holds a [`TaskGuard`] for its whole
+///    body, so [`TaskTrackerScope::running`] reports how many rename/cleanup
+///    background disk tasks are still in flight. This is the concrete "is there
+///    still a background disk write?" signal #1312 asserts after a lock release.
+///
+/// Both are keyed by object and only accumulate for armed/observed objects, so
+/// concurrent tests using distinct object names stay fully isolated. The whole
+/// module is `#[cfg(test)]` and never compiles into production; the fan-out call
+/// sites reach it only through the `#[cfg(not(test))]` no-op seams on `SetDisks`.
+///
+/// Scope of this block (white-box, in-process only): coordinator lock-holding is
+/// asserted by the test at the store/coordinator layer via the guard it already
+/// holds — `io_primitives` has no handle to that namespace lock, and fabricating
+/// a lock-state registry here would be a look-alike rather than the real lock.
+/// Cross-process/black-box fault injection (toxiproxy, blackhole peers, 2-pool)
+/// is a later cluster-harness block, not this one.
+#[cfg(test)]
+pub(in crate::set_disk) mod rename_fanout_barrier {
+    use std::collections::HashMap;
+    use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+    use std::sync::{Arc, Mutex, OnceLock};
+    use tokio::sync::Notify;
+
+    pub use super::rename_fanout_barrier_phase::{CLEANUP as PHASE_CLEANUP, RENAME as PHASE_RENAME};
+
+    /// One armed barrier: the fan-out task matching `(disk_index, phase)` pauses.
+    struct Armed {
+        disk_index: usize,
+        phase: &'static str,
+        /// Signalled (task -> test) when the target task reaches the checkpoint.
+        arrived: Arc<Notify>,
+        /// Signalled (test -> task) to release the paused task.
+        release: Arc<Notify>,
+        /// Set once the target task is parked at the checkpoint.
+        paused: Arc<AtomicBool>,
+    }
+
+    #[derive(Default)]
+    struct Registry {
+        /// object -> armed barrier (at most one per object).
+        armed: HashMap<String, Armed>,
+        /// object -> live in-flight fan-out task count, only for observed objects.
+        observed: HashMap<String, Arc<AtomicUsize>>,
+    }
+
+    fn registry() -> &'static Mutex<Registry> {
+        static REG: OnceLock<Mutex<Registry>> = OnceLock::new();
+        REG.get_or_init(|| Mutex::new(Registry::default()))
+    }
+
+    fn lock() -> std::sync::MutexGuard<'static, Registry> {
+        registry().lock().expect("rename fan-out barrier registry poisoned")
+    }
+
+    /// RAII handle for one armed barrier. Dropping it disarms the barrier and
+    /// releases any still-parked task, so a panicking or forgetful test can never
+    /// leave a spawned fan-out task wedged.
+    #[must_use]
+    pub struct BarrierHandle {
+        object: String,
+        arrived: Arc<Notify>,
+        release: Arc<Notify>,
+        paused: Arc<AtomicBool>,
+    }
+
+    /// Arm a barrier: the fan-out task for `object` at `(disk_index, phase)` will
+    /// pause at its checkpoint until the returned handle is released or dropped.
+    pub fn arm(object: &str, disk_index: usize, phase: &'static str) -> BarrierHandle {
+        let arrived = Arc::new(Notify::new());
+        let release = Arc::new(Notify::new());
+        let paused = Arc::new(AtomicBool::new(false));
+        lock().armed.insert(
+            object.to_string(),
+            Armed {
+                disk_index,
+                phase,
+                arrived: arrived.clone(),
+                release: release.clone(),
+                paused: paused.clone(),
+            },
+        );
+        BarrierHandle {
+            object: object.to_string(),
+            arrived,
+            release,
+            paused,
+        }
+    }
+
+    impl BarrierHandle {
+        /// Await until the armed fan-out task has parked at the checkpoint. Uses a
+        /// stored-permit `Notify`, so this is race-free regardless of whether the
+        /// task reaches the checkpoint before or after this call — no sleeps.
+        pub async fn wait_until_paused(&self) {
+            if self.paused.load(Ordering::SeqCst) {
+                return;
+            }
+            self.arrived.notified().await;
+        }
+
+        /// Whether the target task is currently parked at the checkpoint.
+        pub fn is_paused(&self) -> bool {
+            self.paused.load(Ordering::SeqCst)
+        }
+
+        /// Release the parked task so the fan-out can proceed.
+        pub fn release(&self) {
+            self.release.notify_one();
+        }
+    }
+
+    impl Drop for BarrierHandle {
+        fn drop(&mut self) {
+            // Unblock any task still parked at the checkpoint before disarming, so
+            // a dropped handle can never wedge a spawned fan-out task.
+            self.release.notify_one();
+            lock().armed.remove(&self.object);
+        }
+    }
+
+    /// Seam entry point invoked from inside each spawned fan-out task. A no-op
+    /// unless a barrier is armed for exactly this `(object, disk_index, phase)`.
+    /// The registry mutex is released before awaiting, so it is never held across
+    /// the pause.
+    pub(in crate::set_disk) async fn checkpoint(object: &str, disk_index: usize, phase: &'static str) {
+        let hooks = {
+            let reg = lock();
+            match reg.armed.get(object) {
+                Some(a) if a.disk_index == disk_index && a.phase == phase => {
+                    Some((a.arrived.clone(), a.release.clone(), a.paused.clone()))
+                }
+                _ => None,
+            }
+        };
+        if let Some((arrived, release, paused)) = hooks {
+            paused.store(true, Ordering::SeqCst);
+            arrived.notify_one();
+            release.notified().await;
+        }
+    }
+
+    /// RAII scope that observes in-flight fan-out task counts for a single
+    /// `object`. Counts accrue only while the scope is alive; it clears its own
+    /// entry on drop.
+    #[must_use]
+    pub struct TaskTrackerScope {
+        object: String,
+    }
+
+    /// Begin observing in-flight rename/cleanup fan-out task counts for `object`.
+    pub fn observe_tasks(object: &str) -> TaskTrackerScope {
+        lock()
+            .observed
+            .entry(object.to_string())
+            .or_insert_with(|| Arc::new(AtomicUsize::new(0)));
+        TaskTrackerScope {
+            object: object.to_string(),
+        }
+    }
+
+    impl TaskTrackerScope {
+        /// Number of instrumented fan-out tasks for the observed object that are
+        /// currently in flight (task body entered, guard not yet dropped).
+        pub fn running(&self) -> usize {
+            lock()
+                .observed
+                .get(&self.object)
+                .map(|c| c.load(Ordering::SeqCst))
+                .unwrap_or(0)
+        }
+    }
+
+    impl Drop for TaskTrackerScope {
+        fn drop(&mut self) {
+            lock().observed.remove(&self.object);
+        }
+    }
+
+    /// RAII guard held for the lifetime of one spawned fan-out task. Increments
+    /// the observed counter on creation (if the object is observed) and decrements
+    /// it on drop. Holding the `Arc` keeps the decrement sound even if the scope
+    /// is dropped while a task is still draining.
+    #[must_use]
+    pub struct TaskGuard {
+        counter: Option<Arc<AtomicUsize>>,
+    }
+
+    /// Create a task guard for `object`. A no-op guard unless `object` is observed.
+    pub(in crate::set_disk) fn task_guard(object: &str) -> TaskGuard {
+        let counter = lock().observed.get(object).cloned();
+        if let Some(c) = &counter {
+            c.fetch_add(1, Ordering::SeqCst);
+        }
+        TaskGuard { counter }
+    }
+
+    impl Drop for TaskGuard {
+        fn drop(&mut self) {
+            if let Some(c) = &self.counter {
+                c.fetch_sub(1, Ordering::SeqCst);
+            }
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -4043,6 +4491,226 @@ mod tests {
         });
     }
 
+    /// Builds `count` empty local disks (each in its own tempdir) for the given
+    /// bucket. Returns the tempdir guards (which must outlive the disks) and the
+    /// disk slot vector expected by the metadata fan-out.
+    async fn call_counter_local_disks(bucket: &str, count: usize) -> (Vec<TempDir>, Vec<Option<DiskStore>>) {
+        let mut dirs = Vec::with_capacity(count);
+        let mut disks = Vec::with_capacity(count);
+        for _ in 0..count {
+            let (dir, disk) = read_multiple_test_disk(bucket, &[]).await;
+            dirs.push(dir);
+            disks.push(Some(disk));
+        }
+        (dirs, disks)
+    }
+
+    /// Demo / regression guard for the backlog#1325 per-disk call counters.
+    ///
+    /// The metadata fan-out issues each `read_version` inside its own
+    /// `tokio::spawn` task; on a multi-thread runtime those tasks land on
+    /// arbitrary worker threads. This asserts the process-global registry still
+    /// observes every per-disk increment — the exact gap that makes the
+    /// thread-local `CapturingRecorder` unusable for #1309 / #1314 counting.
+    ///
+    /// Reverting `record_read_version_call` to a no-op drops both the total and
+    /// the per-disk counts to zero and fails this test.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn read_version_call_counter_observes_spawned_fanout() {
+        const DISKS: usize = 4;
+        let bucket = "counter-bucket";
+        let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+
+        let object = "counter-object";
+        let scope = disk_call_counters::observe(object);
+
+        // read_data=false, observe=false -> deterministic full-wait fan-out.
+        let _ = SetDisks::read_all_fileinfo(&disks, bucket, bucket, object, "", false, false, false).await;
+
+        assert_eq!(
+            scope.total(disk_call_counters::KIND_READ_VERSION),
+            DISKS as u64,
+            "every online disk must record exactly one read_version, even from a spawned task"
+        );
+        for idx in 0..DISKS {
+            assert_eq!(
+                scope.for_disk(disk_call_counters::KIND_READ_VERSION, idx),
+                1,
+                "disk {idx} should record exactly one read_version"
+            );
+        }
+
+        // A second observed fan-out accumulates onto the same scope.
+        let _ = SetDisks::read_all_fileinfo(&disks, bucket, bucket, object, "", false, false, false).await;
+        assert_eq!(scope.total(disk_call_counters::KIND_READ_VERSION), (DISKS * 2) as u64);
+
+        drop(dirs);
+    }
+
+    /// Isolation guard: unobserved objects record nothing (so parallel tests do
+    /// not inflate one another), and a scope clears its own counts on drop.
+    #[tokio::test]
+    async fn call_counter_isolates_unobserved_objects_and_clears_on_drop() {
+        let bucket = "counter-bucket-iso";
+        let (dirs, disks) = call_counter_local_disks(bucket, 1).await;
+        let object = "iso-object";
+
+        // No active scope: the fan-out records nothing.
+        let _ = SetDisks::read_all_fileinfo(&disks, bucket, bucket, object, "", false, false, false).await;
+
+        {
+            let scope = disk_call_counters::observe(object);
+            assert_eq!(
+                scope.total(disk_call_counters::KIND_READ_VERSION),
+                0,
+                "reads before the scope existed must not be counted"
+            );
+            let _ = SetDisks::read_all_fileinfo(&disks, bucket, bucket, object, "", false, false, false).await;
+            assert_eq!(scope.total(disk_call_counters::KIND_READ_VERSION), 1);
+        }
+
+        // Previous scope dropped -> counts cleared; a fresh scope starts at zero.
+        let scope = disk_call_counters::observe(object);
+        assert_eq!(
+            scope.total(disk_call_counters::KIND_READ_VERSION),
+            0,
+            "dropping a scope must clear its counts"
+        );
+
+        drop(dirs);
+    }
+
+    /// Bound for the pause handshake. This is a hang-guard, not a timing
+    /// dependency: under a working barrier `wait_until_paused` returns via the
+    /// `Notify` handshake far below this bound regardless of IO pressure, so the
+    /// assertions never depend on the value. It only turns a neutralized-barrier
+    /// hang into a deterministic failure instead of an infinite wait.
+    const BARRIER_PAUSE_GUARD: std::time::Duration = std::time::Duration::from_secs(10);
+
+    fn rename_barrier_fileinfos(object: &str, count: usize) -> Vec<FileInfo> {
+        (0..count).map(|_| metadata_test_fileinfo(object)).collect()
+    }
+
+    /// Demo / regression guard for the backlog#1325 rename fan-out pause barrier
+    /// and background-task introspection. Serves the barrier-style acceptance of
+    /// #1312 ("assert no background disk write remains after release").
+    ///
+    /// It drives the real `SetDisks::rename_data` fan-out: a barrier is armed at
+    /// the first disk's `rename` phase, and the test awaits that pause point,
+    /// asserts a background disk task is still in flight, releases it, and then
+    /// asserts the in-flight count drains to zero once the fan-out completes.
+    ///
+    /// Neutralizing the barrier seam (`rename_fanout_barrier` -> immediate no-op)
+    /// makes `wait_until_paused` never wake, so the guarded await elapses and the
+    /// test fails. Neutralizing the task guard (`rename_fanout_task_guard` ->
+    /// `()`) pins `running()` at zero, so the "still in flight" assertion fails.
+    #[tokio::test]
+    async fn rename_fanout_barrier_pauses_and_reports_running_background_tasks() {
+        const DISKS: usize = 4;
+        let bucket = "rename-barrier-bucket";
+        let object = "rename-barrier-object";
+        let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+        let file_infos = rename_barrier_fileinfos(object, DISKS);
+
+        let tracker = rename_fanout_barrier::observe_tasks(object);
+        assert_eq!(tracker.running(), 0, "no fan-out tasks before rename starts");
+
+        let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_RENAME);
+
+        // Run the real rename fan-out concurrently with the control flow so we can
+        // introspect while it is parked at the first disk's rename checkpoint.
+        let rename_fut = SetDisks::rename_data(&disks, bucket, object, &file_infos, bucket, object, DISKS - 1);
+        let control_fut = async {
+            tokio::time::timeout(BARRIER_PAUSE_GUARD, barrier.wait_until_paused())
+                .await
+                .expect("fan-out must reach the armed rename barrier");
+            assert!(barrier.is_paused(), "target task must be parked at the checkpoint");
+            assert!(tracker.running() >= 1, "a background rename task must still be in flight while paused");
+            barrier.release();
+        };
+        let (_rename_res, ()) = tokio::join!(rename_fut, control_fut);
+
+        // The fan-out has fully joined -> every task guard has dropped.
+        assert_eq!(tracker.running(), 0, "no background rename task may remain once the fan-out has drained");
+
+        drop(dirs);
+    }
+
+    /// Demo / regression guard for the barrier on the commit (old-data-dir)
+    /// cleanup fan-out. Serves the same #1312/#1319 "no background disk write
+    /// after release" shape, on the reclamation path that runs *after* a write is
+    /// ACKed — the classic detached background delete #1312 fences against.
+    ///
+    /// It stages a real `object/<old_data_dir>` on two disks and drives the real
+    /// `commit_rename_data_dir` fan-out, pausing the first disk's cleanup delete.
+    #[tokio::test]
+    async fn commit_cleanup_fanout_barrier_pauses_background_deletes() {
+        let bucket = "cleanup-barrier-bucket";
+        let object = "cleanup-barrier-object";
+        let old_data_dir = "11111111-1111-1111-1111-111111111111";
+        let committed_data_dir = "22222222-2222-2222-2222-222222222222";
+        let path = format!("{object}/{old_data_dir}/part.1");
+        let (_dir1, disk1) = read_multiple_test_disk(bucket, &[(&path, b"one".as_slice())]).await;
+        let (_dir2, disk2) = read_multiple_test_disk(bucket, &[(&path, b"two".as_slice())]).await;
+        let set = io_primitives_test_set(vec![Some(disk1.clone()), Some(disk2.clone())], 1).await;
+        let disks = [Some(disk1.clone()), Some(disk2.clone())];
+
+        let tracker = rename_fanout_barrier::observe_tasks(object);
+        let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_CLEANUP);
+
+        let cleanup_fut = set.commit_rename_data_dir(&disks, bucket, object, old_data_dir, committed_data_dir, 2);
+        let control_fut = async {
+            tokio::time::timeout(BARRIER_PAUSE_GUARD, barrier.wait_until_paused())
+                .await
+                .expect("cleanup fan-out must reach the armed cleanup barrier");
+            assert!(tracker.running() >= 1, "a background cleanup delete must still be in flight while paused");
+            barrier.release();
+        };
+        let (cleanup, ()) = tokio::join!(cleanup_fut, control_fut);
+
+        assert_eq!(tracker.running(), 0, "no background cleanup task may remain after drain");
+        assert_eq!(cleanup.attempted, 2);
+        assert_eq!(cleanup.reclaimed, 2, "the real old-data-dir must still be reclaimed after release");
+
+        drop((disk1, disk2));
+    }
+
+    /// Isolation guard: an armed barrier / observed object only affects its own
+    /// object. A fan-out for a different (unobserved, unarmed) object must not be
+    /// paused and must not accrue any tracked task count — so concurrent tests
+    /// using distinct object names never interfere.
+    #[tokio::test]
+    async fn barrier_and_task_tracker_isolate_by_object() {
+        const DISKS: usize = 3;
+        let bucket = "barrier-iso-bucket";
+        let observed_object = "iso-observed-object";
+        let other_object = "iso-other-object";
+        let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+
+        // Observe + arm one object, then run the fan-out for a *different* object.
+        let tracker = rename_fanout_barrier::observe_tasks(observed_object);
+        let _barrier = rename_fanout_barrier::arm(observed_object, 0, rename_fanout_barrier::PHASE_RENAME);
+
+        let file_infos = rename_barrier_fileinfos(other_object, DISKS);
+        // This must run to completion without ever pausing (no barrier for it) and
+        // must not touch the observed object's counter. A hang here (e.g. if the
+        // barrier ignored the object key) would surface as a timeout.
+        let _ = tokio::time::timeout(
+            BARRIER_PAUSE_GUARD,
+            SetDisks::rename_data(&disks, bucket, other_object, &file_infos, bucket, other_object, DISKS - 1),
+        )
+        .await
+        .expect("fan-out for an unarmed object must not pause");
+
+        assert_eq!(
+            tracker.running(),
+            0,
+            "another object's fan-out must not accrue tracked tasks for the observed object"
+        );
+
+        drop(dirs);
+    }
+
     #[tokio::test]
     async fn multipart_codec_streaming_reader_zero_buffer_is_noop() {
         let reader = tokio::io::BufReader::new(Cursor::new(b"payload".to_vec()));

crates/ecstore/src/set_disk/mod.rs

@@ -140,7 +140,7 @@ use rustfs_lock::local_lock::LocalLock;
 use rustfs_lock::{FastLockGuard, LockManager, NamespaceLock, NamespaceLockGuard, NamespaceLockWrapper, ObjectKey};
 use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem, Infos};
 use rustfs_object_capacity::capacity_scope::{
-    CapacityScope, CapacityScopeDisk, record_capacity_scope, record_global_dirty_scope,
+    CapacityScope, CapacityScopeDisk, current_dirty_generation, record_capacity_scope, record_global_dirty_scope,
 };
 use rustfs_s3_types::EventName;
 use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
@@ -164,6 +164,7 @@ use std::hash::Hash;
 use std::mem::{self};
 use std::pin::Pin;
 use std::sync::OnceLock;
+use std::sync::atomic::{AtomicU64, Ordering};
 use std::task::{Context, Poll};
 use std::time::{Instant, SystemTime, UNIX_EPOCH};
 use std::{
@@ -349,10 +350,145 @@ fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -
         || metadata.contains_key(SSEC_KEY_MD5_HEADER)
 }
 
+/// Per-set memoized capacity dirty scope.
+///
+/// The disk endpoint/path identity of each slot in a set is immutable for the
+/// set's lifetime (heal replaces the [`DiskStore`] instance but keeps the same
+/// endpoint and root), so the dirty scope only needs to be built once instead
+/// of allocating a `String` per online disk on every successful write
+/// (backlog#1315). Slots are filled lazily as disks are observed online; once
+/// every slot has contributed, `complete` latches and the hot path returns the
+/// shared `Arc` without any scan.
+#[derive(Default, Debug)]
+struct CapacityScopeCache {
+    /// Per-slot resolved disk identity; `None` slots have not been seen online.
+    slots: Vec<Option<CapacityScopeDisk>>,
+    /// Deduplicated scope covering every slot resolved so far.
+    scope: Option<Arc<CapacityScope>>,
+    /// Latches once every slot is resolved; the scope is then stable.
+    complete: bool,
+}
+
+impl CapacityScopeCache {
+    /// Whether `disks` contains an online disk whose slot has not been recorded
+    /// yet. Read-only, allocation-free (used under the read lock).
+    fn has_unresolved_slot(&self, disks: &[Option<DiskStore>]) -> bool {
+        disks
+            .iter()
+            .enumerate()
+            .any(|(idx, disk)| disk.is_some() && self.slots.get(idx).is_none_or(|slot| slot.is_none()))
+    }
+}
+
+impl SetDisks {
+    /// Return the memoized dirty scope for this set, resolving any newly-online
+    /// slots first. Steady-state writes hit the fast path (a read lock and an
+    /// `Arc` clone) with no per-disk `String` allocation.
+    fn capacity_scope(&self, disks: &[Option<DiskStore>]) -> Arc<CapacityScope> {
+        {
+            let cache = self.capacity_scope_cache.read().unwrap_or_else(|p| p.into_inner());
+            if let Some(scope) = cache.scope.as_ref()
+                && (cache.complete || !cache.has_unresolved_slot(disks))
+            {
+                return scope.clone();
+            }
+        }
+        self.resolve_capacity_scope(disks)
+    }
+
+    /// Slow path: fill newly-observed slots and rebuild the deduplicated scope.
+    fn resolve_capacity_scope(&self, disks: &[Option<DiskStore>]) -> Arc<CapacityScope> {
+        let mut cache = self.capacity_scope_cache.write().unwrap_or_else(|p| p.into_inner());
+        if cache.slots.len() < self.set_drive_count {
+            cache.slots.resize(self.set_drive_count, None);
+        }
+
+        let mut changed = false;
+        for (idx, disk) in disks.iter().enumerate() {
+            let Some(slot) = cache.slots.get_mut(idx) else {
+                break;
+            };
+            if slot.is_some() {
+                continue;
+            }
+            if let Some(disk) = disk {
+                *slot = Some(CapacityScopeDisk {
+                    endpoint: disk.endpoint().to_string(),
+                    drive_path: disk.to_string(),
+                });
+                changed = true;
+            }
+        }
+
+        if changed || cache.scope.is_none() {
+            let mut unique = HashSet::with_capacity(cache.slots.len());
+            let mut scoped_disks = Vec::with_capacity(cache.slots.len());
+            for slot in cache.slots.iter().flatten() {
+                if unique.insert(slot.clone()) {
+                    scoped_disks.push(slot.clone());
+                }
+            }
+            cache.complete = !cache.slots.is_empty() && cache.slots.iter().all(|slot| slot.is_some());
+            cache.scope = Some(Arc::new(CapacityScope { disks: scoped_disks }));
+        }
+
+        cache.scope.clone().unwrap_or_else(|| Arc::new(CapacityScope::default()))
+    }
+
+    /// Record the set's dirty scope after a successful write.
+    ///
+    /// The global dirty registry is only upgraded on the first write of each
+    /// registry generation: once this set has marked its disks, subsequent
+    /// writes skip the global mutex until a refresh drain advances the
+    /// generation, forcing a re-mark (backlog#1315). The scope-token registry
+    /// (multipart completion) still records per token so the app-side write
+    /// settle can consume it.
+    fn record_capacity_scope_if_needed(&self, scope_token: Option<Uuid>, disks: &[Option<DiskStore>]) {
+        let scope = self.capacity_scope(disks);
+        if scope.disks.is_empty() {
+            return;
+        }
+
+        let generation = current_dirty_generation();
+        if self.capacity_dirty_generation.load(Ordering::Acquire) != generation {
+            // First write of this generation (or after a drain): upgrade the
+            // global registry and cache the generation observed under its lock.
+            let observed = record_global_dirty_scope((*scope).clone());
+            self.capacity_dirty_generation.store(observed, Ordering::Release);
+        }
+
+        if let Some(token) = scope_token {
+            record_capacity_scope(token, (*scope).clone());
+        }
+    }
+
+    /// Mark healed disks dirty from the (infrequent) heal path.
+    ///
+    /// Heal passes disks in erasure-distribution order (`shuffle_disks`), not
+    /// physical-slot order, so they must not seed the slot-indexed memo — doing
+    /// so could record a disk under the wrong slot and drop another from the
+    /// steady-state scope. Heal therefore builds an ad-hoc scope from the disks
+    /// it actually rewrote and marks the global registry directly. This runs at
+    /// heal frequency, so it does not use the per-generation skip fast-path
+    /// (backlog#1315).
+    fn record_healed_capacity_scope(&self, disks: &[Option<DiskStore>]) {
+        let scope = capacity_scope_from_disks(disks);
+        if scope.disks.is_empty() {
+            return;
+        }
+        // Do not advance the set's generation marker here: this ad-hoc scope is
+        // only the subset of disks heal rewrote, whereas the marker asserts the
+        // full set was marked. Leaving the marker untouched keeps the next write
+        // free to upgrade the full-set scope if it has not been marked yet.
+        let _ = record_global_dirty_scope(scope);
+    }
+}
+
+/// Build an ad-hoc, deduplicated dirty scope from `disks`. Used by the heal
+/// path where disks are not in physical-slot order (backlog#1315).
 fn capacity_scope_from_disks(disks: &[Option<DiskStore>]) -> CapacityScope {
     let mut unique = HashSet::with_capacity(disks.len());
     let mut scoped_disks = Vec::with_capacity(disks.len());
-
     for disk in disks.iter().flatten() {
         let scope_disk = CapacityScopeDisk {
             endpoint: disk.endpoint().to_string(),
@@ -362,23 +498,9 @@ fn capacity_scope_from_disks(disks: &[Option<DiskStore>]) -> CapacityScope {
             scoped_disks.push(scope_disk);
         }
     }
-
     CapacityScope { disks: scoped_disks }
 }
 
-fn record_capacity_scope_if_needed(scope_token: Option<Uuid>, disks: &[Option<DiskStore>]) {
-    let scope = capacity_scope_from_disks(disks);
-    if scope.disks.is_empty() {
-        return;
-    }
-
-    record_global_dirty_scope(scope.clone());
-
-    if let Some(token) = scope_token {
-        record_capacity_scope(token, scope);
-    }
-}
-
 /// Get the duplex buffer size from environment variable or use default.
 ///
 /// This function reads `RUSTFS_DUPLEX_BUFFER_SIZE` environment variable
@@ -1628,6 +1750,15 @@ pub struct SetDisks {
     /// Slice 3 sources `local_lock_manager` from this context to give each
     /// instance its own lock namespace.
     ctx: Arc<InstanceContext>,
+    /// Memoized capacity dirty scope so successful writes reuse a prebuilt
+    /// `Arc` instead of allocating a `String` per online disk (backlog#1315).
+    /// `Arc` so clones of a set share one memo.
+    capacity_scope_cache: Arc<std::sync::RwLock<CapacityScopeCache>>,
+    /// Last global dirty-registry generation this set marked. `u64::MAX` until
+    /// the first write; equality with the current generation lets steady-state
+    /// writes skip the global registry mutex (backlog#1315). `Arc` so clones of
+    /// a set share one generation marker.
+    capacity_dirty_generation: Arc<AtomicU64>,
 }
 
 #[derive(Clone, Debug, Eq, Hash, PartialEq)]
@@ -1828,6 +1959,8 @@ impl SetDisks {
             // process lock-manager singleton, so this is unchanged.
             local_lock_manager: ctx.lock_manager(),
             ctx,
+            capacity_scope_cache: Arc::new(std::sync::RwLock::new(CapacityScopeCache::default())),
+            capacity_dirty_generation: Arc::new(AtomicU64::new(u64::MAX)),
         })
     }
 
@@ -4628,6 +4761,109 @@ mod tests {
         .await
     }
 
+    // backlog#1315: the memoized dirty scope must be byte-for-byte identical to
+    // the ad-hoc scope the previous per-write construction produced, otherwise
+    // dirty-disk keys diverge from the disk-cache keys and capacity counts drift.
+    #[tokio::test]
+    #[serial]
+    async fn capacity_scope_memo_matches_adhoc_and_is_reused() {
+        use rustfs_object_capacity::capacity_scope::drain_global_dirty_scopes;
+
+        let (dir_a, disk_a) = make_single_local_disk().await;
+        let (dir_b, disk_b) = make_single_local_disk().await;
+        let disks = vec![Some(disk_a), Some(disk_b)];
+        let set = make_set_disks_with(disks.clone()).await;
+
+        let expected = capacity_scope_from_disks(&disks);
+        let first = set.capacity_scope(&disks);
+        assert_eq!(*first, expected, "memoized scope must equal the ad-hoc per-disk construction bit-for-bit");
+
+        // Second call on a fully-resolved set returns the very same Arc (no new
+        // allocation): proving steady-state writes do not rebuild String/HashSet.
+        let second = set.capacity_scope(&disks);
+        assert!(Arc::ptr_eq(&first, &second), "steady-state scope must reuse the cached Arc");
+
+        let _ = drain_global_dirty_scopes();
+        drop((dir_a, dir_b));
+    }
+
+    // backlog#1315: the global registry mutex must be upgraded only on the first
+    // write of each generation; steady-state writes skip it. Reverting the
+    // generation skip makes the upgrade count grow per write and fails this test.
+    #[tokio::test]
+    #[serial]
+    async fn record_capacity_scope_upgrades_registry_once_per_generation() {
+        use rustfs_object_capacity::capacity_scope::{drain_global_dirty_scopes, global_dirty_upgrade_count};
+
+        let (dir_a, disk_a) = make_single_local_disk().await;
+        let (dir_b, disk_b) = make_single_local_disk().await;
+        let disks = vec![Some(disk_a), Some(disk_b)];
+        let set = make_set_disks_with(disks.clone()).await;
+
+        // Start from a clean registry generation.
+        let _ = drain_global_dirty_scopes();
+        let before = global_dirty_upgrade_count();
+
+        // First write of this generation upgrades the registry exactly once.
+        set.record_capacity_scope_if_needed(None, &disks);
+        assert_eq!(
+            global_dirty_upgrade_count(),
+            before + 1,
+            "first write of a generation must upgrade the global registry"
+        );
+
+        // Subsequent writes in the same generation must not touch the mutex.
+        for _ in 0..16 {
+            set.record_capacity_scope_if_needed(None, &disks);
+        }
+        assert_eq!(
+            global_dirty_upgrade_count(),
+            before + 1,
+            "steady-state writes must reuse the generation mark, not re-upgrade"
+        );
+
+        // A drain advances the generation; the next write must re-mark so the
+        // disks it wrote are captured by the following refresh (no lost update).
+        let drained = drain_global_dirty_scopes();
+        assert_eq!(drained.len(), 2, "both set disks must have been recorded dirty");
+        set.record_capacity_scope_if_needed(None, &disks);
+        assert_eq!(
+            global_dirty_upgrade_count(),
+            before + 2,
+            "the first write after a drain must re-upgrade the registry"
+        );
+
+        let _ = drain_global_dirty_scopes();
+        drop((dir_a, dir_b));
+    }
+
+    // backlog#1315: an offline slot must not force the per-write slow path, and
+    // the resolved scope must still cover every online disk.
+    #[tokio::test]
+    #[serial]
+    async fn capacity_scope_tolerates_offline_slot_without_reallocating() {
+        use rustfs_object_capacity::capacity_scope::drain_global_dirty_scopes;
+
+        let (dir_a, disk_a) = make_single_local_disk().await;
+        // Slot 1 is permanently offline (None); the set never resolves it.
+        let disks = vec![Some(disk_a), None];
+        let set = make_set_disks_with(disks.clone()).await;
+
+        let first = set.capacity_scope(&disks);
+        assert_eq!(first.disks.len(), 1, "only the online disk contributes to the scope");
+        // Even though the set is not "complete" (slot 1 unresolved), repeated
+        // writes with the same online set reuse the cached Arc via the
+        // no-unresolved-slot fast path.
+        let second = set.capacity_scope(&disks);
+        assert!(
+            Arc::ptr_eq(&first, &second),
+            "a stable online subset must reuse the cached Arc even with an offline slot"
+        );
+
+        let _ = drain_global_dirty_scopes();
+        drop(dir_a);
+    }
+
     // issue #4189: an orphan directory tree (empty dirs, no xl.meta) must be purged.
     #[tokio::test]
     async fn purge_orphan_dir_object_removes_empty_tree() {
@@ -6288,33 +6524,107 @@ mod tests {
         signature
     }
 
+    // backlog#1321: `classify_rename_convergence` is the single decision that
+    // replaced the old `Option<Vec<u8>>::is_some()` heal gate. These are the
+    // revert-fails white-box cases behind the issue acceptance matrix — if the
+    // decision regressed to "a signature exists => needs heal", the healthy
+    // 4/4 and 8/8 cases below would flip from `AllSuccessIdentical` to a
+    // heal-worthy variant and fail.
     #[test]
-    fn test_reduce_common_versions_requires_write_quorum() {
-        let common = rename_versions_signature(1, 1);
-        let other = rename_versions_signature(2, 1);
+    fn test_classify_rename_convergence_healthy_identical_needs_no_heal() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        let sig = rename_versions_signature(1, 3);
 
-        let disk_versions = vec![Some(common.clone()), Some(common.clone()), Some(other)];
-        let result = SetDisks::reduce_common_versions(&disk_versions, 2);
-        assert_eq!(result, Some(common));
+        // Healthy 4/4: every disk committed with an identical, known signature.
+        let disk_versions = vec![Some(sig.clone()), Some(sig.clone()), Some(sig.clone()), Some(sig.clone())];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::AllSuccessIdentical);
+        assert!(!convergence.needs_heal(), "a fully converged healthy MPU must not enqueue heal");
 
-        let split_versions = vec![
-            Some(rename_versions_signature(1, 1)),
-            Some(rename_versions_signature(2, 1)),
-            None,
-        ];
-        let result = SetDisks::reduce_common_versions(&split_versions, 2);
-        assert_eq!(result, None);
+        // Healthy 8/8: same property at a wider set width.
+        let disk_versions = vec![Some(sig); 8];
+        let errs = vec![None; 8];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::AllSuccessIdentical);
+        assert!(!convergence.needs_heal());
     }
 
     #[test]
-    fn test_select_rename_data_versions_keeps_longer_success_disparity() {
-        let common = rename_versions_signature(1, 1);
-        let longer = rename_versions_signature(2, 2);
-        let disk_versions = vec![Some(common.clone()), Some(common), Some(longer.clone())];
-        let errs = vec![None, None, None];
+    fn test_classify_rename_convergence_three_agree_one_diverges_heals() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        let common = rename_versions_signature(1, 2);
+        let odd = rename_versions_signature(2, 2);
 
-        let result = SetDisks::select_rename_data_versions(&disk_versions, &errs, 2);
-        assert_eq!(result, Some(longer));
+        // 3-same, 1-divergent, all committed: reconcile the odd replica.
+        let disk_versions = vec![Some(common.clone()), Some(common.clone()), Some(common), Some(odd)];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::SignatureDivergent);
+        assert!(convergence.needs_heal());
+    }
+
+    #[test]
+    fn test_classify_rename_convergence_failed_or_offline_disk_heals() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        let sig = rename_versions_signature(1, 2);
+
+        // 1 disk failed/offline while the rest committed identically: past the
+        // write-quorum gate this is a `PartialCommit` — a replica is missing.
+        let disk_versions = vec![Some(sig.clone()), Some(sig.clone()), Some(sig), None];
+        let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::PartialCommit);
+        assert!(convergence.needs_heal());
+    }
+
+    #[test]
+    fn test_classify_rename_convergence_no_common_quorum_heals() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        // No signature holds a majority (2/2 split), every disk committed:
+        // divergence with no common quorum still reconciles via heal.
+        let a = rename_versions_signature(1, 1);
+        let b = rename_versions_signature(2, 1);
+        let disk_versions = vec![Some(a.clone()), Some(a), Some(b.clone()), Some(b)];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::SignatureDivergent);
+        assert!(convergence.needs_heal());
+    }
+
+    #[test]
+    fn test_classify_rename_convergence_over_ten_versions_by_success() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        // >10 versions: every disk deliberately omits the signature (`None`).
+        // All committed => `Unknown` => scanner-backstopped, no self-enqueue.
+        let disk_versions = vec![None, None, None, None];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::Unknown);
+        assert!(
+            !convergence.needs_heal(),
+            ">10-version healthy commit relies on the scanner, not self-enqueue"
+        );
+
+        // Same >10-version shape but with a failed disk: a failure is
+        // conservative-heal regardless of signatures being unavailable.
+        let errs = vec![None, None, None, Some(DiskError::FileCorrupt)];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::PartialCommit);
+        assert!(convergence.needs_heal());
+    }
+
+    #[test]
+    fn test_classify_rename_convergence_mixed_signed_unsigned_heals() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        // A committed replica with <=10 versions (signed) alongside one with
+        // >10 versions (unsigned) is itself a version-count divergence.
+        let sig = rename_versions_signature(1, 2);
+        let disk_versions = vec![Some(sig.clone()), Some(sig.clone()), Some(sig), None];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::SignatureDivergent);
+        assert!(convergence.needs_heal());
     }
 
     #[test]

crates/ecstore/src/set_disk/ops/heal.rs

@@ -682,7 +682,7 @@ impl SetDisks {
                             ));
                         }
 
-                        record_capacity_scope_if_needed(None, &out_dated_disks);
+                        self.record_healed_capacity_scope(&out_dated_disks);
 
                         // The object is healthy here; sweep any data dirs left behind
                         // by pre-#3510 unversioned overwrites, which the dangling paths

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -1433,7 +1433,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         // The trailing `_` drops the rename_data old-size backfill
         // (rustfs/backlog#1009): CompleteMultipartUpload keeps its pre-commit
         // `get_object_info` lookup, so the backfill has no consumer here yet.
-        let (online_disks, versions, op_old_dir, cleanup_disks, _) = Self::rename_data(
+        let (online_disks, convergence, op_old_dir, cleanup_disks, _) = Self::rename_data(
             &shuffle_disks,
             RUSTFS_META_MULTIPART_BUCKET,
             &upload_id_path,
@@ -1485,17 +1485,46 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
 
         drop(object_lock_guard); // drop object lock guard to release the lock
 
-        if let Some(versions) = versions {
-            let _ =
-                rustfs_common::heal_channel::send_heal_request(rustfs_common::heal_channel::create_heal_request_with_options(
-                    bucket.to_string(),
-                    Some(object.to_string()),
-                    false,
-                    Some(HealChannelPriority::Normal),
-                    Some(self.pool_index),
-                    Some(self.set_index),
-                ))
+        // backlog#1321: enqueue heal only when the committed replicas actually
+        // need to converge — a partial commit (some disk failed/offline) or a
+        // signature divergence between committed replicas. A fully healthy MPU
+        // (identical signatures on every disk) is `AllSuccessIdentical` and
+        // submits nothing, which is the fix: the old `Option::is_some()` gate
+        // treated the mere existence of a version signature as "needs heal", so
+        // every healthy <=10-version completion self-enqueued.
+        //
+        // The submit is detached (`tokio::spawn`) so it stays off the ACK
+        // critical path AND survives cancellation of the completion future: the
+        // write is already durable and ACK-worthy, so the heal admission must
+        // not ride the client's request lifetime. The admission itself is
+        // bounded / deduplicated / observable (`send_heal_request` ->
+        // `HealAdmissionResult`), so this emits at most one submit per
+        // completion and coalesces with any in-flight heal for the same object.
+        //
+        // Scanner backstop (backlog#1321 patch): a `PartialCommit` whose
+        // completion is cancelled in the narrow window after the durable commit
+        // but before this spawn runs is not lost — the divergence it would have
+        // healed is exactly what the background scanner reconciles. `Unknown`
+        // (>10 versions, no signature produced) likewise relies on the scanner
+        // rather than self-enqueuing.
+        if convergence.needs_heal() {
+            let bucket = bucket.to_string();
+            let object = object.to_string();
+            let pool_index = self.pool_index;
+            let set_index = self.set_index;
+            tokio::spawn(async move {
+                let _ = rustfs_common::heal_channel::send_heal_request(
+                    rustfs_common::heal_channel::create_heal_request_with_options(
+                        bucket,
+                        Some(object),
+                        false,
+                        Some(HealChannelPriority::Normal),
+                        Some(pool_index),
+                        Some(set_index),
+                    ),
+                )
                 .await;
+            });
         }
 
         let upload_id_path = upload_id_path.clone();
@@ -1513,7 +1542,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
             }
         }
 
-        record_capacity_scope_if_needed(opts.capacity_scope_token, &online_disks);
+        self.record_capacity_scope_if_needed(opts.capacity_scope_token, &online_disks);
 
         fi.is_latest = true;
 

crates/ecstore/src/set_disk/ops/object.rs

@@ -1076,7 +1076,7 @@ impl SetDisks {
             if fi.is_compressed() {
                 record_compression_total_memory(actual_size as u64, w_size as u64).await;
             }
-            record_capacity_scope_if_needed(opts.capacity_scope_token, &online_disks);
+            self.record_capacity_scope_if_needed(opts.capacity_scope_token, &online_disks);
 
             fi.replication_state_internal = Some(replication_state_to_filemeta(&opts.put_replication_state()));
 
@@ -1801,7 +1801,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             }
         }
 
-        record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
+        self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
 
         let mut rollback_futures = Vec::new();
         for fi_vers in &vers {
@@ -1997,7 +1997,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 .map_err(|e| to_object_err(e, vec![bucket, object]))?;
 
             let disks = self.disk_inventory().await;
-            record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
+            self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
 
             let mut oi = ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended);
             oi.replication_decision = goi.replication_decision;
@@ -2027,7 +2027,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             .map_err(|e| to_object_err(e, vec![bucket, object]))?;
 
         let disks = self.disk_inventory().await;
-        record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
+        self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
 
         let mut obj_info = ObjectInfo::from_file_info(&dfi, bucket, object, opts.versioned || opts.version_suspended);
         obj_info.size = goi.size;
@@ -2326,7 +2326,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 "transition completed on remote tier but source cleanup failed; skipping external lifecycle transition notification"
             );
         } else {
-            record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
+            self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
         }
 
         for disk in disks.iter() {
@@ -2384,7 +2384,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         let (actual_fi, _, _) = fi?;
 
         oi = ObjectInfo::from_file_info(&actual_fi, bucket, object, opts.versioned || opts.version_suspended);
-        let ropts = put_restore_opts(bucket, object, &opts.transition.restore_request, &oi).await?;
+        let mut ropts = put_restore_opts(bucket, object, &opts.transition.restore_request, &oi).await?;
+        // The restore copy-back re-writes this same object via put_object /
+        // new_multipart_upload / complete_multipart_upload, each of which takes
+        // the object write lock in its commit phase. The caller
+        // (handle_restore_transitioned_object, #4877) already holds that write
+        // lock for the whole restore and forwards no_lock=true, so the inner
+        // writes must inherit it or they self-deadlock on the lock we already
+        // hold and time out. put_restore_opts builds fresh options that default
+        // no_lock=false, so propagate it explicitly here.
+        ropts.no_lock = opts.no_lock;
         if oi.parts.len() == 1 {
             let mut opts = opts.clone();
             opts.part_number = Some(1);
@@ -2502,6 +2511,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 uploaded_parts,
                 &ObjectOptions {
                     mod_time: oi.mod_time,
+                    // Inherit the restore write lock (see ropts.no_lock above):
+                    // the commit phase re-acquires this object's write lock.
+                    no_lock: opts.no_lock,
                     ..Default::default()
                 },
             )

crates/kms/src/backends/local.rs

@@ -157,8 +157,8 @@ impl LocalKmsClient {
         Ok(salt)
     }
 
+    #[cfg(unix)]
     async fn set_file_permissions(path: &std::path::Path, permissions: Option<u32>) -> Result<()> {
-        #[cfg(unix)]
         if let Some(mode) = permissions {
             use std::os::unix::fs::PermissionsExt;
 
@@ -166,7 +166,11 @@ impl LocalKmsClient {
             fs::set_permissions(path, perms).await?;
         }
 
-        let _ = permissions;
+        Ok(())
+    }
+
+    #[cfg(not(unix))]
+    async fn set_file_permissions(_path: &std::path::Path, _permissions: Option<u32>) -> Result<()> {
         Ok(())
     }
 

crates/object-capacity/src/capacity_manager.rs

@@ -37,6 +37,7 @@ use std::collections::{HashMap, HashSet};
 use std::future::Future;
 use std::panic::AssertUnwindSafe;
 use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
 use std::time::{Duration, Instant};
 use tokio::sync::{Mutex, RwLock, watch};
 use tracing::{debug, info, warn};
@@ -450,21 +451,81 @@ const REFRESH_JOINER_WAIT_TIMEOUT: Duration = Duration::from_secs(300);
 /// never overflow.
 const MAX_BACKGROUND_INTERVAL: Duration = Duration::from_secs(30 * 24 * 60 * 60);
 
-#[derive(Clone, Copy, Debug, Default)]
-struct WriteBucket {
-    second: u64,
-    count: usize,
+/// A single second-granular write bucket packed into one `u64` so it can be
+/// updated with a lock-free compare-and-swap. The high 32 bits hold the
+/// monotonic-second key, the low 32 bits hold the write count for that second.
+/// A per-second count never approaches `u32::MAX`, and the count saturates
+/// rather than wrapping into the second key (backlog#1315).
+#[derive(Default)]
+struct AtomicWriteBucket(AtomicU64);
+
+const WRITE_BUCKET_COUNT_MASK: u64 = 0xFFFF_FFFF;
+
+impl AtomicWriteBucket {
+    #[inline]
+    fn unpack(packed: u64) -> (u64, usize) {
+        let second = packed >> 32;
+        let count = (packed & WRITE_BUCKET_COUNT_MASK) as usize;
+        (second, count)
+    }
+
+    #[inline]
+    fn pack(second: u64, count: usize) -> u64 {
+        let count = (count as u64).min(WRITE_BUCKET_COUNT_MASK);
+        (second << 32) | count
+    }
+
+    /// Record one write for `now_second`, resetting the bucket if it holds a
+    /// different (older or future) second. Lock-free CAS loop tolerating
+    /// concurrent writers landing on the same bucket.
+    fn record(&self, now_second: u64) {
+        loop {
+            let current = self.0.load(Ordering::Acquire);
+            let (second, count) = Self::unpack(current);
+            let next = if second == now_second {
+                Self::pack(now_second, count.saturating_add(1))
+            } else {
+                Self::pack(now_second, 1)
+            };
+            if self
+                .0
+                .compare_exchange_weak(current, next, Ordering::AcqRel, Ordering::Acquire)
+                .is_ok()
+            {
+                return;
+            }
+        }
+    }
+
+    fn snapshot(&self) -> (u64, usize) {
+        Self::unpack(self.0.load(Ordering::Acquire))
+    }
+
+    #[cfg(test)]
+    fn store(&self, second: u64, count: usize) {
+        self.0.store(Self::pack(second, count), Ordering::Release);
+    }
 }
 
-/// Write record for tracking write operations
-#[derive(Debug)]
+/// Lock-free write record for tracking write operations.
+///
+/// Previously guarded by an async `RwLock` taken on every successful write in
+/// the PUT response path; the write side is now a set of relaxed/CAS atomic
+/// updates so concurrent small-object PUTs no longer serialize on a single
+/// writer lock (backlog#1315). Counters use saturating arithmetic and monotonic
+/// second keys, preserving the exact debounce/frequency semantics the refresh
+/// scheduler relies on.
 pub struct WriteRecord {
-    /// Last write time
-    pub last_write_time: Option<Instant>,
-    /// Write count
-    pub write_count: usize,
+    /// Last write time, encoded as monotonic nanoseconds since `epoch`.
+    last_write_nanos: AtomicU64,
+    /// Whether any write has been recorded (`0` == never). Kept separate from
+    /// `last_write_nanos` so a genuine near-zero-nanos first write is still
+    /// distinguished from "no write yet".
+    has_write: AtomicU64,
+    /// Total write count (saturating). Observability only.
+    write_count: AtomicU64,
     /// Fixed-size time buckets for the recent write window.
-    write_buckets: [WriteBucket; WRITE_WINDOW_BUCKETS],
+    write_buckets: [AtomicWriteBucket; WRITE_WINDOW_BUCKETS],
     /// Monotonic origin for bucket keys. Wall-clock keys made an NTP step
     /// backwards mark recent buckets as "future" and silently suppress
     /// write-triggered refreshes until the clock catches up (backlog#1022 S32).
@@ -474,9 +535,10 @@ pub struct WriteRecord {
 impl WriteRecord {
     fn new() -> Self {
         Self {
-            last_write_time: None,
-            write_count: 0,
-            write_buckets: [WriteBucket::default(); WRITE_WINDOW_BUCKETS],
+            last_write_nanos: AtomicU64::new(0),
+            has_write: AtomicU64::new(0),
+            write_count: AtomicU64::new(0),
+            write_buckets: std::array::from_fn(|_| AtomicWriteBucket::default()),
             epoch: Instant::now(),
         }
     }
@@ -485,31 +547,48 @@ impl WriteRecord {
         self.epoch.elapsed().as_secs()
     }
 
+    /// Total number of writes recorded (saturating). Observability only.
+    fn total_write_count(&self) -> u64 {
+        self.write_count.load(Ordering::Relaxed)
+    }
+
+    /// Elapsed time since the last write, or `None` if no write has been
+    /// recorded yet.
+    fn time_since_last_write(&self) -> Option<Duration> {
+        if self.has_write.load(Ordering::Acquire) == 0 {
+            return None;
+        }
+        let last = self.last_write_nanos.load(Ordering::Acquire);
+        let now = self.epoch.elapsed().as_nanos() as u64;
+        Some(Duration::from_nanos(now.saturating_sub(last)))
+    }
+
     fn recent_write_count(&self, now_second: u64) -> usize {
         self.write_buckets
             .iter()
-            .filter(|bucket| {
-                bucket.count > 0 && bucket.second <= now_second && now_second.saturating_sub(bucket.second) < WRITE_WINDOW_SECS
+            .filter_map(|bucket| {
+                let (second, count) = bucket.snapshot();
+                if count > 0 && second <= now_second && now_second.saturating_sub(second) < WRITE_WINDOW_SECS {
+                    Some(count)
+                } else {
+                    None
+                }
             })
-            .map(|bucket| bucket.count)
             .sum()
     }
 
-    fn record_write(&mut self, now: Instant) -> usize {
+    fn record_write(&self, _now: Instant) -> usize {
         let now_second = self.monotonic_second();
         let bucket_idx = (now_second % WRITE_WINDOW_BUCKETS as u64) as usize;
-        let bucket = &mut self.write_buckets[bucket_idx];
+        self.write_buckets[bucket_idx].record(now_second);
 
-        if bucket.second != now_second {
-            *bucket = WriteBucket {
-                second: now_second,
-                count: 0,
-            };
-        }
-
-        bucket.count = bucket.count.saturating_add(1);
-        self.last_write_time = Some(now);
-        self.write_count = self.write_count.saturating_add(1);
+        self.last_write_nanos
+            .store(self.epoch.elapsed().as_nanos() as u64, Ordering::Release);
+        self.has_write.store(1, Ordering::Release);
+        // Single atomic add (wraps only after 2^64 writes; effectively saturating
+        // for any real deployment). Observability only — the frequency window
+        // used for refresh decisions is tracked per-bucket above.
+        self.write_count.fetch_add(1, Ordering::Relaxed);
 
         self.recent_write_count(now_second)
     }
@@ -641,8 +720,9 @@ impl Drop for RefreshLeaderGuard {
 pub struct HybridCapacityManager {
     /// Capacity cache
     cache: Arc<RwLock<Option<CachedCapacity>>>,
-    /// Write record
-    write_record: Arc<RwLock<WriteRecord>>,
+    /// Write record. Lock-free atomics (backlog#1315): the PUT response path no
+    /// longer takes an async writer lock to record a write.
+    write_record: Arc<WriteRecord>,
     /// Dirty disks recorded from write-side scope propagation, keyed to the
     /// instant they were last marked so a commit only clears marks that
     /// predate its scan (backlog#1020 S19).
@@ -680,7 +760,7 @@ impl HybridCapacityManager {
     pub fn new(config: HybridStrategyConfig) -> Self {
         Self {
             cache: Arc::new(RwLock::new(None)),
-            write_record: Arc::new(RwLock::new(WriteRecord::new())),
+            write_record: Arc::new(WriteRecord::new()),
             dirty_disks: Arc::new(RwLock::new(HashMap::new())),
             disk_cache: Arc::new(RwLock::new(HashMap::new())),
             disk_cache_complete: Arc::new(RwLock::new(false)),
@@ -815,9 +895,8 @@ impl HybridCapacityManager {
 
     /// Record write operation
     pub async fn record_write_operation(&self) {
-        let mut record = self.write_record.write().await;
         let now = Instant::now();
-        let recent_write_count = record.record_write(now);
+        let recent_write_count = self.write_record.record_write(now);
 
         record_capacity_write_operation(recent_write_count);
         debug!(
@@ -825,7 +904,7 @@ impl HybridCapacityManager {
             component = LOG_COMPONENT_CAPACITY,
             subsystem = LOG_SUBSYSTEM_REFRESH,
             state = "recorded",
-            total_writes = record.write_count,
+            total_writes = self.write_record.total_write_count(),
             recent_writes = recent_write_count,
             "capacity refresh write recorded"
         );
@@ -873,15 +952,13 @@ impl HybridCapacityManager {
                 return false;
             }
 
-            let write_record = self.write_record.read().await;
+            let write_record = &self.write_record;
             let write_frequency = write_record.recent_write_count(write_record.monotonic_second());
             if write_frequency <= self.config.write_frequency_threshold {
                 return false;
             }
 
-            if let Some(last_write_time) = write_record.last_write_time {
-                let time_since_write = last_write_time.elapsed();
-
+            if let Some(time_since_write) = write_record.time_since_last_write() {
                 if time_since_write < self.config.write_trigger_delay {
                     debug!(
                         event = EVENT_CAPACITY_REFRESH_DEBOUNCE_STATE,
@@ -923,7 +1000,7 @@ impl HybridCapacityManager {
     /// Get write frequency (writes/minute)
     #[allow(dead_code)]
     pub async fn get_write_frequency(&self) -> usize {
-        let record = self.write_record.read().await;
+        let record = &self.write_record;
         record.recent_write_count(record.monotonic_second())
     }
 
@@ -1465,15 +1542,9 @@ mod tests {
     #[test]
     #[serial]
     fn test_recent_write_count_ignores_future_buckets() {
-        let mut record = WriteRecord {
-            last_write_time: None,
-            write_count: 1,
-            write_buckets: [WriteBucket::default(); WRITE_WINDOW_BUCKETS],
-            epoch: Instant::now(),
-        };
-
-        record.write_buckets[0] = WriteBucket { second: 120, count: 3 };
-        record.write_buckets[1] = WriteBucket { second: 90, count: 2 };
+        let record = WriteRecord::new();
+        record.write_buckets[0].store(120, 3);
+        record.write_buckets[1].store(90, 2);
 
         assert_eq!(
             record.recent_write_count(100),
@@ -1616,6 +1687,35 @@ mod tests {
         assert_eq!(manager.get_write_frequency().await, 10);
     }
 
+    // backlog#1315: the lock-free write record must not drop concurrent writes.
+    // The previous async RwLock serialized every write; the CAS bucket must be
+    // exact under heavy same-second contention or the frequency window (and the
+    // write-trigger decision) would undercount.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 8)]
+    #[serial]
+    async fn test_record_write_operation_lock_free_is_exact_under_contention() {
+        let manager = Arc::new(HybridCapacityManager::from_env());
+        let mut handles = Vec::new();
+        const WRITERS: usize = 16;
+        const PER_WRITER: usize = 64;
+
+        for _ in 0..WRITERS {
+            let mgr = manager.clone();
+            handles.push(tokio::spawn(async move {
+                for _ in 0..PER_WRITER {
+                    mgr.record_write_operation().await;
+                }
+            }));
+        }
+        for handle in handles {
+            handle.await.unwrap();
+        }
+
+        // All writes land in the same monotonic second (the test runs well under
+        // one second), so the recent-window frequency must equal the exact total.
+        assert_eq!(manager.get_write_frequency().await, WRITERS * PER_WRITER);
+    }
+
     #[tokio::test]
     #[serial]
     async fn test_performance_overhead() {

crates/object-capacity/src/capacity_scope.rs

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 use std::collections::{HashMap, HashSet};
+use std::sync::atomic::{AtomicU64, Ordering};
 use std::sync::{Mutex, OnceLock};
 use std::time::{Duration, Instant};
 use uuid::Uuid;
@@ -48,6 +49,36 @@ fn global_dirty_scope_registry() -> &'static Mutex<HashSet<CapacityScopeDisk>> {
     REGISTRY.get_or_init(|| Mutex::new(HashSet::new()))
 }
 
+/// Monotonic generation of the global dirty-scope registry.
+///
+/// Advanced every time a non-empty drain removes disks from the registry. A
+/// storage-set that recorded its disks at generation `g` can safely skip the
+/// registry mutex on subsequent writes as long as the generation is still `g`:
+/// any drain that could have removed its disks would have advanced the counter,
+/// forcing the set to re-mark (backlog#1315). The generation is loaded and
+/// advanced only while the registry mutex is held, so a set that observes
+/// `generation == g` at record time is guaranteed its disks are still present
+/// until the next drain.
+static DIRTY_GENERATION: AtomicU64 = AtomicU64::new(0);
+
+/// Test-only counter of how many times the global registry mutex was upgraded
+/// to insert dirty disks. Steady-state writes must reuse an existing generation
+/// mark and leave this untouched; only the first write of each generation
+/// bumps it. Used by white-box tests to prove the per-generation skip holds and
+/// to fail closed if the optimization regresses (backlog#1315).
+static GLOBAL_DIRTY_UPGRADE_COUNT: AtomicU64 = AtomicU64::new(0);
+
+/// Current global dirty-scope generation. See [`DIRTY_GENERATION`].
+pub fn current_dirty_generation() -> u64 {
+    DIRTY_GENERATION.load(Ordering::Acquire)
+}
+
+/// Number of times the global dirty registry mutex was upgraded to record
+/// disks. Test/observability hook (backlog#1315).
+pub fn global_dirty_upgrade_count() -> u64 {
+    GLOBAL_DIRTY_UPGRADE_COUNT.load(Ordering::Relaxed)
+}
+
 fn prune_expired_entries(entries: &mut HashMap<Uuid, CapacityScopeEntry>, now: Instant) {
     entries.retain(|_, entry| now.duration_since(entry.recorded_at) <= CAPACITY_SCOPE_TTL);
 }
@@ -107,17 +138,43 @@ pub fn take_capacity_scope(token: Uuid) -> Option<CapacityScope> {
     Some(entry.scope)
 }
 
-pub fn record_global_dirty_scope(scope: CapacityScope) {
+/// Record dirty disks in the global registry, returning the registry generation
+/// observed while the mutex was held.
+///
+/// Callers cache the returned generation and, on subsequent writes, compare it
+/// against [`current_dirty_generation`]: while it is unchanged the disks are
+/// still queued for the next drain and the registry mutex can be skipped
+/// entirely (backlog#1315). An empty scope is a no-op that returns the current
+/// generation without touching the mutex.
+pub fn record_global_dirty_scope(scope: CapacityScope) -> u64 {
     if scope.disks.is_empty() {
-        return;
+        return current_dirty_generation();
     }
 
     let mut dirty_scopes = global_dirty_scope_registry().lock().unwrap_or_else(|p| p.into_inner());
     dirty_scopes.extend(scope.disks);
+    GLOBAL_DIRTY_UPGRADE_COUNT.fetch_add(1, Ordering::Relaxed);
+    // Load under the registry lock so the value is coherent with any concurrent
+    // drain (which advances the generation under the same lock). A set that
+    // stores this value only re-marks once a later drain moves past it.
+    DIRTY_GENERATION.load(Ordering::Acquire)
 }
 
 pub fn drain_global_dirty_scopes() -> Vec<CapacityScopeDisk> {
     let mut dirty_scopes = global_dirty_scope_registry().lock().unwrap_or_else(|p| p.into_inner());
+    if dirty_scopes.is_empty() {
+        // Nothing to drain: leave the generation untouched so sets that already
+        // marked keep their skip fast-path. Advancing here would only force
+        // redundant re-marks without changing correctness.
+        return Vec::new();
+    }
+    // Advance before draining so the new generation is visible under the lock;
+    // any set that recorded at the old generation will observe the change and
+    // re-mark on its next write, and the disks it just wrote are read by the
+    // refresh that consumes this drain (the write commits before it records the
+    // scope, and this bump is ordered after that record in the registry's
+    // modification order).
+    DIRTY_GENERATION.fetch_add(1, Ordering::AcqRel);
     dirty_scopes.drain().collect()
 }
 
@@ -141,6 +198,8 @@ mod tests {
             .lock()
             .unwrap_or_else(|poisoned| poisoned.into_inner())
             .clear();
+        DIRTY_GENERATION.store(0, Ordering::Release);
+        GLOBAL_DIRTY_UPGRADE_COUNT.store(0, Ordering::Release);
     }
 
     fn poison_capacity_scope_registry_for_test() {
@@ -329,6 +388,65 @@ mod tests {
         clear_capacity_scope_registry_for_test();
     }
 
+    #[test]
+    fn record_global_dirty_scope_generation_is_stable_until_drain() {
+        let _guard = test_lock().lock().expect("test lock poisoned");
+        clear_capacity_scope_registry_for_test();
+
+        let disk = CapacityScopeDisk {
+            endpoint: "node-a".to_string(),
+            drive_path: "/tmp/disk-a".to_string(),
+        };
+        let scope = CapacityScope {
+            disks: vec![disk.clone()],
+        };
+
+        // First record upgrades the registry and returns the current generation.
+        let gen0 = record_global_dirty_scope(scope.clone());
+        assert_eq!(gen0, current_dirty_generation());
+        assert_eq!(global_dirty_upgrade_count(), 1);
+
+        // A subsequent record while the generation is unchanged still returns
+        // the same generation; the caller's skip fast-path keys off equality.
+        let gen1 = record_global_dirty_scope(scope);
+        assert_eq!(gen1, gen0);
+        assert_eq!(global_dirty_upgrade_count(), 2);
+
+        // Draining advances the generation so a caller that cached gen0 is
+        // forced to re-mark on its next write.
+        let drained = drain_global_dirty_scopes();
+        assert_eq!(drained, vec![disk]);
+        assert_ne!(current_dirty_generation(), gen0);
+        assert_eq!(current_dirty_generation(), gen0 + 1);
+
+        clear_capacity_scope_registry_for_test();
+    }
+
+    #[test]
+    fn drain_empty_registry_does_not_advance_generation() {
+        let _guard = test_lock().lock().expect("test lock poisoned");
+        clear_capacity_scope_registry_for_test();
+
+        let before = current_dirty_generation();
+        assert!(drain_global_dirty_scopes().is_empty());
+        assert_eq!(current_dirty_generation(), before);
+
+        clear_capacity_scope_registry_for_test();
+    }
+
+    #[test]
+    fn record_empty_scope_is_noop_without_upgrade() {
+        let _guard = test_lock().lock().expect("test lock poisoned");
+        clear_capacity_scope_registry_for_test();
+
+        let observed = record_global_dirty_scope(CapacityScope::default());
+        assert_eq!(observed, current_dirty_generation());
+        assert_eq!(global_dirty_upgrade_count(), 0);
+        assert!(drain_global_dirty_scopes().is_empty());
+
+        clear_capacity_scope_registry_for_test();
+    }
+
     #[test]
     fn record_global_dirty_scope_recovers_from_poisoned_registry() {
         let _guard = test_lock().lock().expect("test lock poisoned");

crates/policy/src/auth/credentials.rs

@@ -123,7 +123,7 @@ pub fn create_new_credentials_with_metadata(
     }
 
     if sk.len() < SECRET_KEY_MIN_LEN || sk.len() > SECRET_KEY_MAX_LEN {
-        return Err(Error::InvalidAccessKeyLength);
+        return Err(Error::InvalidSecretKeyLength);
     }
 
     if token_secret.is_empty() {
@@ -420,7 +420,8 @@ impl TryFrom<CredentialsBuilder> for Credentials {
 
 #[cfg(test)]
 mod reserved_chars_tests {
-    use super::contains_reserved_chars;
+    use super::{contains_reserved_chars, create_new_credentials_with_metadata};
+    use std::collections::HashMap;
 
     #[test]
     fn detects_any_reserved_char_not_just_the_substring() {
@@ -440,4 +441,13 @@ mod reserved_chars_tests {
         assert!(!contains_reserved_chars("AKIAIOSFODNN7EXAMPLE"));
         assert!(!contains_reserved_chars("group-name_1"));
     }
+
+    #[test]
+    fn credential_creation_reports_secret_key_length_errors() {
+        let err =
+            create_new_credentials_with_metadata("AKIAIOSFODNN7EXAMPLE", "short", &HashMap::new(), "token-secret-for-tests")
+                .expect_err("short secret key should fail");
+
+        assert!(matches!(err, crate::error::Error::InvalidSecretKeyLength));
+    }
 }

crates/rio/src/http_reader.rs

@@ -976,21 +976,21 @@ impl Stream for ReceiverStream {
     }
 }
 
-pin_project! {
-    pub struct HttpWriter {
-        url:String,
-        method: Method,
-        headers: HeaderMap,
-        err_rx: tokio::sync::oneshot::Receiver<std::io::Error>,
-        start_tx: Option<tokio::sync::oneshot::Sender<()>>,
-        sender: PollSender<Option<Bytes>>,
-        handle: tokio::task::JoinHandle<std::io::Result<()>>,
-        pending_chunk: BytesMut,
-        finish:bool,
-        track_internode_metrics: bool,
-        internode_operation: Option<&'static str>,
-
-    }
+// Not pin-projected: every field is `Unpin` and the `AsyncWrite` impl accesses
+// them through `get_mut()`, so a plain struct lets us add a manual `Drop`
+// (pin-project forbids one) to abort the background HTTP task — see below.
+pub struct HttpWriter {
+    url: String,
+    method: Method,
+    headers: HeaderMap,
+    err_rx: tokio::sync::oneshot::Receiver<std::io::Error>,
+    start_tx: Option<tokio::sync::oneshot::Sender<()>>,
+    sender: PollSender<Option<Bytes>>,
+    handle: tokio::task::JoinHandle<std::io::Result<()>>,
+    pending_chunk: BytesMut,
+    finish: bool,
+    track_internode_metrics: bool,
+    internode_operation: Option<&'static str>,
 }
 
 const HTTP_WRITER_CHANNEL_CAPACITY: usize = 8;
@@ -1476,6 +1476,20 @@ impl AsyncWrite for HttpWriter {
     }
 }
 
+impl Drop for HttpWriter {
+    /// Abort the background HTTP request when the writer is dropped without a
+    /// clean shutdown. On a stall timeout the caller drops this writer to fail
+    /// its shard (rustfs/backlog#1319); a black-hole peer would otherwise leave
+    /// the spawned task holding the connection and its buffered body alive. A
+    /// cleanly shut-down writer has already joined the task, so aborting a
+    /// finished handle is a no-op. This does not — and cannot — unsend bytes
+    /// already handed to the transport: those land only in the upload's unique
+    /// tmp path and are reclaimed by tmp GC, never touching a committed object.
+    fn drop(&mut self) {
+        self.handle.abort();
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1877,18 +1891,10 @@ mod tests {
         };
 
         let writer = HttpWriter::new(url, Method::PUT, HeaderMap::new()).await.unwrap();
-        let HttpWriter {
-            handle,
-            sender,
-            start_tx,
-            ..
-        } = writer;
-        drop(start_tx);
-        drop(sender);
-        handle
-            .await
-            .expect("HttpWriter background task should not panic")
-            .expect("an unstarted HttpWriter should stop cleanly");
+        // Dropping an unstarted writer aborts its parked background task
+        // (rustfs/backlog#1319) before it can ever send a PUT.
+        drop(writer);
+        tokio::time::sleep(Duration::from_millis(50)).await;
 
         assert_eq!(state.put_count.load(Ordering::SeqCst), 0);
         assert!(state.put_bodies.lock().await.is_empty());
@@ -1896,6 +1902,70 @@ mod tests {
         server_handle.abort();
     }
 
+    /// A PUT handler that registers the request, then parks forever without ever
+    /// sending a response — modeling a black-hole peer that accepts the
+    /// connection but never completes the request, so the writer's background
+    /// task stays parked at `request.send().await` until it is aborted.
+    async fn hanging_put(State(state): State<TestState>, _body: Body) -> impl IntoResponse {
+        state.put_count.fetch_add(1, Ordering::SeqCst);
+        std::future::pending::<()>().await;
+        StatusCode::OK
+    }
+
+    // rustfs/backlog#1319: when a stalled remote writer is dropped (the encode
+    // path drops it to fail its shard), the background HTTP task must be aborted
+    // so it stops holding the connection and buffered body — it must not linger.
+    #[tokio::test]
+    async fn http_writer_drop_aborts_background_request_to_hanging_peer() {
+        let state = TestState::default();
+        let listener = match TcpListener::bind("127.0.0.1:0").await {
+            Ok(listener) => listener,
+            Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return,
+            Err(err) => panic!("test listener should bind: {err}"),
+        };
+        let addr = listener.local_addr().expect("listener local address should be available");
+        let app = Router::new()
+            .route("/hang", axum::routing::put(hanging_put))
+            .with_state(state.clone());
+        let server_handle = tokio::spawn(async move {
+            axum::serve(listener, app).await.unwrap();
+        });
+
+        let url = format!("http://{addr}/hang");
+        let mut writer = HttpWriter::new(url, Method::PUT, HeaderMap::new()).await.unwrap();
+        // A >1MiB write starts the request and hands the body to the background
+        // task, which then parks in the handler that never responds.
+        writer.write_all(&vec![0xa5u8; HTTP_WRITER_BUFFER_SIZE + 1]).await.unwrap();
+        // Let the server register the request, so the background task is parked
+        // (alive) at send() before we drop the writer.
+        for _ in 0..100 {
+            if state.put_count.load(Ordering::SeqCst) >= 1 {
+                break;
+            }
+            tokio::time::sleep(Duration::from_millis(10)).await;
+        }
+        assert_eq!(state.put_count.load(Ordering::SeqCst), 1, "server should have accepted the request");
+
+        // Observe the background task via its abort handle; it must still be
+        // running before the drop, then be aborted (finished) after it.
+        let task = writer.handle.abort_handle();
+        assert!(!task.is_finished(), "background task should still be running before drop");
+
+        drop(writer);
+
+        let mut aborted = false;
+        for _ in 0..500 {
+            if task.is_finished() {
+                aborted = true;
+                break;
+            }
+            tokio::time::sleep(Duration::from_millis(10)).await;
+        }
+        assert!(aborted, "dropping the writer must abort the background HTTP task");
+
+        server_handle.abort();
+    }
+
     #[tokio::test]
     async fn http_writer_shutdown_without_write_sends_empty_put() {
         let state = TestState::default();

crates/scanner/tests/lifecycle_integration_test.rs

@@ -1780,7 +1780,7 @@ mod serial_tests {
     /// tier again -> a second restore succeeds.
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
     #[serial]
-    #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-8)"]
+    #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-8); currently excluded there - DeleteRestoredAction/expire_restored delete semantics are unimplemented, so cleanup removes the whole object"]
     async fn test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore() {
         let (_disk_paths, ecstore) = setup_test_env().await;
 
@@ -1788,7 +1788,10 @@ mod serial_tests {
         let backend = register_mock_tier(&tier_name).await;
 
         let bucket_name = format!("test-restore-chain-{}", &Uuid::new_v4().simple().to_string()[..8]);
-        let object_name = "restore/report.bin";
+        // Must live under the `test/` prefix: `set_bucket_lifecycle_transition_with_tier`
+        // installs a transition rule filtered on `test/`, so any other key never
+        // transitions and `wait_for_transition` below times out.
+        let object_name = "test/restore/report.bin";
         // Position-dependent payload so a misaligned read is caught.
         let payload: Vec<u8> = (0..256 * 1024).map(|i| (i % 251) as u8).collect();
 
@@ -1913,7 +1916,10 @@ mod serial_tests {
         let _backend = register_mock_tier(&tier_name).await;
 
         let bucket_name = format!("test-restore-mpu3-{}", &Uuid::new_v4().simple().to_string()[..8]);
-        let object_name = "restore/multipart.bin";
+        // Must live under the `test/` prefix (see the transition rule filter in
+        // `set_bucket_lifecycle_transition_with_tier`) or the object never
+        // transitions and `wait_for_transition` below times out.
+        let object_name = "test/restore/multipart.bin";
         // Three parts: 5 MiB + 5 MiB + small tail, with position-dependent
         // bytes so any part-boundary mixup is caught.
         let part_sizes = [5 * 1024 * 1024usize, 5 * 1024 * 1024, 4096];

docs/architecture/README.md

@@ -30,6 +30,7 @@ Two rules keep this directory healthy:
 ## Contracts & invariants
 
 - [placement-repair-invariants.md](placement-repair-invariants.md)
+- [unified-object-generation.md](unified-object-generation.md) — single per-object generation authority (fencing epoch, transport/encoding/proto/mixed-version contracts)
 - [runtime-capability-contracts.md](runtime-capability-contracts.md)
 - [workload-admission-contracts.md](workload-admission-contracts.md)
 - [background-controller-contract.md](background-controller-contract.md)

docs/architecture/unified-object-generation.md

@@ -0,0 +1,260 @@
+# Unified Per-Object Generation Authority
+
+Establishes a **single per-object generation authority** that spans object
+commit, GET snapshots, garbage collection, and quota accounting, and pins the
+transport, encoding, proto-evolution, and mixed-version contracts that every
+consumer must obey.
+
+This is a **design and contract document**. It changes no storage code. It is
+the shared prerequisite for five implementation sub-issues under the
+[#1307](https://github.com/rustfs/backlog/issues/1307) adversarial-review
+program:
+[#1312](https://github.com/rustfs/backlog/issues/1312) (commit fencing),
+[#1313](https://github.com/rustfs/backlog/issues/1313) (read lease),
+[#1314](https://github.com/rustfs/backlog/issues/1314) (prepared pool read),
+[#1318](https://github.com/rustfs/backlog/issues/1318) (quota reservation), and
+[#1323](https://github.com/rustfs/backlog/issues/1323) (old-dir GC).
+
+Tracks [rustfs/backlog#1326](https://github.com/rustfs/backlog/issues/1326).
+
+## Why one authority
+
+The #1307 adversarial-review verdict (issuecomment-4992565957) found that the
+five sub-issues each reach for their own generation / fencing / lease token to
+solve the same underlying problem — **commit mutual-exclusion plus snapshot
+lifetime**. Left independent, they diverge and punch through one another:
+
+- #1323 old-dir GC can reclaim a directory still referenced by a #1313 lease if
+  the two disagree on what "current generation" means.
+- #1312 fence epoch and #1318 quota reservation token, if derived from two
+  different monotonic sources, cannot be compared — a late commit fenced on one
+  plane can still settle quota on the other.
+
+The fix is a single authority with one monotonic source, one persistence
+semantics, and one transport binding, that every consumer references rather than
+re-derives.
+
+## The authority (single source)
+
+**The per-object fencing epoch defined by #1312 is the sole generation
+authority.** No other monotonic counter, timestamp, or random token may stand in
+for generation.
+
+- The distributed lock grant returns a monotonic `epoch` for the object key.
+  Acquiring the object write-lock is the only way to mint a new generation.
+- The epoch travels down the authoritative commit path (with
+  `RenameDataRequest` / the local `DiskAPI` call) and is compared at each disk's
+  atomic `xl.meta` commit point, rejecting stale epochs. It adds no extra
+  network round trip (#1312 implementation clause 2).
+- Every consumer in the table below **binds** this epoch. None defines its own.
+
+### Monotonicity persistence semantics
+
+The epoch must be **monotonic across lock-plane restart and failover**
+(#1312 B4). Today the distributed lock entry is in-memory only
+(`crates/lock/src/distributed_lock.rs` has no persistence path), so a lock-service
+restart resets the counter to zero: a new writer draws epoch 1 while disks have
+already observed epoch 100, producing either a permanent write rejection or a
+fence *inversion*. To prevent this, the epoch must be one of:
+
+1. **Quorum-persisted** before it is handed to a writer, or
+2. **Derived from a durable monotonic source** — a `(term, counter)` pair where
+   `term` advances on every lock-service leadership change and is itself durable,
+   so the composite never regresses even when `counter` resets.
+
+The comparison at the disk commit point is on the full composite; a lower
+`(term, counter)` is always rejected.
+
+## Consumer binding contracts
+
+| Consumer | How it binds generation | Key invariant |
+|---|---|---|
+| #1312 commit fence | epoch compared at three disk-write points — `rename`, rollback `delete`, and `commit_rename_data_dir` cleanup | stale epoch rejected on **all** disks; an already-ACK'd write is never rolled back |
+| #1313 read lease | lease binds the generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
+| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms no lease referencing a lower generation still points at it | `old_dir != committed_dir`; a still-referenced directory is never deleted |
+| #1314 prepared pool read | the `PreparedPoolRead` bundle carries the generation resolved during pool lookup; the chosen pool's reader setup reuses it only after a match | generation mismatch forces a fallback to full metadata fanout |
+| #1318 quota reservation | reservation / settle token binds the object generation | a late commit holding an old-generation token cannot settle a newer generation |
+
+### Fence coverage is three disk-write points, not one (#1312 B2)
+
+Comparing the epoch at the `rename` commit point alone is insufficient. The
+authoritative commit sequence is `tmp sync → data-dir rename → xl.meta commit →
+directory sync` in `crates/ecstore/src/disk/local.rs`, and there are two further
+detachable disk-write points in
+`crates/ecstore/src/set_disk/core/io_primitives.rs`:
+
+- **Rollback delete** — on quorum failure each disk runs
+  `delete_version(undo_write=true)`. A fenced old writer's rollback must also
+  compare epoch, otherwise it deletes the winner's already-committed version.
+- **`commit_rename_data_dir`** — a cancel-then-detach disk-write point; the
+  coordinator's "reap all child tasks" must explicitly include it so a cancelled
+  writer cannot bypass fence/lease and keep deleting directories.
+
+If the epoch is validated only at the `xl.meta` commit point, a fenced writer
+may already have renamed its data-dir into the object path, leaving a staged
+orphan. Either move the fence ahead of the data-dir rename, or declare that
+orphan an acceptable residue accounted for by GC metrics — the white-box
+acceptance "no background disk write after release" must be rewritten
+accordingly.
+
+### Post-commit convergence is orthogonal to the fence (#1321)
+
+The same `SetDisks::rename_data` path already returns a post-commit
+convergence classification (`RenameConvergence`, rustfs/backlog#1321) that
+tells the caller whether the *committed* replicas need heal to converge —
+`AllSuccessIdentical` (no heal), `PartialCommit` (a replica failed/offline),
+`SignatureDivergent` (committed replicas' version signatures differ), or
+`Unknown` (no signature was produced, e.g. >10 versions — scanner-backstopped).
+This replaced an earlier `Option<Vec<u8>>` heuristic under which any
+version signature looked like "needs heal", so every healthy multipart
+completion self-enqueued.
+
+Convergence is a *post-commit* signal (the write landed; do the replicas need
+reconciliation), whereas the #1312 fence is a *commit* gate (a stale epoch is
+rejected before the write lands, surfaced through the existing `Result::Err`
+channel). They compose on the one `rename_data` path rather than competing:
+the fence decides whether a convergence is produced at all, and
+`RenameConvergence` classifies it once produced. A future fence-aware
+convergence variant, if ever needed, is an additive change to that enum and
+does not disturb the epoch comparison at the disk-write points above.
+
+## Transport and security contract
+
+Generation and all derived tokens (lease, reservation) cross node boundaries in
+internode RPC bodies. Every such flow must be signature-bound.
+
+### RPC signature binding (#1312 B3, #1313, #1318)
+
+**Requirement.** The RPC body digest carrying a generation/epoch/token must be
+folded into the RPC HMAC, binding `method + object key + generation`, and the
+request must carry a nonce / one-shot identifier inside the 300s replay window.
+The nonce is only meaningful if the **receiver enforces it**: each disk keeps a
+bounded seen-nonce cache covering the 300s freshness window and rejects any
+request whose nonce was already observed. A nonce that is merely transmitted but
+not checked provides no replay protection.
+
+This generalizes the existing `walk_dir` pattern: `walk_dir` computes a
+`Sha256` of the request body and places it in the signed URL query as
+`walk_dir_body_sha256`
+(`crates/ecstore/src/cluster/rpc/internode_data_transport.rs:187`), so the body
+digest is transitively covered by the URL signature. New generation-bearing RPCs
+adopt the same `*_body_sha256` mechanism.
+
+**Current gap (verified).** The internode HMAC covers only
+`{path_and_query}|{method}|{timestamp}`
+(`signature_payload`, `crates/ecstore/src/cluster/rpc/http_auth.rs:75-83`). It
+binds neither the request body nor a nonce, and the 300s freshness window has no
+one-shot guard. Without the binding above:
+
+- An on-path or replaying attacker can inject a high epoch (e.g. `u32::MAX`) and
+  **permanently fence out** a key's legitimate writes — monotonicity only
+  rejects *low/old* epochs, never a forged-high one.
+- A captured lease/reservation token can be replayed within 300s to block
+  old-dir GC (storage-exhaustion DoS) or to double-reserve / prematurely settle
+  quota.
+
+Acceptance for each consumer must include: "a replayed old signature to a
+different method, and a forged-high-epoch request, are both rejected."
+
+### Encoding contract (#1312 B1)
+
+The on-disk persistence of generation must not perturb the file format:
+
+- **Do not bump `XL_META_VERSION` / `XL_HEADER_VERSION`.**
+  `crates/filemeta/src/filemeta/codec.rs` rejects `meta_ver > 3` and
+  `header_ver > 3` outright (`decode_xl_headers`), and both constants are `3`
+  (`crates/filemeta/src/filemeta.rs:53-54`). Bumping either makes every new
+  `xl.meta` unreadable by rolling-upgrade old RustFS nodes and by MinIO — a
+  total read failure, not a graceful downgrade.
+- **Do not add generation as a `FileInfo` struct field.** The internode RPC layer serializes `FileInfo` with two different msgpack encoders depending on the call site: `encode_msgpack` uses rmp_serde's default **array** (positional) encoding for the `read_version` family, where a new positional field breaks decode across mixed-version nodes; `encode_msgpack_named` uses `.with_struct_map()` (named-map) encoding for `rename_data` (`crates/ecstore/src/cluster/rpc/remote_disk.rs`), which is more tolerant but still requires `#[serde(default)]` and MinIO-side agreement. Because a `FileInfo` field would have to be correct under *both* encoders and under the JSON compatibility twin (see "Wire-encoding migration" below), do not add one — use the metadata map, which rides through every encoder unchanged.
+- **Where it may live.** Only inside a version's internal metadata **map**
+  (MinIO skips unknown internal keys and the map encoding is extensible) or in a
+  per-disk sidecar outside `xl.meta`. If it goes in the metadata map, it must
+  obey the dual-key contract (`x-rustfs-internal-*` / `x-minio-internal-*`, see
+  AGENTS.md "Cross-Cutting Domain Invariants").
+- **Regression guard.** Preserve the #4377 real-MinIO `xl.meta` interop
+  regression (the fixture family around `crates/filemeta/src/filemeta.rs`):
+  objects written by a new node must still be readable by old RustFS nodes and
+  by MinIO, in both upgrade and downgrade directions.
+
+### Wire-encoding migration (JSON → msgpack) interaction
+
+The internode RPC layer is mid-migration from JSON to msgpack binary, and generation-bearing fields must respect that migration window — this is not optional context, it changes how epoch is transported.
+
+- **Dual-field transport.** Each dual-encoded RPC field exists twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes _bin` field (e.g. `file_info` #4 alongside `file_info_bin` #7 on `RenameDataRequest`). Senders emit both; receivers `decode_msgpack_or_json` prefer the `_bin` form and fall back to the JSON string only when `_bin` is empty (`crates/ecstore/src/cluster/rpc/remote_disk.rs`).
+- **Capability flag, default off.** `rustfs_protos::internode_rpc_msgpack_only()` (env `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY`, default **false**) gates dropping the redundant JSON copy. It may only be flipped after the `record_msgpack_json_fallback` metric reads zero fleet-wide and the convergence runbook is followed (`crates/protos/src/lib.rs:146`). **Reuse this exact capability + metric-reads-zero model as the mixed-version gate for generation** rather than inventing a parallel handshake; the section above ("Capability negotiation") is layered on top of it, not instead of it.
+- **Generation must ride both encodings during the window.** If epoch lives in the version's internal metadata map, that map is carried inside `FileInfo`, so it is present in both the msgpack `_bin` and JSON copies automatically — good. But any new *top-level* generation datum must be added to **both** the msgpack and JSON representations (and, for msgpack, be safe under both the array and named-map encoders). A field added to only one encoding is silently lost the moment a peer falls back to the other — exactly the failure the JSON-fallback metric exists to catch.
+- **Signature must bind a canonical form.** Because a field is transmitted as both JSON and msgpack and a peer may consume either, the body-digest binding in "RPC signature binding" above must be computed over a single canonical representation (the msgpack `_bin` bytes) — not over whichever copy happened to be decoded. Once the `generation` capability is negotiated for a request, a fenced / generation-bearing request must **reject the JSON fallback path** so a downgrade to the unsigned/loosely-bound JSON copy cannot bypass the epoch check.
+
+### Proto evolution
+
+New generation/epoch proto fields use **proto3 `optional`** (explicit presence).
+A non-optional field is forbidden: an old coordinator talking to a new disk
+decodes an absent field as `0`, which is indistinguishable from a real
+`epoch == 0` and silently breaks the "stale epoch rejected" invariant during
+upgrade.
+
+### Mixed-version gate — one direction
+
+When the cluster-level generation capability is **not** negotiated on every
+target disk, the behavior **falls back to current semantics** (existing lock +
+`is_lock_lost()` check for #1312; degraded-allow read-check for #1318 at
+`rustfs/src/app/object_usecase.rs`; full fanout for #1314). Fail-closed is
+**only** an explicit administrator strict mode. Defaulting to fail-closed is
+forbidden — it makes writes unavailable for the whole rolling-upgrade window.
+
+## Capability negotiation
+
+Generation enforcement is a **cluster-level handshake**, not a per-request
+probe:
+
+- A node advertises a `generation` capability once it can (a) mint quorum-durable
+  epochs, (b) compare epochs at all three disk-write points, and (c) verify the
+  body-digest-bound RPC signature.
+- The authoritative writer enables hard enforcement for an object only when
+  **all** target disks in the set advertise the capability. Any missing
+  advertisement pins that commit to the mixed-version fallback above.
+- The capability is surfaced through the existing runtime capability contract
+  surface (see [runtime-capability-contracts.md](runtime-capability-contracts.md)),
+  so consumers read one negotiated flag rather than each re-deriving support.
+- Enforcement tracks the current membership rather than latching: it turns on
+  for a set only while every disk in that set advertises `generation`, and a
+  single old node rejoining drops the affected sets back to the mixed-version
+  fallback rather than failing closed. It never regresses the on-disk epoch —
+  falling back stops *comparing* new epochs, it does not lower any epoch already
+  persisted.
+
+## Implementation order
+
+1. **#1312 first.** It defines the epoch, its persistence, the three fence
+   points, the RPC signature binding, and the encoding location. Everything
+   downstream depends on its epoch existing.
+2. **#1313** (read lease) reuses the #1312 epoch as the lease generation and
+   must land before or alongside #1323.
+3. **#1323** (old-dir GC) depends on #1313 leases being present and
+   cross-node-visible; its "no lease references old_dir" check has nothing to
+   query otherwise.
+4. **#1318** (quota reservation) and **#1314** (prepared pool read) bind the
+   epoch independently; both gate on the same capability handshake.
+
+## Open design decisions (pin before implementation)
+
+This document fixes the transport, encoding, proto, and gate constraints, but it is not yet a complete implementable algorithm. The following must be decided and written down before any of the five consumers is coded (per the #1307 maintainer re-review, issuecomment-4992956256):
+
+- **Epoch type and total order.** The concrete token type and its total-order rule — a term+counter tuple, its persistence, and overflow behavior. Whether monotonicity is global or strictly per-object.
+- **Never-regress on lock-service restart / minority recovery.** The epoch source must survive a lock-service restart or minority-quorum recovery without ever handing out an epoch lower than one already persisted on disk (an in-memory counter reset to zero is a fencing inversion). This is the same requirement as "Monotonicity persistence semantics" above, elevated to a hard, tested acceptance.
+- **Complete xl.meta-writer coverage.** Every code path that writes xl.meta (commit rename, rollback delete/metadata restore, old-dir cleanup, heal, transition) must be enumerated and shown to compare or carry the epoch. A single unfenced writer voids the guarantee.
+- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure rollback at `io_primitives.rs:2646-2691` restores a metadata backup, not just a per-writer tmp delete, so a late rollback by writer A can overwrite writer B's committed xl.meta. Rollback must execute only when `stored_epoch == failed_writer_epoch`; a higher stored epoch must abort the rollback. Task panic / cancel / timeout at `io_primitives.rs:2602-2605` must be reaped into the coordinator's state machine, never bubble out via `?` and skip convergence.
+- **Sidecar is excluded unless proven atomic.** An epoch sidecar outside `xl.meta` is only admissible if it commits at the same atomic/CAS point as `xl.meta` with a defined recovery; otherwise it opens a crash gap and must be rejected in favor of the version-internal metadata map. The earlier "metadata map or sidecar" phrasing does not treat the two as equally safe.
+- **Read-lease and GC crash recovery.** Lease registry location (local vs cross-node), TTL reclamation, and crash recovery for both the lease holder and the GC executor.
+- **Quota reserve → commit → settle idempotency.** The cross-stage reconcile / idempotency story for #1318, including owner-crash reconciliation, so a reservation is neither lost nor double-counted.
+- **PreparedPoolRead is pool-local only.** A #1314 bundle's generation validates freshness only within the pool that produced it. It cannot order commits across different pools unless a cross-pool common authority exists; absent that, the multi-pool wait cannot be short-circuited.
+- **Hot-path cost is a blocking metric.** If per-PUT fencing grant, quota reserve, or cleanup journal adds a consensus write / fsync / centralized serialization point, it must be measured under 4KiB and high-concurrency hot-key / hot-bucket A/B as a blocking gate, not accepted by default.
+
+## Acceptance for this contract
+
+- #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this unified
+  generation and no longer define their own token.
+- The five constraints — transport signature, encoding, proto presence,
+  mixed-version gate direction, and capability negotiation — are pinned here
+  once; each implementation sub-issue follows them rather than re-deciding.

docs/operations/drive-timeout-tuning.md

@@ -0,0 +1,148 @@
+# Drive Timeout Tuning
+
+This document describes the per-operation drive timeout knobs and the
+drive-timeout profile. It is written for operators running RustFS on slow or
+high-latency storage (HDD-class disks, network block devices, throttled
+containers) who see `ListObjects`/`ListObjectsV2` requests fail on large
+prefixes, or who want to widen drive liveness budgets before a walk on a
+healthy-but-slow disk is treated as a stall.
+
+If you are debugging a listing that returns *fewer* keys than the bucket holds,
+read [Listing truncation and the walk stall budget](#listing-truncation-and-the-walk-stall-budget)
+first — that is the case this document exists for.
+
+## Background: what these timeouts bound
+
+Every foreground drive operation carries a liveness budget so a hung disk fails
+fast instead of parking the request forever. The budget answers "is the drive
+still *answering*", not "how much total work is there" — a healthy disk that is
+merely busy keeps making progress and is not timed out.
+
+The most important of these for listings is the **walk stall budget**. A
+directory walk (the filesystem enumeration behind every `ListObjects`) bounds
+each individual filesystem call — `readdir`, `stat`, `xl.meta` read — by the
+stall budget. A call that stops answering for longer than the budget fails with
+a drive timeout; time the walk spends blocked on a slow *consumer* (a client
+draining the listing slowly) does not count against it.
+
+## Configuration and precedence
+
+Each knob is resolved in this order, highest priority first:
+
+1. Its explicit per-operation environment variable
+   (`RUSTFS_DRIVE_*_TIMEOUT_SECS`).
+2. The legacy global fallback `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION` (deprecated;
+   applies to every per-operation knob that has no explicit override).
+3. The drive-timeout profile default (see below).
+4. The built-in default.
+
+Values are whole seconds. Changes take effect on process restart.
+
+### Drive-timeout profile
+
+`RUSTFS_DRIVE_TIMEOUT_PROFILE` selects a preset that raises several defaults at
+once, so slow-storage deployments do not have to set each knob individually:
+
+| Value | Effect |
+|---|---|
+| `default` | Built-in defaults (see the table below). |
+| `high_latency` | Raises the default for every profile-aware knob to **60s**. |
+
+Explicit per-operation overrides always win over the profile, so you can select
+`high_latency` and still pin one knob to a specific value.
+
+## Knobs
+
+| Environment variable | Default | `high_latency` default | Bounds |
+|---|---:|---:|---|
+| `RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS` | `5` | `60` | Max time a single walk filesystem call may go without answering during a listing walk. **The knob for listing failures on large prefixes.** |
+| `RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS` | `5` | `60` | Total wall-clock timeout for a `walk_dir`. Retained for non-foreground callers; the foreground listing path no longer uses it (see below). |
+| `RUSTFS_DRIVE_LIST_DIR_TIMEOUT_SECS` | `5` | `60` | Timeout for a standalone `list_dir` metadata listing. |
+| `RUSTFS_DRIVE_METADATA_TIMEOUT_SECS` | `5` | `60` | Timeout for metadata reads such as `read_metadata`. |
+| `RUSTFS_DRIVE_DISK_INFO_TIMEOUT_SECS` | `5` | `60` | Timeout for `disk_info()` calls. |
+| `RUSTFS_OBJECT_DISK_READ_TIMEOUT` | `10` | `60` | Per-read stall budget while streaming an object body from disk. |
+| `RUSTFS_DRIVE_MAX_TIMEOUT_DURATION` | `30` | — | Deprecated global fallback for every per-operation knob without an explicit override. Prefer the per-operation knobs. |
+
+The health-transition and probe knobs (`RUSTFS_DRIVE_TIMEOUT_HEALTH_ACTION`,
+`RUSTFS_DRIVE_ACTIVE_CHECK_*`, `RUSTFS_DRIVE_SUSPECT_FAILURE_THRESHOLD`, and the
+returning/offline classification knobs) govern how a timeout maps to drive
+health state. They are out of scope here; see `crates/config/src/constants/drive.rs`
+for the full list and defaults.
+
+## Listing truncation and the walk stall budget
+
+### Symptom
+
+`ListObjects`/`ListObjectsV2` on a large prefix either:
+
+- returns `500 InternalError` with `Io error: timeout`; or
+- (on older builds) returns HTTP 200 with `IsTruncated=false` after fewer keys
+  than the bucket actually holds — a **silent** truncation that S3 clients
+  (`mc`, minio-go, SDK pagination loops) cannot detect, because
+  `IsTruncated=false` is the protocol's only end-of-listing signal.
+
+Every "missing" object remains readable by exact key via `GetObject` /
+`StatObject`; only the listing is affected.
+
+### Why it happens
+
+A listing walk is bounded by the stall budget. Because a whole-directory
+enumeration (`list_dir` reading every immediate child in one pass) is bounded by
+that budget **as a single unit**, a very wide *flat* directory — one prefix
+holding hundreds of thousands or millions of immediate children — can make a
+single `readdir` exceed the budget on a perfectly healthy disk, especially on
+HDD-class or throttled storage. That trips a drive timeout, which the listing
+path escalates and surfaces to the client.
+
+### The silent variant is fixed; the loud 500 is tuned away
+
+As of the walk-stall rework (merged to `main`, first released in **1.0.0-beta.9**):
+
+- **The silent variant is eliminated.** A walk that dies mid-stream can no
+  longer be consumed as a clean end-of-listing. Once a walk has streamed any
+  entries and then stalls, the failure is recorded as a hard drive timeout and
+  escalated on that erasure set, so the client always sees an error — never a
+  well-formed short page. This is locked by the
+  `list_path_raw_returns_timeout_when_producer_fails_after_partial_entry`
+  regression test in `crates/ecstore/src/cache_value/metacache_set.rs`.
+- **The remaining 500 is an operator-tunable, not a data-integrity bug.** A
+  genuinely wide flat directory can still exhaust the default 5s stall budget on
+  slow storage and fail the listing loudly. The supported mitigation is to widen
+  the budget.
+
+### Mitigation
+
+Raise the walk stall budget, or select the high-latency profile:
+
+```bash
+# widen just the listing walk budget
+-e RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS=60
+
+# or raise every profile-aware drive default at once
+-e RUSTFS_DRIVE_TIMEOUT_PROFILE=high_latency
+```
+
+> Note: on releases at or before `1.0.0-beta.8`, the foreground listing path was
+> bounded by the *total* wall-clock knob `RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS`
+> instead of the stall budget. If you cannot upgrade, raise that knob — but
+> upgrading to `1.0.0-beta.9` or later is strongly preferred, because only the
+> newer builds convert the *silent* truncation into a detectable error.
+
+The most durable fix for pathologically wide directories is to shard keys under
+additional prefix levels so no single directory holds an enormous flat child
+set; the stall budget then never has to bound one giant `readdir`.
+
+### Diagnosing
+
+Server logs at the failure show the walk timeout escalating through the listing
+pipeline:
+
+```
+WARN  Metacache reader peek timed out            state=peek_timed_out  drive=<endpoint>
+ERROR Metacache listing quorum failed            state=quorum_failed
+```
+
+The `rustfs_list_path_raw_stall_total` counter (labelled by `drive`) increments
+on every walk stall, and gives you a per-drive signal that a budget is being
+hit before any client-visible failure. Watch it after tuning to confirm the
+stalls have stopped.

docs/testing/e2e-suite-inventory.md

@@ -3,7 +3,7 @@
 > Authoritative per-module test counts for the `e2e_test` crate (backlog#1149
 > ci-4), generated from `cargo nextest list -p e2e_test`. Regenerate with:
 > ```bash
-> cargo nextest list -p e2e_test --message-format oneline | awk '{split($2,a,"::"); print a[1]}' | sort | uniq -c
+> cargo nextest list -p e2e_test --message-format json | jq -r '.["rust-suites"][]?.testcases | to_entries[] | select(.value.ignored == false) | .key | split("::")[0]' | sort | uniq -c
 > ```
 > Modules marked ✅ are in the PR smoke profile `e2e-smoke`
 > (`.config/nextest.toml`); admission criteria: `crates/e2e_test/README.md`.
@@ -39,6 +39,7 @@
 | delete_object_no_content_length_test | 1 |  |
 | delete_objects_versioning_test | 2 | ✅ |
 | existing_object_tag_policy_test | 4 |  |
+| fake_s3_target | 4 | ✅ |
 | get_codec_streaming_compat_test | 1 |  |
 | head_object_consistency_test | 1 | ✅ |
 | head_object_range_test | 1 | ✅ |
@@ -74,4 +75,4 @@
 | tls_hot_reload_test | 1 | ✅ |
 | version_id_regression_test | 10 | ✅ |
 
-**Total listed: 435 tests across 56 modules · PR smoke subset: 108 tests / 26 modules** (24 full modules + 4 `reliant` tests + 20 of `replication_extension_test`) **· nightly `e2e-repl-nightly`: 27 tests** · generated 2026-07-15.
+**Total listed: 439 tests across 57 modules · PR smoke subset: 112 tests / 27 modules** (25 full modules + 4 `reliant` tests + 20 of `replication_extension_test`) **· nightly `e2e-repl-nightly`: 27 tests** · generated 2026-07-16.

flake.nix

@@ -60,7 +60,7 @@
         {
           default = rustPlatform.buildRustPackage {
             pname = "rustfs";
-            version = "1.0.0-beta.10-preview.1";
+            version = "1.0.0-beta.10";
 
             src = ./.;
 

helm/README.md

@@ -1,13 +1,51 @@
 # RustFS Helm Mode
 
-RustFS helm chart supports **standalone and distributed mode**. For standalone mode, there is only one pod and one pvc; for distributed mode, there are two styles, 4 pods and 16 pvcs(each pod has 4 pvcs), 16 pods and 16 pvcs(each pod has 1 pvc). You should decide which mode and style suits for your situation. You can specify the parameters `mode` and `replicaCount` to install different mode and style.
+RustFS helm chart supports **standalone** and **distributed** mode.
 
-- **For standalone mode**: Only one pod and one pvc acts as single node single disk; Specify parameters `mode.standalone.enabled="true",mode.distributed.enabled="false"` to install.
-- **For distributed mode**(**default**): Multiple pods and multiple pvcs, acts as multiple nodes multiple disks, there are two styles:
-    - 4 pods and each pods has 4 pvcs(**default**)
-    - 16 pods and each pods has 1 pvc: Specify parameters `replicaCount` with `--set replicaCount="16"` to install.
+- **Standalone mode**: one pod with one PVC (single node, single disk).
+- **Distributed mode** (**default**): multiple pods with multiple PVCs (multiple nodes, multiple disks).
 
-**NOTE**: Please make sure which mode suits for you situation and specify the right parameter to install rustfs on kubernetes.
+## Distributed topology
+
+The distributed topology is defined by two parameters:
+
+- `replicaCount` — number of pods (nodes) in the StatefulSet.
+- `drivesPerNode` — number of data PVCs mounted on each pod.
+
+Total drives in the cluster = `replicaCount * drivesPerNode`.
+
+When `drivesPerNode` is left unset, the chart automatically infers a
+backward-compatible value from each pool's replica count (with pools
+disabled there is a single pool driven by the top-level `replicaCount`):
+
+| `replicaCount` | Inferred `drivesPerNode` | Legacy equivalent |
+|----------------|--------------------------|-------------------|
+| 4              | 4                        | old default 4×4   |
+| anything else  | 1                        | old 16×1, etc.    |
+
+You can override the inference by setting `drivesPerNode` explicitly, e.g.
+`--set drivesPerNode=2` for an 8×2 cluster.
+
+**IMPORTANT**: Kubernetes does **not** allow changes to
+`volumeClaimTemplates` in an existing StatefulSet. If you want to change
+`drivesPerNode` after installation you must delete the StatefulSet
+(with `--cascade=orphan` to keep pods and PVCs) and recreate it, or perform a
+full reinstall.
+
+---
+
+## Upgrade notes
+
+Upgrading from chart versions that did **not** have `drivesPerNode` is safe
+without manual intervention:
+
+- Existing 4×4 deployments (default `replicaCount=4`) continue to receive 4
+  drives per node because the chart infers `drivesPerNode=4`.
+- Existing 16×1 deployments (`replicaCount=16`) continue to receive 1 drive
+  per node because the chart infers `drivesPerNode=1`.
+
+If you previously set `replicaCount=16` and now want a different topology,
+set both `replicaCount` and `drivesPerNode` explicitly.
 
 ---
 
@@ -134,7 +172,7 @@ uer. `ClusterIssuer` or `Issuer`. |
 | pdb.minAvailable | string | `""` |  |
 | podAnnotations | object | `{}` |  |
 | pools.enabled | bool | `false` | Enable multiple server pools (capacity expansion, distributed mode only). |
-| pools.list | list | `[]` | One entry per pool; entries may set `replicaCount` (4 or 16) and `storageclass`, omitted fields inherit top-level values. Append-only. |
+| pools.list | list | `[]` | One entry per pool; entries may set `replicaCount` (>= 2) and `storageclass`, omitted fields inherit top-level values. Append-only. |
 | podLabels | object | `{}` |  |
 | podSecurityContext.fsGroup | int | `10001` |  |
 | podSecurityContext.runAsGroup | int | `10001` |  |
@@ -146,7 +184,8 @@ uer. `ClusterIssuer` or `Issuer`. |
 | readinessProbe.periodSeconds | int | `5` |  |
 | readinessProbe.successThreshold | int | `1` |  |
 | readinessProbe.timeoutSeconds | int | `3` |  |
-| replicaCount | int | `4` | Number of cluster nodes. |
+| replicaCount | int | `4` | Number of cluster nodes. Distributed mode requires >= 2. |
+| drivesPerNode | int | `null` | Number of data PVCs per pod. Inferred from replicaCount when unset (see Distributed topology above). |
 | resources.limits.cpu | string | `"200m"` |  |
 | resources.limits.memory | string | `"512Mi"` |  |
 | resources.requests.cpu | string | `"100m"` |  |
@@ -237,12 +276,12 @@ pools:
   list:
     - {}                  # pool 0: inherits top-level values and keeps the
                           # existing StatefulSet/pod/PVC names and data
-    - replicaCount: 4     # pool 1: new capacity (4 or 16)
+    - replicaCount: 4     # pool 1: new capacity
       storageclass:
         dataStorageSize: 10Gi
 ```
 
-Each entry may set `replicaCount` (4 or 16) and/or a `storageclass` block;
+Each entry may set `replicaCount` (>= 2) and/or a `storageclass` block;
 omitted fields inherit the top-level values. Additional pools render as
 `<fullname>-pool<N>` StatefulSets; all pools share the headless service,
 the main service, the configuration and the credentials.

helm/rustfs/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: rustfs
 description: RustFS helm chart to deploy RustFS on kubernetes cluster.
 type: application
-version: "0.10.0-preview.1"
-appVersion: "1.0.0-beta.10-preview.1"
+version: "0.10.0"
+appVersion: "1.0.0-beta.10"
 home: https://rustfs.com
 icon: https://media.sys.truenas.net/apps/rustfs/icons/icon.svg
 maintainers:

helm/rustfs/templates/_helpers.tpl

@@ -42,6 +42,19 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
 app.kubernetes.io/managed-by: {{ .Release.Service }}
 {{- end }}
 
+{{/*
+Render extra labels for the main Service resource.
+Merges (in order of increasing precedence):
+  - commonLabels
+  - service.labels
+*/}}
+{{- define "rustfs.serviceLabels" -}}
+{{- $labels := mergeOverwrite (dict) (default (dict) .Values.commonLabels) (default (dict) .Values.service.labels) }}
+{{- if $labels }}
+{{- toYaml $labels }}
+{{- end }}
+{{- end }}
+
 {{/*
 Selector labels
 */}}
@@ -185,6 +198,30 @@ Expects a dict with keys "root" (the chart root context) and "index".
 {{- end -}}
 {{- end }}
 
+{{/*
+Return the number of data drives (PVCs) per pod for a pool.
+When .Values.drivesPerNode is set it applies to every pool. When unset the
+value is inferred per pool from its replica count so that rendered output
+stays identical to the pre-drivesPerNode chart: 4 replicas keep the legacy
+4-drive layout, everything else (16x1 and any new topology) gets one drive.
+Expects a dict with keys "root" (the chart root context) and "replicas".
+*/}}
+{{- define "rustfs.poolDrives" -}}
+{{- if kindIs "invalid" .root.Values.drivesPerNode -}}
+{{- if eq (int .replicas) 4 -}}
+4
+{{- else -}}
+1
+{{- end -}}
+{{- else -}}
+{{- $d := int .root.Values.drivesPerNode -}}
+{{- if lt $d 1 -}}
+{{- fail "drivesPerNode must be >= 1 when set" -}}
+{{- end -}}
+{{- $d -}}
+{{- end -}}
+{{- end }}
+
 {{/*
 Return the normalized list of server pools as JSON.
 With pools disabled this is a single pool built from the top-level
@@ -206,14 +243,16 @@ so entries must never be removed or reordered.
 {{- range $i, $p := .Values.pools.list -}}
 {{- $p = default (dict) $p -}}
 {{- $replicas := int (default $.Values.replicaCount $p.replicaCount) -}}
-{{- if and (ne $replicas 4) (ne $replicas 16) -}}
-{{- fail (printf "pools.list[%d].replicaCount must be 4 or 16, got %d" $i $replicas) -}}
+{{- if lt $replicas 2 -}}
+{{- fail (printf "pools.list[%d].replicaCount must be >= 2, got %d" $i $replicas) -}}
 {{- end -}}
 {{- $sc := mergeOverwrite (deepCopy $.Values.storageclass) (default (dict) $p.storageclass) -}}
-{{- $pools = append $pools (dict "index" $i "fullname" (include "rustfs.poolFullname" (dict "root" $ "index" $i)) "replicaCount" $replicas "storageclass" $sc) -}}
+{{- $drives := int (include "rustfs.poolDrives" (dict "root" $ "replicas" $replicas)) -}}
+{{- $pools = append $pools (dict "index" $i "fullname" (include "rustfs.poolFullname" (dict "root" $ "index" $i)) "replicaCount" $replicas "drives" $drives "storageclass" $sc) -}}
 {{- end -}}
 {{- else -}}
-{{- $pools = append $pools (dict "index" 0 "fullname" (include "rustfs.fullname" .) "replicaCount" (int .Values.replicaCount) "storageclass" .Values.storageclass) -}}
+{{- $drives := int (include "rustfs.poolDrives" (dict "root" $ "replicas" (int .Values.replicaCount))) -}}
+{{- $pools = append $pools (dict "index" 0 "fullname" (include "rustfs.fullname" .) "replicaCount" (int .Values.replicaCount) "drives" $drives "storageclass" .Values.storageclass) -}}
 {{- end -}}
 {{- toJson $pools -}}
 {{- end }}
@@ -235,9 +274,10 @@ RUSTFS_VOLUMES on spaces, one pool per expression).
 {{- $exprs := list -}}
 {{- range $pool := include "rustfs.pools" . | fromJsonArray -}}
 {{- $n := int $pool.replicaCount -}}
-{{- if eq $n 4 -}}
-{{- $exprs = append $exprs (printf "%s://%s-{0...%d}.%s.%s.svc.%s:%d/data/rustfs{0...%d}" $protocol $pool.fullname (sub $n 1) $headless $ns $domain $port (sub $n 1)) -}}
-{{- else if eq $n 16 -}}
+{{- $d := int $pool.drives -}}
+{{- if gt $d 1 -}}
+{{- $exprs = append $exprs (printf "%s://%s-{0...%d}.%s.%s.svc.%s:%d/data/rustfs{0...%d}" $protocol $pool.fullname (sub $n 1) $headless $ns $domain $port (sub $d 1)) -}}
+{{- else -}}
 {{- $exprs = append $exprs (printf "%s://%s-{0...%d}.%s.%s.svc.%s:%d/data" $protocol $pool.fullname (sub $n 1) $headless $ns $domain $port) -}}
 {{- end -}}
 {{- end -}}

helm/rustfs/templates/configmap.yaml

@@ -19,6 +19,9 @@ data:
   {{- if .domains }}
   RUSTFS_SERVER_DOMAINS: {{ include "rustfs.serverDomains" $ | quote }}
   {{- end }}
+  {{- if .ec.storage_class_standard }}
+  RUSTFS_STORAGE_CLASS_STANDARD: {{ .ec.storage_class_standard | quote }}
+  {{- end }}
   {{- with .log_rotation }}
     {{- if .size }}
   RUSTFS_OBS_LOG_ROTATION_SIZE_MB: {{ .size | quote }}

helm/rustfs/templates/deployment.yaml

@@ -131,7 +131,7 @@ spec:
               subPath: ca.crt
             - name: client-cert
               mountPath: /opt/tls/client_cert.pem
-              subPath: client_cert.pem 
+              subPath: client_cert.pem
             - name: client-cert
               mountPath: /opt/tls/client_key.pem
               subPath: client_key.pem

helm/rustfs/templates/service.yaml

@@ -39,8 +39,8 @@ metadata:
   {{- end }}
   labels:
     {{- include "rustfs.labels" . | nindent 4 }}
-    {{- with .Values.commonLabels }}
-    {{- toYaml . | nindent 4 }}
+    {{- with (include "rustfs.serviceLabels" .) }}
+    {{- . | nindent 4 }}
     {{- end }}
 spec:
   {{- if eq $serviceType "ClusterIP" }}
@@ -64,6 +64,10 @@ spec:
     {{- toYaml . | nindent 4 }}
   {{- end }}
   {{- end }}
+  {{- with .Values.service.externalIPs }}
+  externalIPs:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   ports:
     - name: endpoint
       port: {{ .Values.service.endpoint.port }}

helm/rustfs/templates/statefulset.yaml

@@ -1,9 +1,13 @@
 {{- $logDir := .Values.config.rustfs.obs_log_directory }}
 {{- $logDirEnabled := ne $logDir "" }}
 {{- $poolsEnabled := and .Values.pools .Values.pools.enabled }}
+{{- if and .Values.mode.distributed.enabled (le (int .Values.replicaCount) 1) -}}
+{{- fail "Distributed mode requires replicaCount >= 2" -}}
+{{- end -}}
 
 {{- if .Values.mode.distributed.enabled }}
 {{- range $pool := include "rustfs.pools" . | fromJsonArray }}
+{{- $drivesPerNode := int $pool.drives }}
 ---
 apiVersion: apps/v1
 kind: StatefulSet
@@ -117,17 +121,17 @@ spec:
           securityContext:
             {{- toYaml $.Values.containerSecurityContext | nindent 12 }}
           env:
-            - name: REPLICA_COUNT
-              value: {{ $pool.replicaCount | quote }}
+            - name: DRIVES_PER_NODE
+              value: {{ $drivesPerNode | quote }}
           command:
             - sh
             - -c
             - |
-              if [ "$REPLICA_COUNT" -eq 4 ]; then
-                for i in $(seq 0 $(($REPLICA_COUNT - 1))); do
+              if [ "$DRIVES_PER_NODE" -gt 1 ]; then
+                for i in $(seq 0 $(($DRIVES_PER_NODE - 1))); do
                   mkdir -p /data/rustfs$i
-                done;
-              elif [ "$REPLICA_COUNT" -eq 16 ]; then
+                done
+              else
                 mkdir -p /data
               fi
               {{- if $logDirEnabled }}
@@ -135,12 +139,12 @@ spec:
               chmod 755 /mnt/rustfs/logs
               {{- end }}
           volumeMounts:
-            {{- if eq (int $pool.replicaCount) 4 }}
-            {{- range $i := until (int $pool.replicaCount) }}
+            {{- if gt $drivesPerNode 1 }}
+            {{- range $i := until $drivesPerNode }}
             - name: data-rustfs-{{ $i }}
               mountPath: /data/rustfs{{ $i }}
             {{- end }}
-            {{- else if eq (int $pool.replicaCount) 16 }}
+            {{- else }}
             - name: data
               mountPath: /data
             {{- end }}
@@ -201,12 +205,12 @@ spec:
               mountPath: {{ $logDir }}
               subPath: logs
             {{- end }}
-            {{- if eq (int $pool.replicaCount) 4 }}
-            {{- range $i := until (int $pool.replicaCount) }}
+            {{- if gt $drivesPerNode 1 }}
+            {{- range $i := until $drivesPerNode }}
             - name: data-rustfs-{{ $i }}
               mountPath: /data/rustfs{{ $i }}
             {{- end }}
-            {{- else if eq (int $pool.replicaCount) 16 }}
+            {{- else }}
             - name: data
               mountPath: /data
             {{- end }}
@@ -246,8 +250,11 @@ spec:
         name: logs
         labels:
           {{- toYaml $.Values.commonLabels | nindent 10 }}
+        {{- $logAnn := (default (dict) $pool.storageclass.pvcAnnotations).logs | default (dict) -}}
+        {{- if gt (len $logAnn) 0 }}
         annotations:
-          {{- toYaml $pool.storageclass.pvcAnnotations.logs | nindent 10 }}
+          {{- toYaml $logAnn | nindent 10 }}
+        {{- end }}
       spec:
         accessModes: ["ReadWriteOnce"]
         storageClassName: {{ $pool.storageclass.name }}
@@ -255,16 +262,19 @@ spec:
           requests:
             storage: {{ $pool.storageclass.logStorageSize }}
     {{- end }}
-    {{- if eq (int $pool.replicaCount) 4 }}
-    {{- range $i := until (int $pool.replicaCount) }}
+    {{- if gt $drivesPerNode 1 }}
+    {{- range $i := until $drivesPerNode }}
     - apiVersion: v1
       kind: PersistentVolumeClaim
       metadata:
         name: data-rustfs-{{ $i }}
         labels:
           {{- toYaml $.Values.commonLabels | nindent 10 }}
+        {{- $dataAnn := (default (dict) $pool.storageclass.pvcAnnotations).data | default (dict) -}}
+        {{- if gt (len $dataAnn) 0 }}
         annotations:
-          {{- toYaml $pool.storageclass.pvcAnnotations.data | nindent 10 }}
+          {{- toYaml $dataAnn | nindent 10 }}
+        {{- end }}
       spec:
         accessModes: ["ReadWriteOnce"]
         storageClassName: {{ $pool.storageclass.name }}
@@ -272,15 +282,18 @@ spec:
           requests:
             storage: {{ $pool.storageclass.dataStorageSize }}
     {{- end }}
-    {{- else if eq (int $pool.replicaCount) 16 }}
+    {{- else }}
     - apiVersion: v1
       kind: PersistentVolumeClaim
       metadata:
         name: data
         labels:
           {{- toYaml $.Values.commonLabels | nindent 10 }}
+        {{- $dataAnn := (default (dict) $pool.storageclass.pvcAnnotations).data | default (dict) -}}
+        {{- if gt (len $dataAnn) 0 }}
         annotations:
-          {{- toYaml $pool.storageclass.pvcAnnotations.data | nindent 10 }}
+          {{- toYaml $dataAnn | nindent 10 }}
+        {{- end }}
       spec:
         accessModes: ["ReadWriteOnce"]
         storageClassName: {{ $pool.storageclass.name }}

helm/rustfs/values.yaml

@@ -2,9 +2,27 @@
 # This is a YAML-formatted file.
 # Declare variables to be passed into your templates.
 
-# This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
+# Number of StatefulSet pods (nodes in the distributed cluster).
+# In distributed mode this must be >= 2 (RustFS requires at least two nodes
+# for an erasure-coded cluster).
 replicaCount: 4
 
+# Number of data drives (PVCs) per pod. Combined with replicaCount this
+# determines the total drive count seen by rustfs.
+# - drivesPerNode > 1: each pod gets N volumes mounted at /data/rustfs0.../data/rustfsN-1
+# - drivesPerNode = 1: each pod gets a single volume mounted at /data
+#
+# When left unset (null) the chart infers a backward-compatible value per
+# pool from that pool's replica count:
+#   replicaCount = 4  -> drivesPerNode = 4  (legacy default)
+#   replicaCount != 4 -> drivesPerNode = 1  (legacy 16x1 and any other topology)
+#
+# WARNING: changing drivesPerNode on an existing StatefulSet is not allowed
+# because Kubernetes forbids updates to volumeClaimTemplates. To change the
+# drive count you must delete the StatefulSet (with --cascade=orphan to keep
+# pods/PVCs) and recreate it, or perform a full reinstall.
+drivesPerNode: null
+
 # Kubernetes cluster DNS domain used to build in-cluster FQDNs for
 # RUSTFS_VOLUMES (distributed mode) and mTLS server certificate SANs.
 # Override this only if your cluster does not use the default `cluster.local`
@@ -60,7 +78,7 @@ mode:
 # With pools.enabled=true, one StatefulSet is rendered per entry of
 # pools.list and RUSTFS_VOLUMES contains every pool's volume expression
 # (space separated), which is how the server discovers multiple pools.
-# Each entry may set replicaCount (4 or 16) and/or a storageclass block;
+# Each entry may set replicaCount (>= 2) and/or a storageclass block;
 # omitted fields inherit the top-level values.
 #
 # IMPORTANT:
@@ -99,23 +117,31 @@ config:
     # Examples
     # volumes: "/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3"
     # volumes: "http://rustfs-{0...3}.rustfs-headless:9000/data/rustfs{0...3}"
+    # NOTE: When overriding volumes manually, the mount paths and drive ranges
+    # must match the number of pods (replicaCount) and drives per pod
+    # (drivesPerNode). Mismatched values will cause broken volume discovery.
     volumes: ""
     address: ":9000"
     console_enable: "true"
     console_address: ":9001"
-    log_level: "info"
     region: "us-east-1"
-    obs_log_directory: "/logs" # Set to "" to disable log PVCs and mounts.
-    obs_environment: "development"
     # Optionally enable support for virtual-hosted-style requests.
     # See more information: https://docs.rustfs.com/integration/virtual.html
     domains: "" # e.g. "example.com"
     ec:
-      storage_class_standard: "EC:4" # Storage class for standard storage class in erasure coding mode, default is "STANDARD"
-    log_rotation: # Specify log rotation settings
-      # size: 100 # Default value: 100 MB
-      # time: hour # Default value: hour, eg: day,hour,minute,second
-      # keep_files: 30 # number of rotated log files to keep
+      # Erasure-coding parity for the STANDARD storage class, e.g. "EC:4".
+      # Left empty by default: the server then picks a parity suited to the
+      # drive count. If you set it, it must be valid for the total drive
+      # count (total drives = replicaCount * drivesPerNode):
+      #
+      # | replicaCount | drivesPerNode | Total drives | Valid parity |
+      # | ------------ | ------------- | ------------ | ------------ |
+      # | 4            | 1             | 4            | 0-2          |
+      # | 4            | 2             | 8            | 0-4          |
+      # | 4            | 4             | 16           | 0-8          |
+      # | 8            | 1             | 8            | 0-4          |
+      # | 16           | 1             | 16           | 0-8          |
+      storage_class_standard: ""
     scanner:
       # Scanner speed preset: fastest|fast|default|slow|slowest
       speed: ""
@@ -155,6 +181,16 @@ config:
     # - default: keep current timeout defaults.
     # - high_latency: use higher timeout defaults for scanner-sensitive operations.
     timeout_profile: ""
+    log_level: "info"
+    log_rotation: # Specify log rotation settings
+      # size: 100 # Default value: 100 MB
+      # time: hour # Default value: hour, eg: day,hour,minute,second
+      # keep_files: 30 # number of rotated log files to keep
+    ## If obs_log_directory is not empty, a separate volume for logs will be created and the logs will NOT be sent to stdout. If not set, will default to default chart value "/logs"
+    obs_log_directory: "/logs"
+    ## Supported values include `production`, `development`, `test`, and `staging`;
+    ## refer to the RustFS configuration documentation/source for the authoritative list.
+    obs_environment: "development"
     obs_endpoint:
       enabled: false # If true, rustfs will export metrics, traces, logs and profiling data to the specified OTLP endpoints. If false, the individual settings for metrics, traces, logs and profiling endpoints will be ignored and all data will not be exported.
       base_endpoint: "" #Root OTLP/HTTP endpoint, e.g. http://otel-collector:4318
@@ -175,7 +211,7 @@ config:
       enabled: false
       type: "vault" # Only Support vault currently.
       vault:
-        vault_backend: "" # Only support vault kv2 and vault transit. 
+        vault_backend: "" # Only support vault kv2 and vault transit.
         vault_address: ""
         vault_token: ""
         vault_mount_path: ""
@@ -235,6 +271,7 @@ priorityClassName: ""
 
 service:
   annotations: {}
+  labels: {} # Additional labels
   headlessAnnotations: {} # Applied to the headless Service when mode.distributed.enabled=true
   traefikAnnotations: # Applied to the Service when mode.distributed.enabled=true and ingress.className=traefik
     traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
@@ -247,6 +284,7 @@ service:
   loadBalancerIP: ""           # Request a specific IP from the load balancer (e.g. for Cilium LB-IPAM or MetalLB)
   loadBalancerClass: ""        # Specify a load balancer implementation (e.g. "io.cilium/bgp", "metallb")
   loadBalancerSourceRanges: [] # Restrict client IPs (e.g. ["10.0.0.0/8"])
+  externalIPs: []             # Expose service via specific external IPs (e.g. ["203.0.113.1"])
   endpoint:
     port: 9000
     nodePort: 32000
@@ -297,7 +335,7 @@ ingress:
 gatewayApi:
   enabled: false
   gatewayClass: traefik # Only support for traefik and contour gatewayClass at the moment.
-  listeners: # Specify which listeners to create on the Gateway. 
+  listeners: # Specify which listeners to create on the Gateway.
     http:
       name: web
       port: 8000
@@ -380,12 +418,9 @@ storageclass:
   name: local-path
   dataStorageSize: 256Mi
   logStorageSize: 256Mi
-  pvcAnnotations: {}
-    #pvcAnnotations:
-    #  data:
-    #    key: value
-    #  logs:
-    #    key: value
+  pvcAnnotations:
+    data: {}
+    logs: {}
 
 pdb:
   create: false

rustfs.spec

@@ -1,9 +1,9 @@
 %global _enable_debug_packages 0
 %global _empty_manifest_terminate_build 0
-%global prerelease beta.10-preview.1
+%global prerelease beta.10
 Name:           rustfs
 Version:        1.0.0
-Release:        beta.10.preview.1
+Release:        beta.10
 Summary:       High-performance distributed object storage for MinIO alternative
 
 License:        Apache-2.0
@@ -58,6 +58,12 @@ install %_builddir/%{name}-%{version}-%{prerelease}/target/%_arch/%_arch-unknown
 %_bindir/rustfs
 
 %changelog
+* Fri Jul 17 2026 overtrue <anzhengchao@gmail.com>
+- Update RPM package to RustFS 1.0.0-beta.10
+
+* Thu Jul 16 2026 houseme <housemecn@gmail.com>
+- Update RPM package to RustFS 1.0.0-beta.10-preview.4
+
 * Thu Jul 16 2026 houseme <housemecn@gmail.com>
 - Update RPM package to RustFS 1.0.0-beta.10-preview.1
 

rustfs/src/admin/handlers/iam_error.rs

@@ -23,6 +23,7 @@ pub(crate) fn iam_error_to_s3_error(err: IamError) -> S3Error {
         | IamError::NoSuchTempAccount(_)
         | IamError::NoSuchGroup(_)
         | IamError::NoSuchPolicy => S3ErrorCode::NoSuchResource,
+        IamError::InvalidAccessKeyLength | IamError::InvalidSecretKeyLength => S3ErrorCode::InvalidArgument,
         _ => S3ErrorCode::InternalError,
     };
 
@@ -54,7 +55,17 @@ mod tests {
     }
 
     #[test]
-    fn non_not_found_iam_errors_remain_internal_errors() {
+    fn credential_length_errors_map_to_invalid_argument() {
+        let errors = [IamError::InvalidAccessKeyLength, IamError::InvalidSecretKeyLength];
+
+        for err in errors {
+            let s3_error = iam_error_to_s3_error(err);
+            assert_eq!(s3_error.code(), &S3ErrorCode::InvalidArgument);
+        }
+    }
+
+    #[test]
+    fn non_validation_iam_errors_remain_internal_errors() {
         let s3_error = iam_error_to_s3_error(IamError::IamSysNotInitialized);
 
         assert_eq!(s3_error.code(), &S3ErrorCode::InternalError);

rustfs/src/admin/handlers/service_account.rs

@@ -397,7 +397,12 @@ impl Operation for AddServiceAccount {
                     error = ?e,
                     "admin service account state"
                 );
-                s3_error!(InternalError, "create service account failed, e: {:?}", e)
+                match e {
+                    rustfs_iam::error::Error::InvalidAccessKeyLength | rustfs_iam::error::Error::InvalidSecretKeyLength => {
+                        iam_error_to_s3_error(e)
+                    }
+                    err => s3_error!(InternalError, "create service account failed, e: {:?}", err),
+                }
             })?;
 
         if let Err(err) = site_replication_iam_change_hook(SRIAMItem {
@@ -1562,6 +1567,24 @@ mod tests {
         assert_eq!(err.message(), Some("service account 'missing' does not exist"));
     }
 
+    #[test]
+    fn add_service_account_maps_iam_create_errors_to_s3_errors() {
+        let src = include_str!("service_account.rs");
+        let create_start = src
+            .find("impl Operation for AddServiceAccount")
+            .expect("AddServiceAccount operation should exist");
+        let create_block = &src[create_start..];
+        let create_end = create_block
+            .find("if let Err(err) = site_replication_iam_change_hook")
+            .expect("site replication hook marker should exist");
+        let create_block = &create_block[..create_end];
+
+        assert!(
+            create_block.contains("iam_error_to_s3_error(e)"),
+            "service-account create validation errors must map to client S3 errors"
+        );
+    }
+
     #[test]
     fn map_temp_account_lookup_error_reports_missing_access_keys_explicitly() {
         let err = map_temp_account_lookup_error(

rustfs/src/admin/handlers/table_catalog.rs

@@ -2034,7 +2034,8 @@ fn table_entry_from_create_table_request(
     let table_uuid = Uuid::new_v4().to_string();
     let warehouse_location = request.location.unwrap_or_else(|| format!("s3://{bucket}/tables/{table_id}"));
     validate_table_location_in_bucket(bucket, &warehouse_location)?;
-    let metadata_location = crate::table_catalog::default_table_metadata_file_path(namespace, &table, "00001.metadata.json");
+    let metadata_location =
+        crate::table_catalog::default_table_metadata_file_path(namespace, &table, &next_metadata_file_name(1, &table_id));
 
     let mut entry = crate::table_catalog::TableEntry {
         version: crate::table_catalog::TABLE_CATALOG_ENTRY_VERSION,
@@ -6890,10 +6891,6 @@ mod tests {
             .await
             .expect("table should be created");
 
-        assert_eq!(
-            response.metadata_location,
-            "s3://warehouse/.rustfs-table/warehouses/default/namespaces/analytics/tables/events/metadata/00001.metadata.json"
-        );
         assert_eq!(response.metadata["format-version"], 2);
         assert_eq!(response.metadata["current-schema-id"], 0);
         assert_eq!(response.metadata["default-spec-id"], 0);
@@ -6907,6 +6904,13 @@ mod tests {
             .await
             .expect("table lookup should succeed")
             .expect("table should exist");
+        assert_eq!(
+            response.metadata_location,
+            format!(
+                "s3://warehouse/.rustfs-table/warehouses/default/namespaces/analytics/tables/events/metadata/00001-{}.metadata.json",
+                entry.table_id
+            )
+        );
         assert_eq!(response.metadata["table-uuid"], entry.table_uuid);
         assert!(
             metadata_backend
@@ -6916,6 +6920,212 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn create_table_response_recreates_dropped_identifier_without_overwriting_retained_metadata() {
+        let store = TestTableCatalogStore::default();
+        let metadata_backend = TestTableCatalogObjectBackend::default();
+        let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace should parse");
+        create_standard_events_table(&store, &metadata_backend, &namespace).await;
+        let first_entry = store
+            .load_table("warehouse", "analytics", "events")
+            .await
+            .expect("first table lookup should succeed")
+            .expect("first table should exist");
+        let retained_initial = metadata_backend
+            .read_object("warehouse", &first_entry.metadata_location)
+            .await
+            .expect("first metadata lookup should succeed")
+            .expect("first metadata should exist");
+
+        let commit_request: RestCommitTableRequest = serde_json::from_value(serde_json::json!({
+            "updates": [
+                {
+                    "action": "set-properties",
+                    "updates": {
+                        "owner": "first-table"
+                    }
+                }
+            ]
+        }))
+        .expect("standard commit request should parse");
+        standard_commit_table_response(&store, &metadata_backend, "warehouse", &namespace, "events", commit_request)
+            .await
+            .expect("first table metadata commit should succeed");
+        let first_committed_entry = store
+            .load_table("warehouse", "analytics", "events")
+            .await
+            .expect("committed table lookup should succeed")
+            .expect("committed table should exist");
+
+        drop_table_in_store(&store, "warehouse", &namespace, "events")
+            .await
+            .expect("first table should drop");
+        drop_namespace_in_store(&store, "warehouse", "analytics")
+            .await
+            .expect("first namespace should drop");
+        create_namespace_response(
+            &store,
+            "warehouse",
+            CreateNamespaceRequest {
+                namespace: vec!["analytics".to_string()],
+                properties: BTreeMap::new(),
+            },
+            true,
+        )
+        .await
+        .expect("namespace should be recreated");
+
+        let create_request: CreateTableRequest = serde_json::from_value(serde_json::json!({
+            "name": "events",
+            "schema": {
+                "type": "struct",
+                "schema-id": 0,
+                "fields": [
+                    {
+                        "id": 1,
+                        "name": "id",
+                        "required": true,
+                        "type": "long"
+                    }
+                ]
+            }
+        }))
+        .expect("recreate table request should parse");
+        let second = create_table_response(&store, &metadata_backend, "warehouse", &namespace, create_request, true)
+            .await
+            .expect("dropped table identifier should be reusable");
+        let second_entry = store
+            .load_table("warehouse", "analytics", "events")
+            .await
+            .expect("recreated table lookup should succeed")
+            .expect("recreated table should exist");
+
+        assert_ne!(first_entry.table_id, second_entry.table_id);
+        assert_ne!(first_entry.table_uuid, second_entry.table_uuid);
+        assert_ne!(first_entry.metadata_location, second_entry.metadata_location);
+        assert_ne!(first_committed_entry.metadata_location, second_entry.metadata_location);
+        assert_eq!(second.metadata["table-uuid"], second_entry.table_uuid);
+        assert_eq!(second.metadata["metadata-log"], serde_json::json!([]));
+        assert_eq!(
+            metadata_backend
+                .read_object("warehouse", &first_entry.metadata_location)
+                .await
+                .expect("retained metadata lookup should succeed")
+                .expect("retained metadata should still exist")
+                .data,
+            retained_initial.data
+        );
+        assert!(
+            metadata_backend
+                .object_exists("warehouse", &first_committed_entry.metadata_location)
+                .await
+                .expect("committed metadata lookup should succeed")
+        );
+        assert!(
+            metadata_backend
+                .object_exists("warehouse", &second_entry.metadata_location)
+                .await
+                .expect("recreated metadata lookup should succeed")
+        );
+    }
+
+    #[tokio::test]
+    async fn concurrent_create_table_responses_keep_one_catalog_winner_with_distinct_metadata() {
+        let catalog_backend = TestTableCatalogObjectBackend::default();
+        let store = crate::table_catalog::ObjectTableCatalogStore::new(catalog_backend);
+        let metadata_backend = TestTableCatalogObjectBackend {
+            objects: Arc::new(tokio::sync::Mutex::new(BTreeMap::new())),
+            put_object_barrier: Some(Arc::new(tokio::sync::Barrier::new(2))),
+        };
+        let namespace = crate::table_catalog::Namespace::parse("analytics").expect("namespace should parse");
+        ensure_table_bucket_entry(&store, "warehouse", true)
+            .await
+            .expect("table bucket entry should be seeded");
+        create_namespace_response(
+            &store,
+            "warehouse",
+            CreateNamespaceRequest {
+                namespace: vec!["analytics".to_string()],
+                properties: BTreeMap::new(),
+            },
+            true,
+        )
+        .await
+        .expect("namespace should be created");
+        let create_request = || {
+            serde_json::from_value::<CreateTableRequest>(serde_json::json!({
+                "name": "events",
+                "schema": {
+                    "type": "struct",
+                    "schema-id": 0,
+                    "fields": [
+                        {
+                            "id": 1,
+                            "name": "id",
+                            "required": true,
+                            "type": "long"
+                        }
+                    ]
+                }
+            }))
+            .expect("concurrent create table request should parse")
+        };
+
+        let (first, second) = tokio::join!(
+            create_table_response(&store, &metadata_backend, "warehouse", &namespace, create_request(), true,),
+            create_table_response(&store, &metadata_backend, "warehouse", &namespace, create_request(), true,)
+        );
+        assert_ne!(first.is_ok(), second.is_ok(), "exactly one concurrent create should succeed");
+        let (winner, loser) = match (first, second) {
+            (Ok(winner), Err(loser)) | (Err(loser), Ok(winner)) => (winner, loser),
+            _ => unreachable!("success count was checked above"),
+        };
+        assert!(format!("{loser:?}").contains("AlreadyExistsException"));
+
+        let tables = store
+            .list_tables("warehouse", "analytics")
+            .await
+            .expect("table listing should succeed");
+        assert_eq!(tables.len(), 1);
+        let winner_entry = &tables[0];
+        assert_eq!(
+            winner.metadata_location,
+            table_metadata_location_for_client("warehouse", &winner_entry.metadata_location)
+        );
+        let metadata_prefix = winner_entry
+            .metadata_location
+            .rsplit_once('/')
+            .map(|(prefix, _)| format!("{prefix}/"))
+            .expect("metadata location should contain a file name");
+        let metadata_objects = metadata_backend
+            .list_objects("warehouse", &metadata_prefix)
+            .await
+            .expect("metadata listing should succeed");
+        assert_eq!(metadata_objects.len(), 2);
+        assert_ne!(metadata_objects[0], metadata_objects[1]);
+
+        let mut table_uuids = Vec::with_capacity(metadata_objects.len());
+        for metadata_object in metadata_objects {
+            let metadata = metadata_backend
+                .read_object("warehouse", &metadata_object)
+                .await
+                .expect("metadata lookup should succeed")
+                .expect("metadata object should exist");
+            let metadata: serde_json::Value =
+                serde_json::from_slice(&metadata.data).expect("metadata object should contain JSON");
+            table_uuids.push(
+                metadata["table-uuid"]
+                    .as_str()
+                    .expect("metadata should contain table uuid")
+                    .to_string(),
+            );
+        }
+        table_uuids.sort();
+        table_uuids.dedup();
+        assert_eq!(table_uuids.len(), 2);
+        assert!(table_uuids.contains(&winner_entry.table_uuid));
+    }
+
     #[tokio::test]
     async fn standard_commit_applies_updates_and_writes_next_metadata() {
         let store = TestTableCatalogStore::default();
@@ -6991,10 +7201,7 @@ mod tests {
         assert_eq!(commit.metadata["current-snapshot-id"], 10);
         assert_eq!(commit.metadata["last-sequence-number"], 1);
         assert_eq!(commit.metadata["refs"]["main"]["snapshot-id"], 10);
-        assert_eq!(
-            commit.metadata["metadata-log"][0]["metadata-file"],
-            "s3://warehouse/.rustfs-table/warehouses/default/namespaces/analytics/tables/events/metadata/00001.metadata.json"
-        );
+        assert_eq!(commit.metadata["metadata-log"][0]["metadata-file"], created.metadata_location);
         let committed = store
             .load_table("warehouse", "analytics", "events")
             .await

rustfs/src/app/lifecycle_transition_api_test.rs

@@ -1530,9 +1530,20 @@ async fn put_bucket_lifecycle_configuration_rejects_zero_day_expiration() {
 /// copy-back completes the object reports `ongoing-request="false"` with a
 /// future expiry-date; and a full GET is then served from the local restored
 /// copy (the mock tier records no further `get` calls).
+///
+/// CURRENTLY EXCLUDED from the CI ILM Integration (serial) lane (see ci.yml):
+/// the #4877 restore self-deadlock is now fixed, so restore completes, but this
+/// test asserts a concurrent `get_object_info` observes `ongoing-request="true"`
+/// mid-restore. #4877 serializes reads against the restore write lock, so the
+/// concurrent read only returns once the copy-back has finished and already
+/// cleared the ongoing flag — it reads `false`, failing the assertion. Whether
+/// a concurrent restore should instead reject fast (`ErrObjectRestoreAlreadyInProgress`)
+/// while keeping HEAD non-blocking (backlog#1148 ilm-8 criterion 1) is an
+/// API-semantics decision, not a bug; revisit before re-enabling. The `test/`
+/// prefix and the rest of the contract are correct.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 #[serial]
-#[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-8)"]
+#[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-8); currently excluded there — asserts a concurrent ongoing-request=true read that #4877's read-vs-restore serialization rules out (API-semantics decision)"]
 async fn restore_object_usecase_reports_ongoing_conflict_and_completion() {
     let (_disk_paths, ecstore) = setup_test_env().await;
     let usecase = DefaultObjectUsecase::from_global();
@@ -1648,3 +1659,53 @@ async fn restore_object_usecase_reports_ongoing_conflict_and_completion() {
         "GET of a restored object must be served locally, not from the tier"
     );
 }
+
+/// rustfs/backlog#1320: a single PUT must compute the replication decision
+/// exactly once. Historically the PUT path computed `must_replicate_object`
+/// twice with the same inputs — once before commit to persist the pending
+/// replication metadata, and again after commit to drive the schedule. Besides
+/// repeating the versioning/config/target traversal on the hot path, a
+/// replication-config hot update between the two computations could split the
+/// two phases (pending-without-schedule or schedule-without-pending). The fix
+/// reuses one immutable decision for both phases; this test observes the
+/// computation counter and fails if a second computation is reintroduced.
+#[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+#[serial]
+#[ignore = "global-state usecase integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
+async fn put_object_computes_replication_decision_exactly_once() {
+    use super::storage_api::object_usecase::bucket::replication::MUST_REPLICATE_OBJECT_CALLS;
+    use std::sync::atomic::Ordering;
+
+    let (_disk_paths, ecstore) = setup_test_env().await;
+    let fs = FS::new();
+    let usecase = DefaultObjectUsecase::from_global();
+
+    let bucket = format!("test-repl-decision-once-{}", &Uuid::new_v4().simple().to_string()[..8]);
+    let object = "test/object.txt";
+    let payload = b"replication decision single-compute payload";
+
+    create_test_bucket(&ecstore, bucket.as_str()).await;
+
+    // Reset immediately before the single PUT so the counter reflects only this
+    // request. `#[serial]` guarantees no other usecase PUT runs concurrently, so
+    // no unrelated `must_replicate_object` call can perturb the count.
+    MUST_REPLICATE_OBJECT_CALLS.store(0, Ordering::SeqCst);
+
+    let input = PutObjectInput::builder()
+        .bucket(bucket.clone())
+        .key(object.to_string())
+        .body(Some(streaming_blob_from_bytes(payload)))
+        .content_length(Some(payload.len() as i64))
+        .build()
+        .unwrap();
+    Box::pin(usecase.execute_put_object(&fs, build_request(input, Method::PUT)))
+        .await
+        .expect("Failed to upload object through usecase");
+
+    let calls = MUST_REPLICATE_OBJECT_CALLS.load(Ordering::SeqCst);
+    assert_eq!(
+        calls, 1,
+        "a single PUT must compute the replication decision exactly once; reverting to a \
+         pre-commit + post-commit recompute makes this observe 2 (rustfs/backlog#1320)"
+    );
+}

rustfs/src/app/storage_api.rs

@@ -617,6 +617,8 @@ pub(crate) mod bucket {
         }
 
         pub(crate) type QuotaOperation = crate::storage::storage_api::ecstore_bucket::quota::QuotaOperation;
+        pub(crate) type QuotaCheckResult = crate::storage::storage_api::ecstore_bucket::quota::QuotaCheckResult;
+        pub(crate) type QuotaError = crate::storage::storage_api::ecstore_bucket::quota::QuotaError;
     }
 
     pub(crate) mod replication {
@@ -638,6 +640,16 @@ pub(crate) mod bucket {
         #[cfg(test)]
         pub(crate) use replication_contracts::replication_statuses_map;
 
+        /// Test-only counter of `must_replicate_object` invocations.
+        ///
+        /// Used by white-box regression tests to assert that a single PUT
+        /// computes the replication decision exactly once (see
+        /// https://github.com/rustfs/backlog/issues/1320). A revert that
+        /// re-introduces a second computation makes the count observe 2 and
+        /// fails the test.
+        #[cfg(test)]
+        pub(crate) static MUST_REPLICATE_OBJECT_CALLS: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0);
+
         pub(crate) async fn check_replicate_delete(
             bucket: &str,
             dobj: &super::super::storage_contracts::ObjectToDelete,
@@ -667,6 +679,8 @@ pub(crate) mod bucket {
             status: ReplicationStatusType,
             opts: crate::storage::storage_api::StorageObjectOptions,
         ) -> ReplicateDecision {
+            #[cfg(test)]
+            MUST_REPLICATE_OBJECT_CALLS.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
             let mopts = ReplicationObjectBridge::must_replicate_options(
                 user_defined,
                 user_tags,
@@ -811,8 +825,8 @@ pub(crate) mod bucket {
 
 pub(crate) mod concurrency {
     pub(crate) use crate::storage::storage_api::concurrency_consumer::{
-        ConcurrencyManager, GetObjectGuard, IoQueueStatus, IoStrategy, PutObjectGuard, get_concurrency_aware_buffer_size,
-        get_concurrency_manager, get_put_concurrency_aware_buffer_size,
+        ConcurrencyManager, DiskReadAdmission, GetObjectGuard, IoQueueStatus, IoStrategy, PutObjectGuard,
+        get_concurrency_aware_buffer_size, get_concurrency_manager, get_put_concurrency_aware_buffer_size,
     };
 }
 

rustfs/src/storage/concurrency/manager.rs

@@ -41,6 +41,12 @@ pub(crate) static CONCURRENCY_MANAGER: LazyLock<ConcurrencyManager> = LazyLock::
 pub struct ConcurrencyManager {
     /// Semaphore to limit concurrent disk reads
     disk_read_semaphore: Arc<Semaphore>,
+    /// Bounded overflow lane for GETs that time out waiting on the primary
+    /// disk-read permit pool. Admitting from this lane instead of reading
+    /// without any permit gives a hard upper bound on concurrent disk-active
+    /// reads (`primary cap + degraded cap`); when it is also full a GET is
+    /// rejected with `SlowDown` rather than proceeding unbounded.
+    degraded_read_semaphore: Arc<Semaphore>,
     /// I/O load metrics for adaptive strategy calculation
     io_metrics: Arc<Mutex<IoLoadMetrics>>,
     /// I/O priority queue for request scheduling
@@ -87,6 +93,27 @@ impl std::fmt::Debug for ConcurrencyManager {
             .finish()
     }
 }
+/// Outcome of [`ConcurrencyManager::admit_disk_read`].
+///
+/// `Primary`/`Degraded` both carry an owned permit that must be held for the
+/// full body transfer; `Rejected` means the hard concurrency cap was reached and
+/// the caller must fail the GET with `SlowDown`/503 instead of reading without a
+/// permit.
+#[derive(Debug)]
+pub enum DiskReadAdmission {
+    /// Admitted from the primary disk-read permit pool.
+    Primary(tokio::sync::OwnedSemaphorePermit),
+    /// Admitted from the bounded degraded overflow lane after the primary pool
+    /// stayed saturated past the configured wait.
+    Degraded(tokio::sync::OwnedSemaphorePermit),
+    /// Disk-read throttling is disabled (primary cap configured to `0`): the GET
+    /// proceeds without an admission token. This is the only permit-less path
+    /// and is an explicit operator opt-out, not a saturation bypass.
+    Unbounded,
+    /// Hard concurrency cap reached; the caller must reject with `SlowDown`.
+    Rejected,
+}
+
 #[allow(dead_code)]
 impl ConcurrencyManager {
     /// Create a new concurrency manager with default settings
@@ -99,6 +126,17 @@ impl ConcurrencyManager {
 
         let max_disk_reads = scheduler_config.max_concurrent_reads;
 
+        // Bounded degraded admission lane. A configured `0` mirrors the primary
+        // cap, so the absolute hard cap on concurrent disk-active reads defaults
+        // to twice the primary disk-read concurrency.
+        let degraded_read_cap = {
+            let configured = rustfs_utils::get_env_usize(
+                rustfs_config::ENV_OBJECT_DISK_DEGRADED_READ_CAP,
+                rustfs_config::DEFAULT_OBJECT_DISK_DEGRADED_READ_CAP,
+            );
+            if configured == 0 { max_disk_reads } else { configured }
+        };
+
         // Detect storage media
         let storage_media =
             detect_storage_media(scheduler_config.storage_detection_enabled, &scheduler_config.storage_media_override);
@@ -129,6 +167,7 @@ impl ConcurrencyManager {
 
         Self {
             disk_read_semaphore: Arc::new(Semaphore::new(max_disk_reads)),
+            degraded_read_semaphore: Arc::new(Semaphore::new(degraded_read_cap)),
             io_metrics: Arc::new(Mutex::new(IoLoadMetrics::new(scheduler_config.load_sample_window))),
             priority_queue: Arc::new(IoPriorityQueue::new(queue_config)),
             bytes_pool: Arc::new(BytesPool::new_tiered()),
@@ -140,6 +179,20 @@ impl ConcurrencyManager {
         }
     }
 
+    /// Build a manager with explicit disk-read admission caps for tests.
+    ///
+    /// Overrides only the primary/degraded semaphores and the snapshot cap so
+    /// admission behavior can be exercised deterministically under a paused
+    /// virtual clock, without depending on process-wide environment variables.
+    #[cfg(test)]
+    pub(crate) fn with_disk_read_caps_for_test(primary: usize, degraded: usize) -> Self {
+        let mut manager = Self::new();
+        manager.disk_read_semaphore = Arc::new(Semaphore::new(primary));
+        manager.degraded_read_semaphore = Arc::new(Semaphore::new(degraded));
+        manager.scheduler_config.max_concurrent_reads = primary;
+        manager
+    }
+
     /// Track a GetObject request
     pub fn track_request() -> GetObjectGuard {
         GetObjectGuard::new()
@@ -177,6 +230,54 @@ impl ConcurrencyManager {
         self.disk_read_semaphore.clone().acquire_owned().await
     }
 
+    /// Admit a GET disk read under a hard concurrency cap.
+    ///
+    /// The permit is held for the whole response body transfer, so this is the
+    /// single admission boundary that bounds concurrent disk-active reads:
+    ///
+    /// - `primary_wait == 0` waits on the primary permit pool indefinitely and
+    ///   always yields a [`DiskReadAdmission::Primary`] permit (no degrade, no
+    ///   reject); this preserves the "wait forever" opt-out.
+    /// - Otherwise it waits up to `primary_wait` for a primary permit. On
+    ///   timeout it takes one permit from the bounded degraded overflow lane
+    ///   without blocking. If that lane is also full the request is
+    ///   [`DiskReadAdmission::Rejected`] — it must surface as `SlowDown`/503
+    ///   rather than reading without any admission token.
+    ///
+    /// Total GETs holding a permit therefore never exceed
+    /// `primary_cap + degraded_cap`.
+    pub async fn admit_disk_read(&self, primary_wait: Duration) -> Result<DiskReadAdmission, tokio::sync::AcquireError> {
+        // Primary cap of 0 means disk-read throttling is disabled (matches the
+        // `AdmissionState::Disabled` snapshot semantics). Serve without an
+        // admission token rather than rejecting every GET; this preserves the
+        // pre-existing "permits disabled" behavior and is the only permit-less
+        // path.
+        if self.scheduler_config.max_concurrent_reads == 0 {
+            return Ok(DiskReadAdmission::Unbounded);
+        }
+
+        if primary_wait.is_zero() {
+            let permit = self.disk_read_semaphore.clone().acquire_owned().await?;
+            return Ok(DiskReadAdmission::Primary(permit));
+        }
+
+        match tokio::time::timeout(primary_wait, self.disk_read_semaphore.clone().acquire_owned()).await {
+            Ok(permit) => Ok(DiskReadAdmission::Primary(permit?)),
+            Err(_) => {
+                // Primary pool saturated within the wait window. Do not read
+                // without a permit; fall through to the bounded degraded lane,
+                // and reject once the hard cap is reached.
+                match self.degraded_read_semaphore.clone().try_acquire_owned() {
+                    Ok(permit) => Ok(DiskReadAdmission::Degraded(permit)),
+                    // NoPermits => hard cap reached; Closed never happens for a
+                    // semaphore we never close. Either way, reject rather than
+                    // bypass admission.
+                    Err(_) => Ok(DiskReadAdmission::Rejected),
+                }
+            }
+        }
+    }
+
     // ============================================
     // Adaptive I/O Strategy Methods
     // ============================================
@@ -1035,4 +1136,151 @@ mod integration_tests {
         assert_eq!(small_file_strategy.priority, IoPriority::High);
         assert_eq!(large_file_strategy.priority, IoPriority::Low);
     }
+
+    // ============================================
+    // Bounded disk-read admission (backlog#1317)
+    // ============================================
+
+    use super::DiskReadAdmission;
+    use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering};
+
+    /// Primary pool saturation degrades into the bounded lane, and the hard cap
+    /// rejects with `Rejected` instead of admitting a permit-less read. Dropping
+    /// a permit frees the token so the next admission succeeds again.
+    #[tokio::test(start_paused = true)]
+    async fn test_admit_disk_read_degrades_then_hard_rejects() {
+        let manager = ConcurrencyManager::with_disk_read_caps_for_test(1, 1);
+
+        // Primary permit granted immediately.
+        let primary = match manager.admit_disk_read(Duration::from_millis(100)).await.unwrap() {
+            DiskReadAdmission::Primary(permit) => permit,
+            other => panic!("expected primary admission, got {other:?}"),
+        };
+
+        // Primary pool saturated: next admission waits out the primary timeout
+        // and falls through to the bounded degraded lane.
+        let degraded = match manager.admit_disk_read(Duration::from_millis(100)).await.unwrap() {
+            DiskReadAdmission::Degraded(permit) => permit,
+            other => panic!("expected degraded admission, got {other:?}"),
+        };
+
+        // Both lanes full: hard cap reached, must reject rather than bypass.
+        match manager.admit_disk_read(Duration::from_millis(100)).await.unwrap() {
+            DiskReadAdmission::Rejected => {}
+            other => panic!("expected hard rejection, got {other:?}"),
+        }
+
+        // Releasing the degraded permit re-opens exactly one degraded slot.
+        drop(degraded);
+        match manager.admit_disk_read(Duration::from_millis(100)).await.unwrap() {
+            DiskReadAdmission::Degraded(_permit) => {}
+            other => panic!("expected degraded admission after release, got {other:?}"),
+        }
+
+        // Releasing the primary permit re-opens the primary lane immediately
+        // (no timeout wait), proving EOF/drop returns the token.
+        drop(primary);
+        match manager.admit_disk_read(Duration::from_millis(100)).await.unwrap() {
+            DiskReadAdmission::Primary(_permit) => {}
+            other => panic!("expected primary admission after release, got {other:?}"),
+        }
+    }
+
+    /// With max primary = 1 and degraded = 1, 100 concurrent GETs never hold
+    /// more than `1 + 1` admission tokens simultaneously. Reverting the hard cap
+    /// (unbounded bypass) would let this exceed 2 and fail the test.
+    #[tokio::test(start_paused = true)]
+    async fn test_admit_disk_read_hard_caps_concurrent_admissions() {
+        let manager = std::sync::Arc::new(ConcurrencyManager::with_disk_read_caps_for_test(1, 1));
+        let in_flight = std::sync::Arc::new(AtomicUsize::new(0));
+        let max_in_flight = std::sync::Arc::new(AtomicUsize::new(0));
+        let admitted = std::sync::Arc::new(AtomicUsize::new(0));
+        let rejected = std::sync::Arc::new(AtomicUsize::new(0));
+
+        let mut handles = Vec::with_capacity(100);
+        for _ in 0..100 {
+            let manager = manager.clone();
+            let in_flight = in_flight.clone();
+            let max_in_flight = max_in_flight.clone();
+            let admitted = admitted.clone();
+            let rejected = rejected.clone();
+            handles.push(tokio::spawn(async move {
+                match manager.admit_disk_read(Duration::from_millis(50)).await.unwrap() {
+                    DiskReadAdmission::Primary(permit) | DiskReadAdmission::Degraded(permit) => {
+                        let current = in_flight.fetch_add(1, AtomicOrdering::SeqCst) + 1;
+                        max_in_flight.fetch_max(current, AtomicOrdering::SeqCst);
+                        admitted.fetch_add(1, AtomicOrdering::SeqCst);
+                        // Hold the admission token across an await point, as the
+                        // real body transfer does, then release it.
+                        tokio::time::sleep(Duration::from_millis(10)).await;
+                        in_flight.fetch_sub(1, AtomicOrdering::SeqCst);
+                        drop(permit);
+                    }
+                    DiskReadAdmission::Rejected => {
+                        rejected.fetch_add(1, AtomicOrdering::SeqCst);
+                    }
+                    DiskReadAdmission::Unbounded => {
+                        unreachable!("throttling is enabled (primary cap = 1) in this test");
+                    }
+                }
+            }));
+        }
+
+        for handle in handles {
+            handle.await.unwrap();
+        }
+
+        // The invariant the hard cap guarantees: never more than
+        // primary + degraded = 2 GETs admitted at once.
+        assert!(
+            max_in_flight.load(AtomicOrdering::SeqCst) <= 2,
+            "max concurrent admissions {} exceeded the hard cap of 2",
+            max_in_flight.load(AtomicOrdering::SeqCst)
+        );
+        // Every request is accounted for: admitted or explicitly rejected, never
+        // a permit-less bypass.
+        assert_eq!(admitted.load(AtomicOrdering::SeqCst) + rejected.load(AtomicOrdering::SeqCst), 100);
+        assert!(
+            rejected.load(AtomicOrdering::SeqCst) > 0,
+            "expected some hard rejections under saturation"
+        );
+    }
+
+    /// A primary cap of 0 (throttling disabled) serves every GET without an
+    /// admission token instead of rejecting them, preserving the pre-existing
+    /// "disk read permits disabled" behavior.
+    #[tokio::test(start_paused = true)]
+    async fn test_admit_disk_read_disabled_cap_is_unbounded() {
+        let manager = ConcurrencyManager::with_disk_read_caps_for_test(0, 0);
+        for _ in 0..10 {
+            match manager.admit_disk_read(Duration::from_millis(50)).await.unwrap() {
+                DiskReadAdmission::Unbounded => {}
+                other => panic!("expected unbounded admission when throttling disabled, got {other:?}"),
+            }
+        }
+    }
+
+    /// `primary_wait == 0` preserves the wait-forever opt-out: it always yields
+    /// a primary permit and never degrades or rejects.
+    #[tokio::test]
+    async fn test_admit_disk_read_zero_wait_waits_on_primary() {
+        let manager = ConcurrencyManager::with_disk_read_caps_for_test(1, 1);
+        let held = match manager.admit_disk_read(Duration::ZERO).await.unwrap() {
+            DiskReadAdmission::Primary(permit) => permit,
+            other => panic!("expected primary admission, got {other:?}"),
+        };
+
+        // A second zero-wait admission blocks on the primary pool rather than
+        // degrading; it completes only once the first permit is released.
+        let manager2 = manager.clone();
+        let waiter = tokio::spawn(async move { manager2.admit_disk_read(Duration::ZERO).await.unwrap() });
+        tokio::task::yield_now().await;
+        assert!(!waiter.is_finished(), "zero-wait admission must block on the primary lane");
+
+        drop(held);
+        match waiter.await.unwrap() {
+            DiskReadAdmission::Primary(_permit) => {}
+            other => panic!("expected primary admission after release, got {other:?}"),
+        }
+    }
 }

rustfs/src/storage/concurrency/mod.rs

@@ -54,7 +54,7 @@ pub use io_schedule::{
 pub use request_guard::{GetObjectGuard, PutObjectGuard};
 
 // Concurrency manager
-pub use manager::ConcurrencyManager;
+pub use manager::{ConcurrencyManager, DiskReadAdmission};
 
 // ============================================
 // New Module Re-exports (for gradual migration)

rustfs/src/storage/storage_api.rs

@@ -114,8 +114,8 @@ pub(crate) mod access_consumer {
 
 pub(crate) mod concurrency_consumer {
     pub(crate) use super::super::concurrency::{
-        ConcurrencyManager, GetObjectGuard, IoQueueStatus, IoStrategy, PutObjectGuard, get_concurrency_aware_buffer_size,
-        get_concurrency_manager, get_put_concurrency_aware_buffer_size,
+        ConcurrencyManager, DiskReadAdmission, GetObjectGuard, IoQueueStatus, IoStrategy, PutObjectGuard,
+        get_concurrency_aware_buffer_size, get_concurrency_manager, get_put_concurrency_aware_buffer_size,
     };
 }
 

scripts/test_helm_templates.sh

@@ -303,3 +303,70 @@ if grep -q 'svc\.cluster\.local"' <<<"$cert_custom"; then
   echo "Custom clusterDomain must fully replace cluster.local in mTLS server cert SANs" >&2
   exit 1
 fi
+
+# Legacy topology compatibility: default replicaCount=4 (no drivesPerNode set)
+# must render the old 4x4 PVC names (data-rustfs-0 .. data-rustfs-3).
+legacy_four_by_four=$(render_distributed_statefulset)
+for i in 0 1 2 3; do
+  if ! grep -q "name: data-rustfs-${i}" <<<"$legacy_four_by_four"; then
+    echo "Legacy 4x4 topology must contain PVC data-rustfs-${i}" >&2
+    exit 1
+  fi
+done
+if grep -q "name: data$" <<<"$legacy_four_by_four"; then
+  echo "Legacy 4x4 topology must NOT contain a single 'data' PVC" >&2
+  exit 1
+fi
+
+# Legacy topology compatibility: replicaCount=16 (no drivesPerNode set)
+# must render a single 'data' PVC (old 16x1 behaviour).
+legacy_sixteen_by_one=$(render_distributed_statefulset --set replicaCount=16)
+if ! grep -q "name: data$" <<<"$legacy_sixteen_by_one"; then
+  echo "Legacy 16x1 topology must contain a single 'data' PVC" >&2
+  exit 1
+fi
+if grep -q "name: data-rustfs-" <<<"$legacy_sixteen_by_one"; then
+  echo "Legacy 16x1 topology must NOT contain data-rustfs-* PVCs" >&2
+  exit 1
+fi
+
+# Generic topology: explicit replicaCount=8 drivesPerNode=2 must render
+# exactly two PVCs per pod.
+generic_eight_by_two=$(render_distributed_statefulset --set replicaCount=8 --set drivesPerNode=2)
+for i in 0 1; do
+  if ! grep -q "name: data-rustfs-${i}" <<<"$generic_eight_by_two"; then
+    echo "Generic 8x2 topology must contain PVC data-rustfs-${i}" >&2
+    exit 1
+  fi
+done
+if grep -q "name: data$" <<<"$generic_eight_by_two"; then
+  echo "Generic 8x2 topology must NOT contain a single 'data' PVC" >&2
+  exit 1
+fi
+
+# volumeClaimTemplates must not contain empty annotations when pvcAnnotations are unset,
+# because Kubernetes treats annotations: {} as a mutation of the immutable field.
+no_ann_output=$(render_distributed_statefulset)
+if grep -A1 'kind: PersistentVolumeClaim' <<<"$no_ann_output" | grep -q 'annotations:'; then
+  echo "Empty pvcAnnotations must not render an annotations key in volumeClaimTemplates" >&2
+  exit 1
+fi
+
+# Distributed mode with replicaCount < 2 must fail rendering.
+low_replica_status=0
+render_distributed_statefulset --set replicaCount=1 >/dev/null 2>&1 || low_replica_status=$?
+if [[ $low_replica_status -eq 0 ]]; then
+  echo "Distributed mode with replicaCount=1 must fail rendering" >&2
+  exit 1
+fi
+
+# service.externalIPs must render correctly when supplied.
+external_ips_output=$(helm template rustfs "$CHART_DIR" \
+  --namespace rustfs \
+  --set secret.rustfs.access_key=test-access-key \
+  --set secret.rustfs.secret_key=test-secret-key \
+  --set 'service.externalIPs[0]=203.0.113.1')
+if ! grep -A2 'externalIPs:' <<<"$external_ips_output" | grep -q '203.0.113.1'; then
+  echo "service.externalIPs must contain 203.0.113.1" >&2
+  exit 1
+fi