chore(release): prepare 1.0.0-beta.10 (#4946 )

docs(operations): add drive timeout tuning guide for slow storage (#4944 )
Issue #4810's production incident showed the drive timeout knobs are undiscoverable from symptoms: a listing that outruns the walk budget gives operators no signal pointing at RUSTFS_DRIVE_* tuning. The knobs existed only as code constants in crates/config/src/constants/drive.rs with no operator-facing documentation. Add docs/operations/drive-timeout-tuning.md covering the per-operation drive timeout knobs, resolution precedence, the high_latency profile, and a dedicated section on listing truncation and the walk stall budget - including the correction that since the stall-budget rework the foreground listing path is governed by RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS, not the total-timeout knob the issue suggested. Link the guide from the README. Ref #4810
2026-07-29 09:38:59 +00:00 · 2026-07-17 10:57:58 +08:00 · 2026-07-17 10:34:16 +08:00 · 2026-07-17 10:26:54 +08:00 · 2026-07-17 01:04:32 +00:00 · 2026-07-17 01:02:38 +00:00
269 changed files with 31144 additions and 3699 deletions
@@ -0,0 +1,169 @@
+---
+name: rustfs-release-publish
+description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
+---
+# RustFS Release Publish (preview-validated pipeline)
+
+This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
+
+Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
+
+Pipeline shape:
+
+```
+bump version files to <target> (final version, ONE commit) -> merge
+  -> tag <preview-tag> at that commit -> CI green
+  -> verify release artifacts -> run binary locally + console checks
+  -> validate with latest rc client
+  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
+```
+
+On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
+
+## Required inputs
+
+- Final target version, for example `1.0.0-beta.10`.
+- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
+
+If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
+
+## Semver gate — confirm the target version before touching anything
+
+Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
+
+```
+1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
+```
+
+Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
+
+Rules:
+
+- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
+- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
+- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
+
+## Preview tag naming
+
+- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
+- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
+
+## Hard rules
+
+- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
+- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
+- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
+- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
+- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
+- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
+
+## Phase 0 — Preflight
+
+- `git status --short` clean; `git fetch origin main --tags`.
+- `gh auth status` works; confirm you can view `gh release list -L 3`.
+- Confirm the exact final target version with the user if not explicit.
+
+## Phase 1 — Version bump to the final target (once)
+
+- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
+- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
+- Get the PR merged into main. Record the resulting main commit:
+
+```bash
+git fetch origin main
+PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
+```
+
+`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
+
+## Phase 2 — Publish the preview tag
+
+```bash
+git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
+git push origin "<preview-tag>"
+```
+
+Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
+
+On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
+
+## Phase 3 — CI and artifact verification
+
+- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
+- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
+  - `isPrerelease` must be `true`.
+  - Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
+- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
+- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
+
+## Phase 4 — Run the artifact locally, verify the console
+
+Work inside the session scratchpad directory; never leave stray data dirs.
+
+```bash
+gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
+cd "$SCRATCH" && unzip -o rustfs-*.zip
+./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
+mkdir -p data
+RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
+```
+
+Defaults: S3 endpoint `:9000`, embedded console `:9001`.
+
+Checks (all must pass):
+
+- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
+- `curl -fsS http://localhost:9000/health/ready` returns ready.
+- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
+- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
+
+## Phase 5 — Validate with the latest rc client
+
+`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
+
+- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
+- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
+
+```bash
+rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
+rc ls preview/
+rc mb preview/rel-check
+rc cp <local-file> preview/rel-check/
+rc stat preview/rel-check/<file>
+rc cat preview/rel-check/<file>          # matches source
+rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
+rc cp -r <local-dir>/ preview/rel-check/dir/
+rc find preview/rel-check --name "*"
+rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
+rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
+rc rb preview/rel-check
+rc admin user list preview/
+rc admin user add preview/ relcheckuser relchecksecret12
+rc admin user remove preview/ relcheckuser
+rc alias remove preview
+```
+
+- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
+
+## Phase 6 — Publish the final tag on the validated commit
+
+No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
+
+```bash
+git fetch origin --tags
+git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
+git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
+git push origin "<target>"
+```
+
+- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
+- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
+- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
+
+## Output contract
+
+Always report:
+
+- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
+- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
+- Any deviation from this pipeline and why the user approved it.
@@ -18,6 +18,8 @@ Validated baseline: release pattern used in PR `#2957`.

 If target version is missing or ambiguous, stop and ask before editing.

+Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
+
 ## Read before editing

 - `AGENTS.md` (root and nearest path-specific files).
@@ -0,0 +1,26 @@
+## —— Coverage --------------------------------------------------------------------------------------
+
+# Local equivalent of the weekly coverage workflow (.github/workflows/coverage.yml,
+# backlog#1153 infra-5): same measurement scope (--workspace --exclude e2e_test,
+# nextest `ci` profile) and the same per-crate table. Slow — the instrumented
+# build cannot reuse your normal target cache and then runs the whole suite.
+# Doctests are not measured (needs nightly). Outputs land in target/llvm-cov/.
+.PHONY: coverage
+coverage: core-deps ## Workspace line coverage (cargo-llvm-cov + nextest; slow, writes target/llvm-cov/)
+	@if ! command -v cargo-llvm-cov >/dev/null 2>&1; then \
+		echo >&2 "❌ cargo-llvm-cov is required for 'make coverage' but was not found."; \
+		echo >&2 "   Install it with:"; \
+		echo >&2 "     cargo install cargo-llvm-cov --locked"; \
+		echo >&2 "     rustup component add llvm-tools-preview"; \
+		exit 1; \
+	fi
+	@if ! command -v cargo-nextest >/dev/null 2>&1; then \
+		echo >&2 "❌ cargo-nextest is required for 'make coverage' (see 'make test')."; \
+		echo >&2 "   Install it with: cargo install cargo-nextest --locked"; \
+		exit 1; \
+	fi
+	NEXTEST_PROFILE=ci cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
+	@mkdir -p target/llvm-cov
+	cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
+	cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
+	python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json
@@ -45,6 +45,15 @@ e2e-reliability = { max-threads = 1 }
 filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'

+# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
+# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
+# commit + GET cycles — the same cross-disk-commit IO shape that made
+# concurrent_resend load-sensitive. Preventive serialization only, no retries.
+# The matching ci-profile override is after [profile.ci].
+[[profile.default.overrides]]
+filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
+test-group = 'ecstore-serial-flaky'
+
 # Serialize the 4-disk reliability / degraded-read e2e tests (see the
 # e2e-reliability test-group note above). The matching ci-profile override is at
 # the end of the file, after [profile.ci] is declared.
@@ -109,6 +118,13 @@ retries = 2
 filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'

+# Serialize the multipart crash-consistency scenarios under the ci profile too
+# (see the matching default-profile override near the top). Not a quarantine:
+# no retries, just serialized 4-disk cross-disk-commit IO.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
+test-group = 'ecstore-serial-flaky'
+
 # ---------------------------------------------------------------------------
 # e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
 # ---------------------------------------------------------------------------
@@ -122,6 +138,10 @@ test-group = 'e2e-reliability'
 # Each e2e test spawns its own rustfs server on a random port with an isolated
 # temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
 #
+# Replication failure harness (backlog#1147 repl-8): the first clause admits
+# its four in-process fake-target self-tests. They bind random loopback ports,
+# use no external service, and finish in under a second.
+#
 # Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
 # FAST bucket-replication tests from replication_extension_test — the
 # target-registration / replication-check / list / remove / delete admin paths
@@ -136,7 +156,7 @@ test-group = 'e2e-reliability'
 # the nightly profile derives its set as "the replication module MINUS this
 # allowlist", so any new replication test lands in nightly by default (never
 # silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. Count invariant: 20 here + 18 nightly = 38 total
+# regexes byte-identical. Count invariant: 20 here + 27 nightly = 47 total
 # (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
 # HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
 # (#4724) because they set a loopback (127.0.0.1) replication target that the
@@ -144,12 +164,38 @@ test-group = 'e2e-reliability'
 # the guard now honours an off-by-default opt-in and this suite's source servers
 # set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
 # restored.
+#
+# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
+# S3 auth-rejection suites join the first clause above by module name —
+# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
+# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
+# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
+# smoke admission criteria unchanged. This is the wiring step that makes those
+# merged suites actually execute on every PR (they were dead until listed here).
+# A rename that drops any of them out of this filter would silently thin the
+# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
+# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
+# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
+# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
+# (protocols::test_protocol_core_suite) binds fixed ports and needs the
+# ftps,webdav features, so it cannot join this random-port, default-feature
+# profile; its GHSA-r5qv sibling is a unit test that already runs in the
+# test-and-lint `--all --exclude e2e_test` pass. See
+# docs/testing/security-regressions.md for the full CI-execution map.
+#
+# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
+# admits the hermetic transition e2e. Like the fast replication pair checks it
+# spawns a second independent single-node server (the cold RustFS tier), not a
+# cluster, so it keeps the lane's parallel-safe / no-external-dependency
+# properties. The RustFS warm backend has no loopback guard (that guard is
+# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
 [profile.e2e-smoke]
 default-filter = """
    package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative)_test::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud)_test::|^fake_s3_target::/)
        | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
        | test(/^reliant::lifecycle::/)
+        | test(/^reliant::tiering::/)
    )
 """
 fail-fast = false
@@ -160,10 +206,14 @@ fail-fast = false
 # backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
 # tests that are unfit for the per-PR e2e-smoke gate:
 #
-#   * 8 bucket-replication data-plane tests — they PUT/delete objects and poll
-#     until source and target converge; two replicate over HTTPS.
-#   * 9 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
+#   * 2 remote-target TLS validation tests.
+#   * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
+#     and poll until source and target converge; two replicate over HTTPS, two
+#     pin active SSE failure contracts, and one guards event/history observers.
+#     The SSE-S3 contract remains ignored under backlog#1291.
+#   * 11 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
 #     servers and drives the cross-process site-replication control plane.
+#   * 1 `_real_three_node` site-replication test.
 #   * 1 `_real_single_node` service-account round-trip test.
 #
 # The set is defined as "everything in replication_extension_test that is NOT
@@ -199,3 +249,79 @@ fail-fast = false
 # Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
 # workflow as the failure-triage artifact.
 path = "junit.xml"
+
+# ---------------------------------------------------------------------------
+# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
+# ---------------------------------------------------------------------------
+# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
+# workflow_dispatch). Runs the never-automated user-visible suites — KMS (40),
+# object_lock (33), multipart_auth (109), quota, checksum, encryption,
+# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
+# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
+# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
+#
+# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
+#   * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed
+#     ports; they join a scheduled lane once ci-6 randomises the ports (ci-7).
+#   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
+#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
+#     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
+#     object_lambda) — too heavy for the merge budget; they run in ci-7's
+#     nightly 4-node lane.
+#   * replication_extension_test — repl-1 already splits it into the PR
+#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
+#     it for those, so e2e-full does not double-run it.
+#   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
+#     manual-localhost:9000 reliant/policy tests are ci-13's migration.
+#
+# Each e2e test spawns its own single-node rustfs server on a random port with
+# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
+# parallel-safe — the same property e2e-smoke relies on. The exception is the
+# 4-disk reliability / degraded-read fault-injection tests, serialized below
+# (identical to the ci profile) so several 4-disk servers never run at once.
+# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
+# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
+# product failures cannot be quarantined away with retries, so each family is
+# excluded here with its tracking issue, under the same discipline as the
+# ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
+# OPEN issue, and the fixing PR MUST delete the exclusion. The passing
+# negative-path siblings of each family stay in as regression guards.
+#   * rustfs#4842 — extract/snowball expand pipeline 500s (mtime=0
+#     OffsetDateTime deserialization + same-path failures).
+#   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
+#     archive even under ignore-errors semantics.
+#   * rustfs#4844 — anonymous POST-object with SSE-S3 / bucket-default SSE
+#     returns 500.
+#   * rustfs#4845 — 403 on allowed anonymous POST object-lock fields and on
+#     the list metadata=true extension.
+#   * rustfs#4846 — distributed-lock quorum tests misclassify as timeout
+#     under parallel load (multi-node in-process clusters; natural home is
+#     ci-7's nightly cluster lane).
+[profile.e2e-full]
+default-filter = """
+    package(e2e_test)
+    & !test(/^protocols::/)
+    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
+    & !test(/^replication_extension_test::/)
+    & !test(/^multipart_auth_test::test_signed_put_object_extract_(accepts_compat_header|expands_tar_entries_with_prefix_headers|expands_tar_gz_archive|expands_tbz2_archive|expands_tgz_archive|expands_txz_archive|expands_tzst_archive|normalizes_prefix_header_value|preserves_directory_markers_by_default|preserves_object_lock_legal_hold|preserves_object_lock_retention|preserves_pax_metadata_and_version_id|preserves_request_metadata_on_extracted_objects|preserves_sse_c|preserves_sse_s3_and_redirect|preserves_storage_class|uses_bucket_default_sse_s3)$/)
+    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(prefers_exact_minio_prefix_over_suffix_fallback|supports_minio_prefix_and_directory_markers)$/)
+    & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
+    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
+    & !test(/^multipart_auth_test::test_anonymous_post_object_(accepts_sse_s3|rejects_sse_s3_missing_from_policy_conditions|uses_bucket_default_sse_kms|uses_bucket_default_sse_s3)$/)
+    & !test(/^multipart_auth_test::test_anonymous_post_object_(accepts_object_lock_legal_hold_field|accepts_object_lock_retention_fields)$/)
+    & !test(/^list_object(s_v2|_versions)_metadata_extension_test::/)
+    & !test(/^reliant::lock::test_distributed_lock_(2_nodes_grpc_read_survives_failed_node|4_nodes_grpc_read_write_quorum_split_with_two_failed_nodes)$/)
+"""
+fail-fast = false
+
+[profile.e2e-full.junit]
+# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
+path = "junit.xml"
+
+# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
+# (see the e2e-reliability test-group note near the top of this file). Not a
+# quarantine: no retries, just single-threaded so several 4-disk servers never
+# run concurrently.
+[[profile.e2e-full.overrides]]
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
+test-group = 'e2e-reliability'
@@ -0,0 +1,12 @@
+# Committed floor for the number of security negative-auth tests selected by the
+# e2e-smoke PR profile (see scripts/check_security_smoke_count.sh, backlog#1151
+# sec-5).
+#
+# The floor equals the exact count of e2e_test cases whose name starts with a
+# security module prefix (negative_sigv4_test, presigned_negative_test,
+# admin_auth_test) that the [profile.e2e-smoke] default-filter in
+# .config/nextest.toml selects, at the time this file was last updated. CI fails
+# if the selected count drops below this number, so a rename or removal that
+# thins the security smoke gate must update this file in the same PR.
+# Adding tests does not require a bump, but bumping keeps the guard tight.
+16
@@ -56,6 +56,7 @@ runs:
          libssl-dev \
          ripgrep \
          unzip \
+          zip \
          protobuf-compiler

    - name: Install protoc
@@ -136,11 +136,13 @@ jobs:
            echo "⚡ Manual/scheduled build detected"
          fi

-          echo "should_build=$should_build" >> $GITHUB_OUTPUT
-          echo "build_type=$build_type" >> $GITHUB_OUTPUT
-          echo "version=$version" >> $GITHUB_OUTPUT
-          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
-          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
+          {
+            echo "should_build=$should_build"
+            echo "build_type=$build_type"
+            echo "version=$version"
+            echo "short_sha=$short_sha"
+            echo "is_prerelease=$is_prerelease"
+          } >> "$GITHUB_OUTPUT"

          echo "📊 Build Summary:"
          echo "  - Should build: $should_build"
@@ -188,7 +190,6 @@ jobs:
            {"target_id":"linux-x86_64-gnu","os":"sm-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
            {"target_id":"linux-aarch64-gnu","os":"sm-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-26-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
          ]}'

@@ -220,8 +221,8 @@ jobs:
    name: Build RustFS
    needs: [ build-check, prepare-platform-matrix ]
    if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
-    runs-on: ${{ matrix.platform == 'linux' && fromJSON('["self-hosted","linux","sm-standard-2"]') || matrix.os }}
-    timeout-minutes: 90
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 150
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
      # Release binaries ship without dial9 telemetry and therefore do not need
@@ -288,6 +289,7 @@ jobs:
            local console_url
            local console_sha256
            local curl_auth_args=()
+            console_tag=""

            if [[ "${{ matrix.platform }}" == "windows" ]]; then
              curl_bin="curl.exe"
@@ -310,7 +312,7 @@ jobs:
            "$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
              -o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1

-            read -r console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
+            read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
          import json
          import re
          import sys
@@ -318,6 +320,8 @@ jobs:
          with open(sys.argv[1], encoding="utf-8") as handle:
              release = json.load(handle)

+          tag = release.get("tag_name", "") or "unknown"
+
          for asset in release.get("assets", []):
              name = asset.get("name", "")
              digest = asset.get("digest", "")
@@ -326,7 +330,7 @@ jobs:
                  sha256 = digest.split(":", 1)[1]
                  if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
                      raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
-                  sys.stdout.buffer.write(f"{url} {sha256}\n".encode("utf-8"))
+                  sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
                  break
          else:
              raise SystemExit("no console zip asset with sha256 digest found")
@@ -338,6 +342,8 @@ jobs:
              return 1
            fi

+            echo "Console release: ${console_tag}"
+            echo "Downloading console asset: ${console_url}"
            "$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
            verify_sha256 "$console_sha256" console.zip || return 2
            unzip -o console.zip -d ./rustfs/static || return 2
@@ -350,12 +356,19 @@ jobs:
            rm -f console.zip console-release.json
            if [[ "$status" -eq 2 ]]; then
              echo "Console asset integrity verification failed" >&2
-              exit 1
            fi
-            echo "Warning: Failed to download verified console assets, continuing without them"
-            echo "// Static assets not available" > ./rustfs/static/empty.txt
+            echo "Failed to download verified console assets" >&2
+            exit 1
          fi

+          if [[ ! -s ./rustfs/static/index.html ]]; then
+            echo "Console asset archive is missing static/index.html" >&2
+            exit 1
+          fi
+
+          asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
+          echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
+
      - name: Build RustFS
        shell: bash
        run: |
@@ -437,11 +450,11 @@ jobs:
            # Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
            PACKAGE_NAME="${PACKAGE_BASENAME}-v${PACKAGE_VERSION}"
          fi
-
+          
          # Create zip packages for all platforms
          # Ensure zip is available
          if ! command -v zip &> /dev/null; then
-            if [[ "${{ matrix.os }}" == "ubuntu-latest" ]]; then
+            if [[ "${{ matrix.os }}" == "ubuntu-latest" || "${{ matrix.platform }}" == "linux" ]]; then
              sudo apt-get update && sudo apt-get install -y zip
            fi
          fi
@@ -493,7 +506,7 @@ jobs:
            if [[ "${{ matrix.platform }}" == "windows" ]]; then
              dir
            else
-              ls -lh ${PACKAGE_NAME}.zip
+              ls -lh "${PACKAGE_NAME}.zip"
            fi
          else
            echo "❌ Failed to create package: ${PACKAGE_NAME}.zip"
@@ -542,11 +555,13 @@ jobs:
            fi
          fi

-          echo "package_name=${PACKAGE_NAME}" >> $GITHUB_OUTPUT
-          echo "package_file=${PACKAGE_NAME}.zip" >> $GITHUB_OUTPUT
-          echo "latest_files=${LATEST_FILES}" >> $GITHUB_OUTPUT
-          echo "build_type=${BUILD_TYPE}" >> $GITHUB_OUTPUT
-          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+          {
+            echo "package_name=${PACKAGE_NAME}"
+            echo "package_file=${PACKAGE_NAME}.zip"
+            echo "latest_files=${LATEST_FILES}"
+            echo "build_type=${BUILD_TYPE}"
+            echo "version=${VERSION}"
+          } >> "$GITHUB_OUTPUT"

          echo "📦 Package created: ${PACKAGE_NAME}.zip"
          if [[ -n "$LATEST_FILES" ]]; then
@@ -555,6 +570,60 @@ jobs:
          echo "🔧 Build type: ${BUILD_TYPE}"
          echo "📊 Version: ${VERSION}"

+      - name: Verify packaged console
+        if: matrix.target == 'x86_64-unknown-linux-gnu'
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          find_free_port() {
+            python3 - <<'PY'
+          import socket
+
+          with socket.socket() as sock:
+              sock.bind(("127.0.0.1", 0))
+              print(sock.getsockname()[1])
+          PY
+          }
+
+          package_dir="$(mktemp -d)"
+          data_dir="$(mktemp -d)"
+          log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
+          server_pid=""
+          trap '
+            if [[ -n "${server_pid:-}" ]]; then
+              kill "$server_pid" 2>/dev/null || true
+              wait "$server_pid" 2>/dev/null || true
+            fi
+            rm -rf "$package_dir" "$data_dir"
+          ' EXIT
+
+          unzip -q "${{ steps.package.outputs.package_file }}" -d "$package_dir"
+          api_port="$(find_free_port)"
+          console_port="$(find_free_port)"
+          console_url="http://127.0.0.1:${console_port}/rustfs/console/"
+
+          "$package_dir/rustfs" server \
+            --address "127.0.0.1:${api_port}" \
+            --console-enable \
+            --console-address "127.0.0.1:${console_port}" \
+            --access-key console-smoke \
+            --secret-key console-smoke-secret \
+            "$data_dir" >"$log_file" 2>&1 &
+          server_pid=$!
+
+          for _ in {1..40}; do
+            response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
+            if [[ "$response" == 200\ text/html* ]]; then
+              exit 0
+            fi
+            sleep 0.25
+          done
+
+          echo "Console endpoint did not return 200 text/html: $response" >&2
+          cat "$log_file"
+          exit 1
+
      - name: Upload to GitHub artifacts
        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
        with:
@@ -579,9 +648,19 @@ jobs:
            exit 0
          fi

+          # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
+          # images did). Install it on demand so R2 uploads survive a fresh runner
+          # instead of hard-failing here.
          if ! command -v aws >/dev/null 2>&1; then
-            echo "❌ aws CLI not found on runner; cannot upload to R2"
-            exit 1
+            echo "aws CLI not found on runner; installing..."
+            if command -v apt-get >/dev/null 2>&1; then
+              sudo apt-get update && sudo apt-get install -y awscli
+            elif command -v brew >/dev/null 2>&1; then
+              brew install awscli
+            else
+              echo "❌ aws CLI missing and no apt-get/brew to install it; cannot upload to R2"
+              exit 1
+            fi
          fi

          export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
@@ -743,8 +822,8 @@ jobs:
            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
          fi

-          echo "release_id=$RELEASE_ID" >> $GITHUB_OUTPUT
-          echo "release_url=$RELEASE_URL" >> $GITHUB_OUTPUT
+          echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
+          echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
          echo "Created release: $RELEASE_URL"

  # Prepare and upload release assets
@@ -793,9 +872,9 @@ jobs:
          cd ./release-assets

          # Generate checksums for all files (including latest versions)
-          if ls *.zip >/dev/null 2>&1; then
-            sha256sum *.zip > SHA256SUMS
-            sha512sum *.zip > SHA512SUMS
+          if compgen -G "*.zip" >/dev/null; then
+            sha256sum -- *.zip > SHA256SUMS
+            sha512sum -- *.zip > SHA512SUMS
          fi

          cd ..
@@ -835,12 +914,14 @@ jobs:

          echo "✅ All assets uploaded successfully"

-  # Update latest.json for stable releases only: prerelease tags (alpha/beta/
-  # rc) must never overwrite the stable version pointer.
+  # Update latest.json for every release tag (stable and prerelease): the
+  # project currently ships prerelease tags only, so gating this to stable
+  # left the version pointer permanently stale. release_type records whether
+  # the pointed-to version is a prerelease.
  update-latest-version:
    name: Update Latest Version
    needs: [ build-check, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type == 'release'
+    if: startsWith(github.ref, 'refs/tags/')
    runs-on: ubuntu-latest
    steps:
      - name: Update latest.json
@@ -859,6 +940,12 @@ jobs:
          VERSION="${{ needs.build-check.outputs.version }}"
          TAG="${{ needs.build-check.outputs.version }}"

+          if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
+            RELEASE_TYPE="prerelease"
+          else
+            RELEASE_TYPE="stable"
+          fi
+
          # Install ossutil
          OSSUTIL_VERSION="2.1.1"
          OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
@@ -879,7 +966,7 @@ jobs:
            "version": "${VERSION}",
            "tag": "${TAG}",
            "release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
-            "release_type": "stable",
+            "release_type": "${RELEASE_TYPE}",
            "download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
          }
          EOF
@@ -887,7 +974,7 @@ jobs:
          # Upload to OSS
          "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force

-          echo "✅ Updated latest.json for stable release $VERSION"
+          echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"

  # Publish release (remove draft status)
  publish-release:
@@ -221,15 +221,29 @@ jobs:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          cache-save-if: ${{ github.ref == 'refs/heads/main' }}

-      # Three scanner tests fail on main independently of this lane (restore of
-      # a transitioned multipart object, noncurrent transition/expiry after an
-      # immediate compensation transition); they keep #[ignore] with a backlog
-      # reference and are excluded here by name until fixed (rustfs/backlog#1148).
+      # The #4877 restore self-deadlock is fixed in this PR, which re-enabled
+      # test_multipart_restore_preserves_parts_and_etag. The remaining exclusions
+      # each hit a DIFFERENT, independent issue (all tracked under
+      # rustfs/backlog#1148; they keep #[ignore] with a backlog reference):
+      #   - test_noncurrent_{expiry,transition}_still_works_after_immediate_compensation_transition:
+      #     noncurrent transition/expiry after an immediate compensation transition.
+      #   - test_transition_and_restore_flows: transition metadata is missing on
+      #     one drive (assert_transition_meta_consistent: "missing xl.meta ... on
+      #     disk2") - an EC metadata-distribution issue, not the restore lock.
+      #   - test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore:
+      #     DeleteRestoredAction sets opts.transition.expire_restored, but no
+      #     delete path reads that flag, so cleanup deletes the whole object
+      #     instead of only the local restored copy (ObjectNotFound afterwards).
+      #     The expire_restored delete semantics are unimplemented.
+      #   - restore_object_usecase_reports_ongoing_conflict_and_completion: asserts
+      #     a concurrent get_object_info observes ongoing-request=true mid-restore,
+      #     which #4877's read-vs-restore serialization rules out (see backlog#1148
+      #     ilm-8 criterion 1 - an API-semantics decision, not a bug).
      - name: Run ignored ILM integration tests serially
        run: |
          cargo nextest run -j1 --run-ignored ignored-only \
            -p rustfs-scanner -p rustfs \
-            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
+            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition) or test(restore_object_usecase_reports_ongoing_conflict_and_completion) or test(test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore))'

  test-and-lint-rio-v2:
    name: Test and Lint (rio-v2)
@@ -435,6 +449,16 @@ jobs:
      - name: Make binary executable
        run: chmod +x ./target/debug/rustfs

+      # Guard the security negative-auth smoke subset (backlog#1151 sec-5)
+      # against a rename or deletion silently dropping it out of the e2e-smoke
+      # filter. The script lists what the profile selects and fails if the count
+      # of security auth-rejection tests falls below the committed floor in
+      # .config/security-smoke-floor.txt (infra-12 count-floor mechanism). Run
+      # before the smoke suite so a thinned gate fails fast; the `nextest list`
+      # here compiles the e2e_test binaries the run below reuses.
+      - name: Check security smoke subset count floor
+        run: ./scripts/check_security_smoke_count.sh check
+
      # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
      # profile.e2e-smoke default-filter in .config/nextest.toml is the single
      # wiring mechanism for e2e tests in CI — extend that filter instead of
@@ -468,6 +492,59 @@ jobs:
          path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
          retention-days: 3

+  e2e-full:
+    name: End-to-End Tests (full merge gate)
+    # Merge gate only (backlog#1149 ci-5): the never-automated user-visible
+    # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
+    # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
+    # every PR, so it is gated to main pushes, the merge queue, and manual
+    # dispatch. protocols / the 6 cluster suites / replication / #[ignore] are
+    # owned by other lanes (see .config/nextest.toml profile.e2e-full).
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'merge_group' ||
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    needs: [ build-rustfs-debug-binary ]
+    runs-on: sm-standard-2
+    timeout-minutes: 55
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-e2e
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      # Download after the cache restore so the freshly built binary from the
+      # build job always wins over anything restored into target/debug.
+      - name: Download debug binary
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      # Full single-node e2e lane (backlog#1149 ci-5). The e2e-full
+      # default-filter in .config/nextest.toml is the single wiring mechanism —
+      # extend that filter, never add ad-hoc e2e jobs here. Reuses the downloaded
+      # debug binary; each test spawns its own rustfs server on a random port.
+      - name: Run e2e full suite
+        run: cargo nextest run --profile e2e-full -p e2e_test
+
+      - name: Upload junit
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: e2e-full-junit-${{ github.run_number }}
+          path: target/nextest/e2e-full/junit.xml
+          retention-days: 7
+
  e2e-tests-rio-v2:
    name: End-to-End Tests (rio-v2)
    needs: [ build-rustfs-debug-binary-rio-v2 ]
@@ -494,6 +571,14 @@ jobs:
      - name: Setup Rust toolchain for s3s-e2e installation
        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable

+      # The sm-standard-* custom runner images (introduced in #4884) ship no C
+      # toolchain, unlike GitHub-hosted ubuntu-latest. Installing s3s-e2e below
+      # compiles it from source on a cache miss, and build scripts need cc.
+      - name: Install build tools for s3s-e2e compilation
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential cmake pkg-config libssl-dev
+
      - name: Install s3s-e2e test tool
        uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
        with:
@@ -0,0 +1,122 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Weekly workspace line-coverage baseline (backlog#1153 infra-5).
+#
+# NON-BLOCKING by design: this workflow only runs on schedule and manual
+# dispatch, so it never attaches a status to a PR and must never be made a
+# required check. It exists to give coverage a visible baseline and trend
+# (per-crate table in the job summary, lcov artifact kept 90 days) — the
+# per-crate ratchet for the security-critical crates builds on it later
+# (backlog#1153 infra-6, report-only first per the ci-11 ladder).
+#
+# Measurement scope matches the PR test gate (ci.yml "Run tests"):
+# `--workspace --exclude e2e_test` with the `ci` nextest profile. Doctests are
+# NOT measured (ci.yml runs them uninstrumented; `cargo llvm-cov` needs a
+# nightly toolchain to cover doctests). Trend-comparison workflow:
+# docs/testing/README.md "Coverage" section. `make coverage` is the local
+# equivalent. Scheduled failures alert via the ci-8 composite action.
+
+name: coverage
+
+on:
+  workflow_dispatch:
+  schedule:
+    # 07:00 UTC Sunday — staggered clear of the other Sunday crons: ci (00:00),
+    # build (01:00), e2e-s3tests (02:00), audit (03:00), nix-flake-update
+    # (05:00), mint (06:00), and the daily fuzz (02:00), minio-interop (03:17),
+    # e2e-replication-nightly (04:00) and performance-ab (06:00) lanes.
+    - cron: "0 7 * * 0"
+
+# Only alert-on-failure needs more than read access; it declares its own
+# job-level `issues: write`.
+permissions:
+  contents: read
+
+jobs:
+  coverage:
+    name: Workspace coverage (weekly)
+    runs-on: sm-standard-4
+    # The instrumented build cannot reuse the regular CI cache (different
+    # RUSTFLAGS), so a cold week rebuilds the workspace before running the
+    # full suite; give it double the test job's 60-minute budget.
+    timeout-minutes: 120
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      # Match the PR gate's nextest semantics (ci.yml runs `--profile ci`):
+      # retries=0 plus the quarantine list and the ecstore-serial-flaky
+      # serialization. Set via env because `cargo llvm-cov`'s own --profile
+      # flag selects the *cargo build* profile, not the nextest profile.
+      NEXTEST_PROFILE: ci
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-coverage
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install cargo-llvm-cov
+        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
+        with:
+          tool: cargo-llvm-cov
+
+      - name: Install llvm-tools component
+        run: rustup component add llvm-tools-preview
+
+      - name: Run instrumented test suite
+        run: cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
+
+      - name: Generate lcov and JSON reports
+        run: |
+          mkdir -p target/llvm-cov
+          cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
+          cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
+
+      - name: Write per-crate summary
+        run: python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload coverage artifact
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: coverage-lcov-${{ github.run_number }}
+          path: |
+            target/llvm-cov/lcov.info
+            target/llvm-cov/coverage.json
+          retention-days: 90
+          if-no-files-found: ignore
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [coverage]
+    # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
+    # manual workflow_dispatch runs stay quiet so a debugging run never files a
+    # spurious alert.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -247,13 +247,15 @@ jobs:
            esac
          fi

-          echo "should_build=$should_build" >> $GITHUB_OUTPUT
-          echo "should_push=$should_push" >> $GITHUB_OUTPUT
-          echo "build_type=$build_type" >> $GITHUB_OUTPUT
-          echo "version=$version" >> $GITHUB_OUTPUT
-          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
-          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
-          echo "create_latest=$create_latest" >> $GITHUB_OUTPUT
+          {
+            echo "should_build=$should_build"
+            echo "should_push=$should_push"
+            echo "build_type=$build_type"
+            echo "version=$version"
+            echo "short_sha=$short_sha"
+            echo "is_prerelease=$is_prerelease"
+            echo "create_latest=$create_latest"
+          } >> "$GITHUB_OUTPUT"

          echo "🐳 Docker Build Summary:"
          echo "  - Should build: $should_build"
@@ -320,7 +322,6 @@ jobs:
        run: |
          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
          VERSION="${{ needs.build-check.outputs.version }}"
-          SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
          CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
          VARIANT_SUFFIX="${{ matrix.suffix }}"

@@ -343,8 +344,8 @@ jobs:
              ;;
          esac

-          echo "docker_release=$DOCKER_RELEASE" >> $GITHUB_OUTPUT
-          echo "docker_channel=$DOCKER_CHANNEL" >> $GITHUB_OUTPUT
+          echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
+          echo "docker_channel=$DOCKER_CHANNEL" >> "$GITHUB_OUTPUT"

          echo "🐳 Docker build parameters:"
          echo "  - Original version: $VERSION"
@@ -376,7 +377,7 @@ jobs:
          fi

          # Output tags
-          echo "tags=$TAGS" >> $GITHUB_OUTPUT
+          echo "tags=$TAGS" >> "$GITHUB_OUTPUT"

          # Generate labels
          LABELS="org.opencontainers.image.title=RustFS"
@@ -387,7 +388,7 @@ jobs:
          LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
          LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"

-          echo "labels=$LABELS" >> $GITHUB_OUTPUT
+          echo "labels=$LABELS" >> "$GITHUB_OUTPUT"

          echo "🐳 Generated Docker tags:"
          echo "$TAGS" | tr ',' '\n' | sed 's/^/  - /'
@@ -15,20 +15,24 @@
 # Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
 #
 # The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
-# FAST bucket-replication tests. This scheduled lane runs the remaining 18
+# FAST replication tests. This scheduled lane runs the remaining 27
 # heavier replication e2e tests that are unfit for a per-PR gate:
 #
-#   * 8 bucket-replication data-plane tests (PUT/delete + poll for convergence;
-#     two replicate over HTTPS).
-#   * 9 `_real_dual_node` site-replication tests (each spawns TWO rustfs
+#   * 2 remote-target TLS validation tests.
+#   * 12 bucket-replication data-plane/helper tests (PUT/delete + poll for
+#     convergence; two replicate over HTTPS, two pin active SSE failure
+#     contracts, and one guards event/history observers). The SSE-S3 contract
+#     remains ignored under backlog#1291.
+#   * 11 `_real_dual_node` site-replication tests (each spawns TWO rustfs
 #     servers and drives the cross-process site-replication control plane).
+#   * 1 `_real_three_node` site-replication test.
 #   * 1 `_real_single_node` service-account round-trip test.
 #
 # The selection is the [profile.e2e-repl-nightly] default-filter in
 # .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
 # add ad-hoc cargo-test steps here; change the filterset instead.
 #
-# Explicit division of labor: these 18 tests run ONLY here, never double-run
+# Explicit division of labor: these 27 tests run ONLY here, never double-run
 # in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
 # consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
 # into it rather than growing a second scheduled entrypoint.
@@ -80,7 +84,12 @@ jobs:
          python-version: "3.12"

      - name: Install awscurl
-        run: python3 -m pip install --user --upgrade pip awscurl
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl
+          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
+
+      - name: Verify awscurl
+        run: test -x "$AWSCURL_PATH"

      # Build the rustfs binary once up front. The e2e tests spawn it as a
      # child process (crates/e2e_test/src/common.rs) and will build it on
@@ -0,0 +1,70 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# MinIO on-disk interop: prove RustFS reads MinIO-written erasure-coded SSE
+# objects with byte-identical data and correct logical size.
+#
+# This is NOT a PR gate. The fixtures are real MinIO backend trees generated on
+# the fly (they are gitignored, never committed), so the job regenerates them
+# each run with Docker and then runs the `#[ignore]` reader tests in
+# crates/ecstore/tests/minio_generated_read_test.rs.
+#
+# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
+# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
+# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
+name: minio-interop
+
+on:
+  workflow_dispatch:
+  schedule:
+    # Nightly at 03:17 UTC (offset from other nightly jobs).
+    - cron: "17 3 * * *"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  minio-interop:
+    name: MinIO interop (EC + SSE read parity)
+    # Skip on forks: needs the repo's runners and is not a contributor gate.
+    if: github.repository == 'rustfs/rustfs'
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    env:
+      # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
+      RUSTFS_MINIO_STATIC_KMS_KEY_B64: IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g=
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-minio-interop
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Generate real MinIO fixtures via Docker
+        run: bash crates/rio-v2/tests/minio_fixture_lab/capture_via_docker.sh
+
+      - name: Run MinIO interop reader tests
+        run: |
+          cargo nextest run --run-ignored ignored-only \
+            -p rustfs-ecstore --features rio-v2 \
+            -E 'binary(minio_generated_read_test)'
@@ -135,6 +135,10 @@ jobs:
        run: |
          docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
          docker rm -f rustfs-mint >/dev/null 2>&1 || true
+          # The four disks share one physical device on the runner (a single
+          # loopback filesystem), so the local physical-disk-independence guard
+          # would refuse to start. Bypass it — this is the CI use case the guard
+          # explicitly sanctions via RUSTFS_UNSAFE_BYPASS_DISK_CHECK.
          docker run -d --name rustfs-mint \
            --network rustfs-net \
            -p 9000:9000 \
@@ -142,6 +146,7 @@ jobs:
            -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
            -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
            -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
+            -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
            -v /tmp/rustfs-mint:/data \
            rustfs-ci

@@ -41,6 +41,10 @@ on:
        type: boolean
  pull_request:
    types: [labeled, synchronize, reopened]
+  push:
+    # Every main commit pre-builds and caches its release binary (perf-3) so the
+    # nightly A/B restores a ready baseline instead of paying the double build.
+    branches: [main]

 permissions:
  contents: read
@@ -58,22 +62,81 @@ env:
  RUST_BACKTRACE: 1

 jobs:
+  # perf-3: on every push to main, build the release binary once and cache it
+  # keyed by commit SHA (rustfs-baseline-<sha>). The nightly A/B (and, later, the
+  # perf-7 PR gate) restore this instead of paying the ~32min-per-side source
+  # build. That double build is what pushed the expanded 24-cell nightly past its
+  # ceiling — 2026-07-11..07-14 all cancelled on the 120min timeout. Incremental
+  # builds off the shared cargo cache keep each push cheap, and building on the
+  # same sm-standard-2 runner the A/B measures on guarantees the cached binary is
+  # ABI-identical. Do NOT source this from build.yml's per-merge artifact: those
+  # are cancelled ~7/8 of the time and are not a reliable baseline.
+  build-baseline-cache:
+    name: Build + cache baseline binary
+    if: github.event_name == 'push'
+    runs-on: sm-standard-2
+    # Latest-wins: consumers only ever restore the binary for the *current*
+    # origin/main tip, so when pushes land faster than the ~65min build, a
+    # superseded build's output is dead weight — cancel it instead of stacking
+    # hour-long jobs on the shared runner pool. A skipped intermediate SHA at
+    # most costs one same-commit self-heal in the A/B job.
+    concurrency:
+      group: perf-baseline-build-main
+      cancel-in-progress: true
+    # #4806 put thin LTO + codegen-units=1 on [profile.release], pushing a
+    # single release build past 60min on this runner — every cache build on
+    # 2026-07-15 died on the old 60min ceiling ("exceeded the maximum execution
+    # time of 1h0m0s") and the cache never populated. The measured binary must
+    # keep the production profile, so the budget absorbs the build instead.
+    timeout-minutes: 100
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build release rustfs
+        run: cargo build --release --bin rustfs
+
+      - name: Stage binary for cache
+        run: |
+          set -euo pipefail
+          mkdir -p baseline-bin
+          cp target/release/rustfs baseline-bin/rustfs
+
+      - name: Cache baseline binary by SHA
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
+        with:
+          path: baseline-bin/rustfs
+          key: rustfs-baseline-${{ github.sha }}
+
  warp-ab:
    name: Warp A/B budget gate
-    # Opt-in on PRs: only run when the `perf-ab` label is present, and for
-    # `labeled` events only when the label being added is `perf-ab` itself —
-    # adding an unrelated label to an opted-in PR must not re-run the gate.
-    # Always run on schedule / manual dispatch.
+    # Always run on schedule / manual dispatch. Opt-in on PRs: only when the
+    # `perf-ab` label is present, and for `labeled` events only when the label
+    # being added is `perf-ab` itself (adding an unrelated label to an opted-in
+    # PR must not re-run the gate). Never on push — that event only feeds
+    # build-baseline-cache above.
    if: >-
-      github.event_name != 'pull_request' ||
-      (contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
+      github.event_name == 'schedule' ||
+      github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'pull_request' &&
+       contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
       (github.event.action != 'labeled' || github.event.label.name == 'perf-ab'))
    runs-on: sm-standard-2
-    # Phase-0 stopgap: the baseline+candidate release double-build alone is
-    # ~65min on this runner, so 90min left no room for a real full-matrix
-    # measurement (the earlier nightly runs only ever failed *before*
-    # measuring). 120min gives the 24-cell short matrix headroom until perf-3
-    # caches the baseline binary and restores a tighter budget.
+    # With perf-3's cached baseline binary the common (cache-hit) nightly is
+    # measurement-only and finishes well under 50min. This ceiling stays
+    # generous only to absorb the same-commit cache-miss self-heal (~65min
+    # single build with the post-#4806 LTO profile + measurement). A timeout
+    # surfaces via the alert-on-failure job (it fires on cancelled/timed-out,
+    # not just failure). perf-6 recalibrates the budget once the noise study
+    # lands.
    timeout-minutes: 120
    steps:
      - name: Checkout repository
@@ -110,21 +173,106 @@ jobs:
          fi
          echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"

+      # perf-3: resolve the commits so the cache can be keyed by SHA. The
+      # baseline is origin/main; the candidate is the checked-out ref. On the
+      # nightly (checkout == main) they are the same commit, so one cached binary
+      # serves both phases and the run does zero source builds.
+      - name: Resolve baseline / candidate commits
+        id: commits
+        run: |
+          set -euo pipefail
+          baseline_sha="$(git rev-parse origin/main)"
+          candidate_sha="$(git rev-parse HEAD)"
+          echo "baseline_sha=$baseline_sha" >> "$GITHUB_OUTPUT"
+          echo "candidate_sha=$candidate_sha" >> "$GITHUB_OUTPUT"
+          echo "baseline commit: $baseline_sha"
+          echo "candidate commit: $candidate_sha"
+
+      # Exact-key restore of the baseline binary built by build-baseline-cache
+      # when origin/main last landed. A miss (binary evicted or not built yet)
+      # leaves cache-hit unset and the rig falls back to a source build.
+      - name: Restore cached baseline binary
+        id: baseline_cache
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
+        with:
+          path: baseline-bin/rustfs
+          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
+
+      # Self-heal: on a nightly/dispatch run where the candidate commit IS the
+      # baseline commit, a cache miss would make the rig build the same commit
+      # twice (~65min per side with the post-#4806 LTO profile — no job budget
+      # fits that). Build it once here, reuse it for both phases, and save it
+      # back to the cache so the next run hits.
+      - name: Build baseline on cache miss (same-commit self-heal)
+        id: selfheal
+        if: >-
+          steps.baseline_cache.outputs.cache-hit != 'true' &&
+          steps.commits.outputs.baseline_sha == steps.commits.outputs.candidate_sha
+        run: |
+          set -euo pipefail
+          cargo build --release --bin rustfs
+          mkdir -p baseline-bin
+          cp target/release/rustfs baseline-bin/rustfs
+          echo "built=true" >> "$GITHUB_OUTPUT"
+
+      - name: Save self-healed baseline to cache
+        if: steps.selfheal.outputs.built == 'true'
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
+        with:
+          path: baseline-bin/rustfs
+          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
+
      - name: Run warp A/B and gate
        id: ab
        run: |
          set -euo pipefail
-          # Budget note: the baseline+candidate release double-build (~65 min,
-          # cached away later by perf-3) dominates the 90-min job, so the
-          # measurement runs a short warp matrix — duration/rounds/cooldown are
-          # kept small to fit all 24 cells (6 workloads x 2 phases x 2 drive-sync)
-          # under budget rather than dropping cells. --health-timeout 180 outlasts
-          # the server's own 120s startup-readiness budget, which is what the
-          # rig's previous 60s health poll undershot (the first two nightly
-          # failures). perf-6 will recalibrate these once the pipeline is green.
+          # Budget note: with perf-3's cached baseline the nightly does no source
+          # build on a cache hit, so the wall-clock is dominated by the short warp
+          # matrix — duration/rounds/cooldown are kept small to fit all 24 cells
+          # (6 workloads x 2 phases x 2 drive-sync) rather than dropping cells.
+          # --health-timeout 180 outlasts the server's own 120s startup-readiness
+          # budget, which the rig's previous 60s health poll undershot (the first
+          # two nightly failures). perf-6 recalibrates these once the noise study
+          # lands.
          duration="${{ github.event.inputs.duration || '12s' }}"
-          args=(--baseline-ref origin/main
-            --duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
+          baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
+          candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
+          baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
+          selfheal_built="${{ steps.selfheal.outputs.built }}"
+
+          args=(--duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
+
+          if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" ]]; then
+            chmod +x baseline-bin/rustfs
+            base_bin="$PWD/baseline-bin/rustfs"
+            args+=(--baseline-bin "$base_bin")
+            if [[ "$baseline_hit" == "true" ]]; then
+              base_src="actions-cache (rustfs-baseline-$baseline_sha)"
+            else
+              base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
+            fi
+            if [[ "$candidate_sha" == "$baseline_sha" ]]; then
+              # Nightly on main: the candidate is the same commit as the baseline,
+              # so reuse the one binary for both phases and skip all builds.
+              args+=(--candidate-bin "$base_bin" --skip-build)
+              cand_src="same binary as baseline (same commit)"
+            else
+              cand_src="source build of the checked-out ref"
+            fi
+          else
+            # Cache miss with candidate != baseline (opt-in PR gate only): fall
+            # back to the source double-build. With the post-#4806 LTO profile
+            # this will overrun the job budget and alert; rerun once the push
+            # cache build for origin/main has completed, or wait for perf-7's
+            # merge-base caching.
+            args+=(--baseline-ref origin/main)
+            base_src="source build of origin/main (cache miss)"
+            cand_src="source build of the checked-out ref"
+          fi
+
+          args+=(--provenance-note "baseline commit: $baseline_sha - $base_src")
+          args+=(--provenance-note "candidate commit: $candidate_sha - $cand_src")
+
          if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
            args+=(--allow-regression --exemption-reason "labeled perf-deliberate-tradeoff / dispatch override")
          fi
@@ -201,12 +349,8 @@ jobs:
        run: |
          gh pr comment "${{ github.event.pull_request.number }}" --body-file "${{ steps.ab.outputs.gate_md }}"

-      # TODO(ci-8/perf-2): scheduled/dispatch failures are currently silent. Once
-      # ci-8 lands the .github/actions/schedule-failure-issue composite action,
-      # perf-2 adds a step here guarded by
-      #   if: failure() && github.event_name != 'pull_request'
-      # that calls it (label perf-nightly-failure, append to an existing open
-      # issue) instead of hand-rolling gh CLI dedup. Do not implement it here.
+      # Scheduled failure alerting is handled by the alert-on-failure job below
+      # (perf-2 consuming ci-8's schedule-failure-issue composite action).

      - name: Enforce gate
        if: always()
@@ -224,7 +368,16 @@ jobs:
    # `always()` is required: without it this job is skipped when a needed
    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
    # ci-8); PR and manual dispatch failures are already watched by a human.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    # `cancelled` is included alongside `failure` on purpose: a job that hits
+    # timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
+    # timeouts went silent precisely because the guard was failure-only. The
+    # composite action already reports cancelled/timed-out jobs in the issue
+    # body. (Scheduled runs get a unique concurrency group with
+    # cancel-in-progress off, so a cancellation here means a timeout/manual
+    # abort, never a superseding run.)
+    if: >-
+      always() && github.event_name == 'schedule' &&
+      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
@@ -11,6 +11,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.

 ### Added
+- **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
+  - Three configuration keys per target: `JETSTREAM_ENABLE`, `JETSTREAM_STREAM_NAME`, and `JETSTREAM_ACK_TIMEOUT_SECS`, under the `RUSTFS_NOTIFY_NATS_` and `RUSTFS_AUDIT_NATS_` prefixes
+  - Durable store-and-forward with a stable dedup id sent as the `Nats-Msg-Id` header, so a replay after a crash is collapsed by the server duplicate window
+  - Pre-flight stream validation, and a bounded failed-events store (count and TTL). Only a non-retryable rejection is recorded in the failed-events store. A retryable condition keeps the entry on the live queue until it is delivered
+  - Operator guide at `docs/operations/nats-jetstream.md`
 - **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
  - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
  - Task-local storage for async-safe credential passing between middleware and auth handlers
@@ -31,6 +31,8 @@ make build-docker BUILD_OS=ubuntu22.04
 - Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md)
 - Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md)
 - CI gates: `.github/workflows/ci.yml` (source of truth; never copy its steps into docs)
+- Test-layer taxonomy, per-layer entry commands, serial/nextest rules, flake
+  policy: [docs/testing/README.md](docs/testing/README.md)
 - Tier/ILM transition debugging (xl.meta inspection, versionId tracing):
  [docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md)

@@ -68,6 +68,8 @@ make pre-pr

 > `make test` requires [cargo-nextest](https://nexte.st) (CI runs it and only nextest honours `.config/nextest.toml` test-groups). Install it with `cargo install cargo-nextest --locked` or a prebuilt binary (see https://nexte.st/docs/installation/). To run the plain `cargo test` fallback anyway (results not authoritative — serialization semantics differ from CI), set `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1`.

+> For the full test-layer taxonomy (unit / ecstore black-box / e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench), each layer's entry command, the naming conventions the migration gate depends on, and the serial/nextest rules, see [docs/testing/README.md](docs/testing/README.md).
+
 ### 🔒 Automated Pre-commit Hooks
 #### What `make pre-commit` and `make pre-pr` actually run

@@ -52,6 +52,7 @@ members = [
    "crates/extension-schema", # Extension schema contracts
    "crates/signer", # client signer
    "crates/storage-api", # Storage API contracts
+    "crates/test-utils", # Shared test bootstrap helpers (dev-dependency only)
    "crates/targets", # Target-specific configurations and utilities
    "crates/trusted-proxies", # Trusted proxies management
    "crates/tls-runtime", # Project-wide TLS runtime foundation
@@ -67,7 +68,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.96.0"
-version = "1.0.0-beta.8"
+version = "1.0.0-beta.10"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -84,55 +85,56 @@ redundant_clone = "warn"

 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.8" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.8" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.8" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.8" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.8" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.8" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.8" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.8" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.8" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.8" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.8" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.8" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.8" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.8" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.8" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.8" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.8" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.8" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.8" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.8" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.8" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.8" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.8" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.8" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.8" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.8" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.8" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.8" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.8" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.8" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.8" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.8" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.8" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.8" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.8" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.8" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.8" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.8" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.8" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.8" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.8" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.8" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.8" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.8" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.10" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.10" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.10" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.10" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.10" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.10" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.10" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.10" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.10" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.10" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.10" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.10" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.10" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.10" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.10" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.10" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.10" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.10" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.10" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.10" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.10" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.10" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.10" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.10" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.10" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.10" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.10" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.10" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.10" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.10" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.10" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.10" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.10" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.10" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.10" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.10" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.10" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.10" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.10" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.10" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.10" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.10" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.10" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.10" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.10" }

 # Async Runtime and Networking
 async-channel = "2.5.0"
-async_zip = { version = "0.0.18", default-features = false, features = ["tokio", "deflate"] }
-mysql_async = { version = "0.37", default-features = false, features = ["default-rustls", "tracing"] }
+async_zip = { default-features = false, version = "0.0.18" }
+mysql_async = { default-features = false, version = "0.37" }
 async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
 async-trait = "0.1.89"
@@ -143,32 +145,32 @@ futures-core = "0.3.32"
 futures-lite = "2.6.1"
 futures-util = "0.3.32"
 pollster = "1.0.1"
-pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime", "telemetry"] }
-lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
-hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
-hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
-hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
+pulsar = { default-features = false, version = "6.8.0" }
+lapin = { default-features = false, version = "4.10.0" }
+hyper = { version = "1.10.1" }
+hyper-rustls = { default-features = false, version = "0.27.9" }
+hyper-util = { version = "0.1.20" }
 http = "1.4.2"
-http-body = "1.0.1"
-http-body-util = "0.1.3"
+http-body = "1.1.0"
+http-body-util = "0.1.4"
 minlz = "1.2.3"
-reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
+reqwest = { default-features = false, version = "0.13.4" }
 rustfs-kafka-async = { version = "1.2.0" }
-socket2 = { version = "0.6.4", features = ["all"] }
-tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
-tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
+socket2 = { version = "0.6.5" }
+tokio = { version = "1.52.3" }
+tokio-rustls = { default-features = false, version = "0.26.4" }
 tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
-tokio-util = { version = "0.7.18", features = ["io", "compat"] }
-tonic = { version = "0.14.6", features = ["gzip", "deflate"] }
+tokio-util = { version = "0.7.18" }
+tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
-tower = { version = "0.5.3", features = ["timeout"] }
-tower-http = { version = "0.7.0", features = ["cors"] }
+tower = { version = "0.5.3" }
+tower-http = { version = "0.7.0" }

 # Serialization and Data Formats
 apache-avro = "0.21.0"
-bytes = { version = "1.12.1", features = ["serde"] }
+bytes = { version = "1.12.1" }
 bytesize = "2.4.2"
 byteorder = "1.5.0"
 flatbuffers = "25.12.19"
@@ -177,8 +179,8 @@ prost = "0.14.4"
 quick-xml = "0.41.0"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
-serde = { version = "1.0.228", features = ["derive"] }
-serde_json = { version = "1.0.150", features = ["raw_value"] }
+serde = { version = "1.0.228" }
+serde_json = { version = "1.0.150" }
 serde_urlencoded = "0.7.1"

 # Cryptography and Security
@@ -186,33 +188,33 @@ serde_urlencoded = "0.7.1"
 # matching stable releases are not available yet, while previous stable lines
 # have incompatible APIs. Keep them exact-pinned and monitor upstream for stable
 # releases.
-aes-gcm = { version = "=0.11.0", features = ["rand_core"] }
+aes-gcm = { version = "=0.11.0" }
 argon2 = { version = "=0.6.0-rc.8" }
 blake2 = "=0.11.0-rc.6"
 chacha20poly1305 = { version = "=0.11.0" }
 crc-fast = "1.10.0"
 hmac = { version = "0.13.0" }
-jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
-openidconnect = { version = "4.0", default-features = false, features = ["accept-rfc3339-timestamps"] }
+jsonwebtoken = { version = "10.4.0" }
+openidconnect = { default-features = false, version = "4.0" }
 pbkdf2 = "0.13.0"
 rsa = { version = "=0.10.0-rc.18" }
-rustls = { version = "0.23.41", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
+rustls = { default-features = false, version = "0.23.42" }
 rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.0"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
 subtle = "2.6"
-zeroize = { version = "1.9.0", features = ["derive"] }
+zeroize = { version = "1.9.0" }

 # Time and Date
-chrono = { version = "0.4.45", features = ["serde"] }
+chrono = { version = "0.4.45" }
 humantime = "2.4.0"
-jiff = { version = "0.2.32", features = ["serde"] }
-time = { version = "0.3.53", features = ["std", "parsing", "formatting", "macros", "serde"] }
+jiff = { version = "0.2.32" }
+time = { version = "0.3.53" }

 # Database
-deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
-tokio-postgres = { version = "0.7.18", default-features = false, features = ["runtime", "with-serde_json-1"] }
+deadpool-postgres = { version = "0.14" }
+tokio-postgres = { default-features = false, version = "0.7.18" }
 tokio-postgres-rustls = "0.14.0"

 # Utilities and Tools
@@ -223,34 +225,34 @@ atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.9.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-s3 = { version = "1.138.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-http-client = { version = "1.2.0", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-runtime-api = { version = "1.13.0", features = ["http-1x"] }
+aws-sdk-s3 = { default-features = false, version = "1.138.0" }
+aws-smithy-http-client = { default-features = false, version = "1.2.0" }
+aws-smithy-runtime-api = { version = "1.13.0" }
 aws-smithy-types = { version = "1.6.1" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.1", features = ["derive", "env"] }
-const-str = { version = "1.1.0", features = ["std", "proc"] }
+clap = { version = "4.6.2" }
+const-str = { version = "1.1.0" }
 convert_case = "0.11.0"
-criterion = { version = "0.8", features = ["html_reports"] }
+criterion = { version = "0.8" }
 crossbeam-queue = "0.3.13"
 crossbeam-channel = "0.5.16"
 crossbeam-deque = "0.8.7"
 crossbeam-utils = "0.8.22"
-datafusion = { git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
+datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
 derive_builder = "0.20.2"
 enumset = "1.1.13"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
 glob = "0.3.3"
-google-cloud-storage = "1.15.0"
-google-cloud-auth = "1.13.0"
-hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
+google-cloud-storage = "1.16.0"
+google-cloud-auth = "1.14.0"
+hashbrown = { version = "0.17.1" }
 hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
-ipnetwork = { version = "0.21.1", features = ["serde"] }
+ipnetwork = { version = "0.21.1" }
 lazy_static = "1.5.0"
 libc = "0.2.186"
 libsystemd = "0.7.2"
@@ -261,7 +263,7 @@ matchit = "0.9.2"
 md-5 = "0.11.0"
 md5 = "0.8.1"
 mime_guess = "2.0.5"
-moka = { version = "0.12.15", features = ["future"] }
+moka = { version = "0.12.15" }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.1"
@@ -271,27 +273,26 @@ path-clean = "1.0.1"
 percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
-rand = { version = "0.10.2", features = ["serde"] }
+rand = { version = "0.10.2" }
 ratelimit = "0.10.1"
 rayon = "1.12.0"
-reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "7.0.1", features = ["simd-accel"] }
-#reed-solomon-erasure = { version = "6.0", features = ["simd-accel"], git = "https://github.com/houseme/reed-solomon-erasure",rev = "main" }
+reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.0" }
 reed-solomon-simd = "3.1.0"
-regex = { version = "1.13.0" }
-rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
-redis = { version = "1.3.0", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
-rustix = { version = "1.1.4", features = ["fs"] }
+regex = { version = "1.13.1" }
+rumqttc = { package = "rumqttc-next", version = "0.33.2" }
+redis = { version = "1.4.0" }
+rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { version = "0.14.1", features = ["minio"] }
+s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "ce69c3f10824535c7c24b2f71cdb2aaa4dffb5e0" }
 serial_test = "3.5.0"
-shadow-rs = { version = "2.0.0", default-features = false }
+shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
-smallvec = { version = "1.15.2", features = ["serde"] }
+smallvec = { version = "1.15.2" }
 smartstring = "1.0.1"
-snap = "1.1.1"
-starshard = { version = "2.2.1", features = ["rayon", "async", "serde"] }
-strum = { version = "0.28.0", features = ["derive"] }
+snap = "1.1.2"
+starshard = { version = "2.2.2" }
+strum = { version = "0.28.0" }
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
@@ -301,15 +302,15 @@ tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-error = "0.2.1"
 tracing-opentelemetry = { version = "0.33" }
-tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
+tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.23.5", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.24.0" }
 vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
 windows = { version = "0.62.2" }
-xxhash-rust = { version = "0.8.16", features = ["xxh64", "xxh3"] }
+xxhash-rust = { version = "0.8.17" }
 zip = "8.6.0"
 zstd = "0.13.3"

@@ -317,19 +318,19 @@ zstd = "0.13.3"
 metrics = "0.24.6"
 dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.32.0" }
-opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
-opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
-opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
-opentelemetry-semantic-conventions = { version = "0.32.1", features = ["semconv_experimental"] }
+opentelemetry-appender-tracing = { version = "0.32.0" }
+opentelemetry-otlp = { version = "0.32.0" }
+opentelemetry_sdk = { version = "0.32.1" }
+opentelemetry-semantic-conventions = { version = "0.32.1" }
 opentelemetry-stdout = { version = "0.32.0" }
-pyroscope = { version = "2.1.0", features = ["backend-pprof-rs"] }
+pyroscope = { version = "2.1.0" }

 # FTP and SFTP
-libunftp = { version = "0.23.0", features = ["experimental"] }
+libunftp = { version = "0.23.0" }
 unftp-core = "0.1.0"
-suppaftp = { version = "10.0.0", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+suppaftp = { version = "10.0.1" }
 rcgen = "0.14.8"
-russh = { version = "0.62.2", features = ["serde"] }
+russh = { version = "0.62.2" }
 russh-sftp = "2.3.0"

 # WebDAV
@@ -339,7 +340,7 @@ dav-server = "0.11.0"
 mimalloc = "0.1"
 hotpath = "0.21"
 # Snapshot testing for output format regression detection
-insta = { version = "1.48", features = ["yaml", "json"] }
+insta = { version = "1.48" }

 [workspace.metadata.cargo-shear]
 ignored = ["rustfs"]
@@ -353,12 +354,34 @@ debug = "line-tables-only"

 [profile.release]
 opt-level = 3
+lto = "thin"
+codegen-units = 1
+debug = 0
+split-debuginfo = "off"
+strip = "symbols"

 [profile.production]
 inherits = "release"
-lto = "fat"
-codegen-units = 1

 [profile.profiling]
 inherits = "release"
 debug = true
+strip = "none"
+
+# Pin hyper to a revision that carries the HTTP/1 "flush buffered data before
+# shutdown" fix (hyperium/hyper#4018, commit 72046cc7). This lands as a
+# `[patch.crates-io]` entry — not on the `hyper` workspace dependency — so that
+# every consumer in the tree, including the transitive `hyper-util` server path
+# (`conn::auto` / `GracefulShutdown`) that actually drives our connections,
+# resolves to the fixed hyper rather than the buggy crates.io copy.
+#
+# hyper <= 1.10.1 can call `poll_shutdown()` on the socket while response bytes
+# are still buffered (a prior `poll_flush()` returned `Poll::Pending` and the
+# result was discarded). A backpressured / slow-reading peer then receives a
+# graceful FIN before the full Content-Length body is flushed, which standard S3
+# clients (minio-go / warp) report as `unexpected EOF` on large-object GET under
+# load. The fix is not in any crates.io release yet as of hyper 1.10.1; drop
+# this patch once a released version (> 1.10.1) contains commit 72046cc7.
+# See rustfs/backlog#1232.
+[patch.crates-io]
+hyper = { git = "https://github.com/hyperium/hyper.git", rev = "ccc1e850dc0cda3e71b0acd11f60ca3d48d09034" }
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-FROM alpine:3.23.4 AS build
+FROM alpine:3.24.1 AS build

 ARG TARGETARCH
 ARG RELEASE=latest
@@ -70,7 +70,7 @@ RUN set -eux; \
    rm -rf rustfs.zip /build/.tmp || true


-FROM alpine:3.23.4
+FROM alpine:3.24.1

 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -88,7 +88,9 @@ LABEL name="RustFS" \
      url="https://rustfs.com" \
      license="Apache-2.0"

-RUN apk update && \
+# Upgrade base-image packages so published images pick up security fixes
+# (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
+RUN apk upgrade --no-cache && \
    apk add --no-cache ca-certificates coreutils curl

 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-FROM rust:1.95-trixie
+FROM rust:1.97-trixie

 RUN set -eux; \
    export DEBIAN_FRONTEND=noninteractive; \
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-FROM ubuntu:24.04 AS build
+FROM ubuntu:26.04 AS build

 ARG TARGETARCH
 ARG RELEASE=latest
@@ -76,7 +76,7 @@ RUN set -eux; \
    chmod +x /build/rustfs; \
    rm -rf rustfs.zip /build/.tmp || true

-FROM ubuntu:24.04
+FROM ubuntu:26.04

 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -93,7 +93,10 @@ LABEL name="RustFS" \
      url="https://rustfs.com" \
      license="Apache-2.0"

-RUN apt-get update && apt-get install -y --no-install-recommends \
+# Upgrade base-image packages so published images pick up security fixes
+# (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
+RUN apt-get update && apt-get upgrade -y \
+    && apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
    && rm -rf /var/lib/apt/lists/*
@@ -32,7 +32,7 @@ ARG RUSTFS_BUILD_FEATURES=""
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.95-trixie AS builder
+FROM rust:1.97-trixie AS builder

 # Re-declare args after FROM
 ARG TARGETPLATFORM
@@ -208,7 +208,7 @@ CMD ["cargo", "run", "--bin", "rustfs", "--"]
 # -----------------------------
 # Runtime stage (Ubuntu minimal)
 # -----------------------------
-FROM ubuntu:24.04
+FROM ubuntu:26.04

 ARG BUILD_DATE
 ARG VCS_REF
@@ -223,6 +223,7 @@ LABEL name="RustFS (dev-local)" \
 RUN set -eux; \
    export DEBIAN_FRONTEND=noninteractive; \
    apt-get update; \
+    apt-get upgrade -y; \
    apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10
 ```

 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -218,7 +218,10 @@ For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status,
 and single-node single-disk idle CPU tuning, see
 [Scanner Runtime Controls](docs/operations/scanner-runtime-controls.md). For
 repeatable scanner-pressure validation, see
-[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md).
+[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md). For
+drive timeout knobs on slow storage — including the walk stall budget that
+governs `ListObjects` on large prefixes — see
+[Drive Timeout Tuning](docs/operations/drive-timeout-tuning.md).

 ### 5\. Nix Flake (Option 5)

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10
 ```

 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
@@ -217,7 +217,7 @@ setup_rust_environment() {
    # Set up environment variables for musl targets
    if [[ "$PLATFORM" == *"musl"* ]]; then
        print_message $YELLOW "Setting up environment for musl target..."
-        export RUSTFLAGS="--cfg tokio_unstable -C target-feature=-crt-static"
+        export RUSTFLAGS="${RUSTFLAGS:+$RUSTFLAGS }-C target-feature=-crt-static"

        # For cargo-zigbuild, set up additional environment variables
        if command -v cargo-zigbuild &> /dev/null; then
@@ -434,7 +434,7 @@ build_binary() {
        fi
    else
        # Native compilation
-        build_cmd="RUSTFLAGS='--cfg tokio_unstable -Clink-arg=-lm' cargo build"
+        build_cmd="RUSTFLAGS='${RUSTFLAGS:+$RUSTFLAGS }-Clink-arg=-lm' cargo build"
    fi

    if [ "$BUILD_TYPE" = "release" ]; then
@@ -27,17 +27,17 @@ categories = ["web-programming", "development-tools", "asynchronous", "api-bindi

 [dependencies]
 rustfs-targets = { workspace = true }
-rustfs-config = { workspace = true, features = ["audit", "constants", "server-config-model"] }
+rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
 rustfs-s3-types = { workspace = true }
-chrono = { workspace = true }
-const-str = { workspace = true }
+chrono = { workspace = true, features = ["serde"] }
+const-str = { workspace = true, features = ["std", "proc"] }
 futures = { workspace = true }
-hashbrown = { workspace = true }
+hashbrown = { workspace = true, features = ["serde", "rayon"] }
 metrics = { workspace = true }
-serde = { workspace = true }
-serde_json = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
+serde_json = { workspace = true, features = ["raw_value"] }
 thiserror = { workspace = true }
-tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
+tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "time", "macros"] }
 tracing = { workspace = true, features = ["std", "attributes"] }

 [dev-dependencies]
@@ -32,6 +32,7 @@ const EVENT_AUDIT_BATCH_DISPATCH_COMPLETED: &str = "audit_batch_dispatch_complet
 const EVENT_AUDIT_TARGET_STATE_CHANGED: &str = "audit_target_state_changed";
 const EVENT_AUDIT_REPLAY_DELIVERED: &str = "audit_replay_delivered";
 const EVENT_AUDIT_REPLAY_RETRY_SCHEDULED: &str = "audit_replay_retry_scheduled";
+const EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED: &str = "audit_replay_retry_exhausted";
 const EVENT_AUDIT_REPLAY_DROPPED: &str = "audit_replay_dropped";
 const EVENT_AUDIT_REPLAY_STREAM_STATUS: &str = "audit_replay_stream_status";

@@ -281,6 +282,7 @@ impl AuditPipeline {
                let delivery = target.delivery_snapshot();
                AuditTargetMetricSnapshot {
                    failed_messages: delivery.failed_messages,
+                    failed_store_length: delivery.failed_store_length,
                    queue_length: delivery.queue_length,
                    target_id: target.id().to_string(),
                    total_messages: delivery.total_messages,
@@ -463,18 +465,16 @@ impl AuditRuntimeFacade {
                            target.record_final_failure();
                            observability::record_target_failure();
                        }
-                        ReplayEvent::RetryExhausted { key, target } => {
+                        ReplayEvent::RetryExhausted { detail, key, target } => {
                            warn!(
-                                event = EVENT_AUDIT_REPLAY_DROPPED,
+                                event = EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED,
                                component = LOG_COMPONENT_AUDIT,
                                subsystem = LOG_SUBSYSTEM_PIPELINE,
                                target_id = %target.id(),
                                replay_key = %key,
-                                reason = "retry_exhausted",
-                                "audit replay delivery"
+                                error = %detail,
+                                "audit replay retry budget exhausted, entry stays queued and retries"
                            );
-                            target.record_final_failure();
-                            observability::record_target_failure();
                        }
                        ReplayEvent::UnreadableEntry { key, error, target } => {
                            warn!(
@@ -30,6 +30,7 @@ const EVENT_AUDIT_CONFIG_RELOADED: &str = "audit_config_reloaded";
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
 pub struct AuditTargetMetricSnapshot {
    pub failed_messages: u64,
+    pub failed_store_length: u64,
    pub queue_length: u64,
    pub target_id: String,
    pub total_messages: u64,
@@ -26,13 +26,14 @@ categories = ["web-programming", "development-tools", "network-programming"]
 documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"

 [dependencies]
-bytes = { workspace = true }
+bytes = { workspace = true, features = ["serde"] }
 crc-fast = { workspace = true }
 http = { workspace = true }
 base64-simd = { workspace = true }
 md-5 = { workspace = true }
 sha1 = { workspace = true }
 sha2 = { workspace = true }
+xxhash-rust = { workspace = true, features = ["xxh64", "xxh3"] }

 [dev-dependencies]
 pretty_assertions = { workspace = true }
@@ -36,7 +36,7 @@ impl fmt::Display for UnknownChecksumAlgorithmError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(
            f,
-            r#"unknown checksum algorithm "{}", please pass a known algorithm name ("crc32", "crc32c", "crc64nvme", "sha1", "sha256")"#,
+            r#"unknown checksum algorithm "{}", please pass a known algorithm name ("crc32", "crc32c", "crc64nvme", "sha1", "sha256", "sha512", "xxhash3", "xxhash64", "xxhash128")"#,
            self.checksum_algorithm
        )
    }
@@ -16,13 +16,20 @@ use crate::base64;
 use http::header::{HeaderMap, HeaderValue};

 use crate::Crc64Nvme;
-use crate::{CRC_32_C_NAME, CRC_32_NAME, CRC_64_NVME_NAME, Checksum, Crc32, Crc32c, Md5, SHA_1_NAME, SHA_256_NAME, Sha1, Sha256};
+use crate::{
+    CRC_32_C_NAME, CRC_32_NAME, CRC_64_NVME_NAME, Checksum, Crc32, Crc32c, Md5, SHA_1_NAME, SHA_256_NAME, Sha1, Sha256, Sha512,
+    Xxhash3, Xxhash64, Xxhash128,
+};

 pub const CRC_32_HEADER_NAME: &str = "x-amz-checksum-crc32";
 pub const CRC_32_C_HEADER_NAME: &str = "x-amz-checksum-crc32c";
 pub const SHA_1_HEADER_NAME: &str = "x-amz-checksum-sha1";
 pub const SHA_256_HEADER_NAME: &str = "x-amz-checksum-sha256";
 pub const CRC_64_NVME_HEADER_NAME: &str = "x-amz-checksum-crc64nvme";
+pub const SHA_512_HEADER_NAME: &str = "x-amz-checksum-sha512";
+pub const XXHASH_3_HEADER_NAME: &str = "x-amz-checksum-xxhash3";
+pub const XXHASH_64_HEADER_NAME: &str = "x-amz-checksum-xxhash64";
+pub const XXHASH_128_HEADER_NAME: &str = "x-amz-checksum-xxhash128";

 #[allow(dead_code)]
 pub(crate) static MD5_HEADER_NAME: &str = "content-md5";
@@ -85,6 +92,30 @@ impl HttpChecksum for Sha256 {
    }
 }

+impl HttpChecksum for Sha512 {
+    fn header_name(&self) -> &'static str {
+        SHA_512_HEADER_NAME
+    }
+}
+
+impl HttpChecksum for Xxhash3 {
+    fn header_name(&self) -> &'static str {
+        XXHASH_3_HEADER_NAME
+    }
+}
+
+impl HttpChecksum for Xxhash64 {
+    fn header_name(&self) -> &'static str {
+        XXHASH_64_HEADER_NAME
+    }
+}
+
+impl HttpChecksum for Xxhash128 {
+    fn header_name(&self) -> &'static str {
+        XXHASH_128_HEADER_NAME
+    }
+}
+
 impl HttpChecksum for Md5 {
    fn header_name(&self) -> &'static str {
        MD5_HEADER_NAME
@@ -35,6 +35,10 @@ pub const CRC_32_C_NAME: &str = "crc32c";
 pub const CRC_64_NVME_NAME: &str = "crc64nvme";
 pub const SHA_1_NAME: &str = "sha1";
 pub const SHA_256_NAME: &str = "sha256";
+pub const SHA_512_NAME: &str = "sha512";
+pub const XXHASH_3_NAME: &str = "xxhash3";
+pub const XXHASH_64_NAME: &str = "xxhash64";
+pub const XXHASH_128_NAME: &str = "xxhash128";
 pub const MD5_NAME: &str = "md5";

 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
@@ -46,6 +50,10 @@ pub enum ChecksumAlgorithm {
    Sha1,
    Sha256,
    Crc64Nvme,
+    Sha512,
+    Xxhash3,
+    Xxhash64,
+    Xxhash128,
 }

 impl FromStr for ChecksumAlgorithm {
@@ -62,6 +70,14 @@ impl FromStr for ChecksumAlgorithm {
            Ok(Self::Sha256)
        } else if checksum_algorithm.eq_ignore_ascii_case(CRC_64_NVME_NAME) {
            Ok(Self::Crc64Nvme)
+        } else if checksum_algorithm.eq_ignore_ascii_case(SHA_512_NAME) {
+            Ok(Self::Sha512)
+        } else if checksum_algorithm.eq_ignore_ascii_case(XXHASH_3_NAME) {
+            Ok(Self::Xxhash3)
+        } else if checksum_algorithm.eq_ignore_ascii_case(XXHASH_64_NAME) {
+            Ok(Self::Xxhash64)
+        } else if checksum_algorithm.eq_ignore_ascii_case(XXHASH_128_NAME) {
+            Ok(Self::Xxhash128)
        } else {
            Err(UnknownChecksumAlgorithmError::new(checksum_algorithm))
        }
@@ -76,6 +92,10 @@ impl ChecksumAlgorithm {
            Self::Crc64Nvme => Box::<Crc64Nvme>::default(),
            Self::Sha1 => Box::<Sha1>::default(),
            Self::Sha256 => Box::<Sha256>::default(),
+            Self::Sha512 => Box::<Sha512>::default(),
+            Self::Xxhash3 => Box::<Xxhash3>::default(),
+            Self::Xxhash64 => Box::<Xxhash64>::default(),
+            Self::Xxhash128 => Box::<Xxhash128>::default(),
        }
    }

@@ -86,6 +106,10 @@ impl ChecksumAlgorithm {
            Self::Crc64Nvme => CRC_64_NVME_NAME,
            Self::Sha1 => SHA_1_NAME,
            Self::Sha256 => SHA_256_NAME,
+            Self::Sha512 => SHA_512_NAME,
+            Self::Xxhash3 => XXHASH_3_NAME,
+            Self::Xxhash64 => XXHASH_64_NAME,
+            Self::Xxhash128 => XXHASH_128_NAME,
        }
    }
 }
@@ -285,6 +309,165 @@ impl Checksum for Sha256 {
        Self::size()
    }
 }
+
+#[derive(Debug, Default)]
+struct Sha512 {
+    hasher: sha2::Sha512,
+}
+
+impl Sha512 {
+    fn update(&mut self, bytes: &[u8]) {
+        use sha2::Digest;
+        self.hasher.update(bytes);
+    }
+
+    fn finalize(self) -> Bytes {
+        use sha2::Digest;
+        Bytes::copy_from_slice(self.hasher.finalize().as_slice())
+    }
+
+    fn size() -> u64 {
+        use sha2::Digest;
+        sha2::Sha512::output_size() as u64
+    }
+}
+
+impl Checksum for Sha512 {
+    fn update(&mut self, bytes: &[u8]) {
+        Self::update(self, bytes);
+    }
+    fn finalize(self: Box<Self>) -> Bytes {
+        Self::finalize(*self)
+    }
+    fn size(&self) -> u64 {
+        Self::size()
+    }
+}
+
+/// XXH3 (64-bit) hasher with the canonical seed of 0.
+///
+/// The raw digest is a `u64` serialized as 8 big-endian bytes so that the value
+/// matches the server-side (`rustfs-rio`) computation for the same algorithm.
+struct Xxhash3 {
+    hasher: xxhash_rust::xxh3::Xxh3,
+}
+
+impl Default for Xxhash3 {
+    fn default() -> Self {
+        Self {
+            hasher: xxhash_rust::xxh3::Xxh3::new(),
+        }
+    }
+}
+
+impl Xxhash3 {
+    fn update(&mut self, bytes: &[u8]) {
+        self.hasher.update(bytes);
+    }
+
+    fn finalize(self) -> Bytes {
+        Bytes::copy_from_slice(self.hasher.digest().to_be_bytes().as_slice())
+    }
+
+    fn size() -> u64 {
+        8
+    }
+}
+
+impl Checksum for Xxhash3 {
+    fn update(&mut self, bytes: &[u8]) {
+        Self::update(self, bytes)
+    }
+    fn finalize(self: Box<Self>) -> Bytes {
+        Self::finalize(*self)
+    }
+    fn size(&self) -> u64 {
+        Self::size()
+    }
+}
+
+/// XXH3 (128-bit) hasher with the canonical seed of 0.
+///
+/// The raw digest is a `u128` serialized as 16 big-endian bytes.
+struct Xxhash128 {
+    hasher: xxhash_rust::xxh3::Xxh3,
+}
+
+impl Default for Xxhash128 {
+    fn default() -> Self {
+        Self {
+            hasher: xxhash_rust::xxh3::Xxh3::new(),
+        }
+    }
+}
+
+impl Xxhash128 {
+    fn update(&mut self, bytes: &[u8]) {
+        self.hasher.update(bytes);
+    }
+
+    fn finalize(self) -> Bytes {
+        Bytes::copy_from_slice(self.hasher.digest128().to_be_bytes().as_slice())
+    }
+
+    fn size() -> u64 {
+        16
+    }
+}
+
+impl Checksum for Xxhash128 {
+    fn update(&mut self, bytes: &[u8]) {
+        Self::update(self, bytes)
+    }
+    fn finalize(self: Box<Self>) -> Bytes {
+        Self::finalize(*self)
+    }
+    fn size(&self) -> u64 {
+        Self::size()
+    }
+}
+
+/// XXH64 hasher with the canonical seed of 0.
+///
+/// The raw digest is a `u64` serialized as 8 big-endian bytes.
+struct Xxhash64 {
+    hasher: xxhash_rust::xxh64::Xxh64,
+}
+
+impl Default for Xxhash64 {
+    fn default() -> Self {
+        Self {
+            hasher: xxhash_rust::xxh64::Xxh64::new(0),
+        }
+    }
+}
+
+impl Xxhash64 {
+    fn update(&mut self, bytes: &[u8]) {
+        self.hasher.update(bytes);
+    }
+
+    fn finalize(self) -> Bytes {
+        Bytes::copy_from_slice(self.hasher.digest().to_be_bytes().as_slice())
+    }
+
+    fn size() -> u64 {
+        8
+    }
+}
+
+impl Checksum for Xxhash64 {
+    fn update(&mut self, bytes: &[u8]) {
+        Self::update(self, bytes)
+    }
+    fn finalize(self: Box<Self>) -> Bytes {
+        Self::finalize(*self)
+    }
+    fn size(&self) -> u64 {
+        Self::size()
+    }
+}
+
 #[allow(dead_code)]
 #[derive(Debug, Default)]
 struct Md5 {
@@ -455,4 +638,97 @@ mod tests {
        let error = "MD5".parse::<ChecksumAlgorithm>().expect_err("md5 should not parse");
        assert_eq!("MD5", error.checksum_algorithm());
    }
+
+    #[test]
+    fn test_additional_algorithms_parse_and_round_trip() {
+        // The AWS 2026-04 additional checksum algorithms must be recognised
+        // (case-insensitively) and round-trip through as_str().
+        for (name, expected) in [
+            ("sha512", ChecksumAlgorithm::Sha512),
+            ("SHA512", ChecksumAlgorithm::Sha512),
+            ("xxhash3", ChecksumAlgorithm::Xxhash3),
+            ("XXHASH3", ChecksumAlgorithm::Xxhash3),
+            ("xxhash64", ChecksumAlgorithm::Xxhash64),
+            ("xxhash128", ChecksumAlgorithm::Xxhash128),
+        ] {
+            let parsed = name.parse::<ChecksumAlgorithm>().expect("algorithm should parse");
+            assert_eq!(parsed, expected);
+            assert_eq!(expected.as_str().parse::<ChecksumAlgorithm>().unwrap(), expected);
+        }
+    }
+
+    #[test]
+    fn test_unknown_algorithm_never_panics_and_fails_closed() {
+        // Fail-closed contract: an unknown or garbage algorithm name must return
+        // an error instead of panicking or silently substituting another hasher.
+        for name in ["", "xxhash", "sha3", "crc16", "not-a-real-algo", "🦀"] {
+            assert!(name.parse::<ChecksumAlgorithm>().is_err(), "unknown algorithm {name:?} must fail closed");
+        }
+    }
+
+    #[test]
+    fn test_sha512_matches_direct_computation() {
+        use crate::Sha512;
+        use crate::http::SHA_512_HEADER_NAME;
+        use sha2::{Digest, Sha512 as Sha512Ref};
+
+        let mut checksum = Sha512::default();
+        checksum.update(TEST_DATA.as_bytes());
+        let header = Box::new(checksum).headers();
+        let encoded = header.get(SHA_512_HEADER_NAME).expect("sha512 header present");
+        let got = base64_encoded_checksum_to_hex_string(encoded);
+
+        let mut reference = Sha512Ref::new();
+        reference.update(TEST_DATA.as_bytes());
+        let expected = reference.finalize().iter().fold(String::from("0x"), |mut acc, b| {
+            write!(acc, "{b:02X?}").unwrap();
+            acc
+        });
+        assert_eq!(got, expected);
+    }
+
+    #[test]
+    fn test_xxhash3_matches_direct_computation_big_endian_seed0() {
+        use crate::Xxhash3;
+        use xxhash_rust::xxh3::Xxh3;
+
+        let mut checksum = Xxhash3::default();
+        checksum.update(TEST_DATA.as_bytes());
+        let raw = Box::new(checksum).finalize();
+
+        let mut reference = Xxh3::new();
+        reference.update(TEST_DATA.as_bytes());
+        assert_eq!(raw.len(), 8);
+        assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
+    }
+
+    #[test]
+    fn test_xxhash128_matches_direct_computation_big_endian_seed0() {
+        use crate::Xxhash128;
+        use xxhash_rust::xxh3::Xxh3;
+
+        let mut checksum = Xxhash128::default();
+        checksum.update(TEST_DATA.as_bytes());
+        let raw = Box::new(checksum).finalize();
+
+        let mut reference = Xxh3::new();
+        reference.update(TEST_DATA.as_bytes());
+        assert_eq!(raw.len(), 16);
+        assert_eq!(&raw[..], reference.digest128().to_be_bytes().as_slice());
+    }
+
+    #[test]
+    fn test_xxhash64_matches_direct_computation_big_endian_seed0() {
+        use crate::Xxhash64;
+        use xxhash_rust::xxh64::Xxh64;
+
+        let mut checksum = Xxhash64::default();
+        checksum.update(TEST_DATA.as_bytes());
+        let raw = Box::new(checksum).finalize();
+
+        let mut reference = Xxh64::new(0);
+        reference.update(TEST_DATA.as_bytes());
+        assert_eq!(raw.len(), 8);
+        assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
+    }
 }
@@ -28,14 +28,14 @@ categories = ["web-programming", "development-tools", "data-structures"]
 workspace = true

 [dependencies]
-tokio = { workspace = true }
-tonic = { workspace = true }
-uuid = { workspace = true }
-chrono = { workspace = true }
+tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
+tonic = { workspace = true, features = ["gzip", "deflate"] }
+uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
+chrono = { workspace = true, features = ["serde"] }
 metrics = { workspace = true }
-serde = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
 rmp-serde = { workspace = true }
-s3s = { workspace = true }
+s3s = { workspace = true, features = ["minio"] }
 tracing = { workspace = true }

 [lib]
@@ -13,15 +13,15 @@ categories = ["concurrency", "filesystem"]
 [dependencies]
 # Internal crates
 rustfs-io-core = { workspace = true }
-serde = { workspace = true }
+serde = { workspace = true, features = ["derive"] }

 # Async runtime
-tokio = { workspace = true, features = ["sync"] }
+tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }

 # Logging
 tracing = { workspace = true }

 [dev-dependencies]
-insta = { workspace = true }
-serde_json = { workspace = true }
-tokio = { workspace = true, features = ["test-util", "macros", "rt-multi-thread"] }
+insta = { workspace = true, features = ["yaml", "json"] }
+serde_json = { workspace = true, features = ["raw_value"] }
+tokio = { workspace = true, features = ["test-util", "macros", "rt-multi-thread", "fs"] }
@@ -25,9 +25,9 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
 categories = ["web-programming", "development-tools", "config"]

 [dependencies]
-const-str = { workspace = true, optional = true }
-serde = { workspace = true, optional = true }
-serde_json = { workspace = true, optional = true }
+const-str = { workspace = true, optional = true, features = ["std", "proc"] }
+serde = { workspace = true, optional = true, features = ["derive"] }
+serde_json = { workspace = true, optional = true, features = ["raw_value"] }

 [lints]
 workspace = true
@@ -25,8 +25,11 @@ pub const ENV_AUDIT_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_AUDIT_NATS_TLS_CLIENT_KE
 pub const ENV_AUDIT_NATS_TLS_REQUIRED: &str = "RUSTFS_AUDIT_NATS_TLS_REQUIRED";
 pub const ENV_AUDIT_NATS_QUEUE_DIR: &str = "RUSTFS_AUDIT_NATS_QUEUE_DIR";
 pub const ENV_AUDIT_NATS_QUEUE_LIMIT: &str = "RUSTFS_AUDIT_NATS_QUEUE_LIMIT";
+pub const ENV_AUDIT_NATS_JETSTREAM_ENABLE: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_ENABLE";
+pub const ENV_AUDIT_NATS_JETSTREAM_STREAM_NAME: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_STREAM_NAME";
+pub const ENV_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "RUSTFS_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS";

-pub const ENV_AUDIT_NATS_KEYS: &[&str; 13] = &[
+pub const ENV_AUDIT_NATS_KEYS: &[&str; 16] = &[
    ENV_AUDIT_NATS_ENABLE,
    ENV_AUDIT_NATS_ADDRESS,
    ENV_AUDIT_NATS_SUBJECT,
@@ -40,6 +43,9 @@ pub const ENV_AUDIT_NATS_KEYS: &[&str; 13] = &[
    ENV_AUDIT_NATS_TLS_REQUIRED,
    ENV_AUDIT_NATS_QUEUE_DIR,
    ENV_AUDIT_NATS_QUEUE_LIMIT,
+    ENV_AUDIT_NATS_JETSTREAM_ENABLE,
+    ENV_AUDIT_NATS_JETSTREAM_STREAM_NAME,
+    ENV_AUDIT_NATS_JETSTREAM_ACK_TIMEOUT_SECS,
 ];

 pub const AUDIT_NATS_KEYS: &[&str] = &[
@@ -56,5 +62,8 @@ pub const AUDIT_NATS_KEYS: &[&str] = &[
    crate::NATS_TLS_REQUIRED,
    crate::NATS_QUEUE_DIR,
    crate::NATS_QUEUE_LIMIT,
+    crate::NATS_JETSTREAM_ENABLE,
+    crate::NATS_JETSTREAM_STREAM_NAME,
+    crate::NATS_JETSTREAM_ACK_TIMEOUT_SECS,
    crate::COMMENT_KEY,
 ];
@@ -0,0 +1,52 @@
+//  Copyright 2024 RustFS Team
+//
+//  Licensed under the Apache License, Version 2.0 (the "License");
+//  you may not use this file except in compliance with the License.
+//  You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+//  Unless required by applicable law or agreed to in writing, software
+//  distributed under the License is distributed on an "AS IS" BASIS,
+//  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//  See the License for the specific language governing permissions and
+//  limitations under the License.
+
+/// Enable or disable per-client rate limiting for the S3 API.
+///
+/// When enabled (and `RUSTFS_API_RATE_LIMIT_RPM` > 0), requests are throttled
+/// per client IP using a token bucket; over-limit requests receive
+/// `429 Too Many Requests` with a `Retry-After` header. Internode RPC/gRPC,
+/// health probes, and the console (which has its own limiter) are exempt.
+/// Environment variable: RUSTFS_API_RATE_LIMIT_ENABLE
+/// Example: RUSTFS_API_RATE_LIMIT_ENABLE=true
+pub const ENV_API_RATE_LIMIT_ENABLE: &str = "RUSTFS_API_RATE_LIMIT_ENABLE";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_ENABLE`.
+///
+/// Disabled by default: RustFS ships permissive and operators opt in to
+/// abuse-protection hardening. When disabled the request path is unchanged.
+pub const DEFAULT_API_RATE_LIMIT_ENABLE: bool = false;
+
+/// Sustained S3 API request budget per client IP, in requests per minute.
+///
+/// `0` means unlimited (rate limiting stays inert even when enabled).
+/// Environment variable: RUSTFS_API_RATE_LIMIT_RPM
+/// Example: RUSTFS_API_RATE_LIMIT_RPM=6000
+pub const ENV_API_RATE_LIMIT_RPM: &str = "RUSTFS_API_RATE_LIMIT_RPM";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_RPM`.
+///
+/// `0` (unlimited) so that setting only the enable switch cannot throttle
+/// traffic by surprise; operators must choose an explicit budget.
+pub const DEFAULT_API_RATE_LIMIT_RPM: u32 = 0;
+
+/// Burst capacity per client IP (maximum tokens in the bucket).
+///
+/// Allows short spikes above the sustained rate. `0` means "same as RPM".
+/// Environment variable: RUSTFS_API_RATE_LIMIT_BURST
+/// Example: RUSTFS_API_RATE_LIMIT_BURST=200
+pub const ENV_API_RATE_LIMIT_BURST: &str = "RUSTFS_API_RATE_LIMIT_BURST";
+
+/// Default for `RUSTFS_API_RATE_LIMIT_BURST` (`0` = same as RPM).
+pub const DEFAULT_API_RATE_LIMIT_BURST: u32 = 0;
@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+pub(crate) mod api;
 pub(crate) mod app;
 pub(crate) mod body_limits;
 pub(crate) mod capacity;
@@ -73,15 +73,36 @@ pub const DEFAULT_OBJECT_MAX_CONCURRENT_DISK_READS: usize = 64;
 /// - Example: `export RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT=5`
 pub const ENV_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_PERMIT_WAIT_TIMEOUT";

-/// Maximum time a GET request waits for a disk read permit (seconds).
+/// Maximum time a GET request waits for a primary disk read permit (seconds).
 ///
 /// Permits are held for the whole response body transfer, so slow clients can
 /// occupy all of them while the disks sit idle. Instead of stalling until the
-/// request-level timeout fires, a GET that waits longer than this proceeds
-/// without a permit (degraded pass-through) and the bypass is counted in
-/// metrics/logs. Set to 0 to wait indefinitely (previous behavior).
+/// request-level timeout fires, a GET that waits longer than this falls through
+/// to a bounded degraded admission lane; if that lane is also full the request
+/// is rejected with `SlowDown`/503 rather than proceeding without any permit.
+/// Set to 0 to wait on the primary lane indefinitely (never degrade or reject).
 pub const DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT: u64 = 5;

+/// Environment variable for the bounded degraded disk-read admission lane size.
+/// - Purpose: Cap how many GETs may proceed after the primary disk-read permit
+///   pool is saturated, giving a hard upper bound on concurrent disk-active
+///   reads (primary cap + degraded cap) instead of an unbounded pass-through.
+/// - Unit: request count (usize). `0` means "mirror the primary cap", so the
+///   absolute hard cap defaults to twice the primary disk-read cap.
+/// - Example: `export RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP=16`
+pub const ENV_OBJECT_DISK_DEGRADED_READ_CAP: &str = "RUSTFS_OBJECT_DISK_DEGRADED_READ_CAP";
+
+/// Size of the bounded degraded disk-read admission lane.
+///
+/// When the primary disk-read permit pool is saturated and a GET exceeds
+/// [`DEFAULT_OBJECT_DISK_PERMIT_WAIT_TIMEOUT`], it may take one permit from this
+/// bounded overflow lane instead of reading without any admission token. The
+/// total number of GETs performing disk-active reads is therefore hard-capped at
+/// `primary_cap + degraded_cap`; beyond that a GET is rejected with `SlowDown`.
+/// The default `0` mirrors the primary cap, so the hard cap is twice the primary
+/// disk-read concurrency.
+pub const DEFAULT_OBJECT_DISK_DEGRADED_READ_CAP: usize = 0;
+
 /// Skip bitrot hash verification on GetObject reads.
 ///
 /// When enabled, GetObject reads skip the per-shard hash
@@ -128,6 +149,39 @@ pub const ENV_OBJECT_DISK_READ_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_READ_TIMEOUT"
 /// Default disk read timeout in seconds.
 pub const DEFAULT_OBJECT_DISK_READ_TIMEOUT: u64 = 10;

+/// Environment variable for the per-shard erasure write stall timeout (seconds).
+///
+/// A single shard write (or shard-writer shutdown) that makes no forward
+/// progress for longer than this budget is failed and its disk is dropped
+/// before commit, so a black-hole peer that accepts the connection but never
+/// drains the body cannot pin an otherwise-healthy write quorum forever
+/// (see `MultiWriter` in `erasure/coding/encode.rs`). The budget is re-armed on
+/// every shard write, so it bounds a *stall* rather than the total transfer
+/// time of a large object.
+///
+/// Unit: seconds (u64). `0` disables the stall deadline (previous behavior:
+/// wait indefinitely). Default: 30 seconds.
+pub const ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT: &str = "RUSTFS_OBJECT_DISK_WRITE_STALL_TIMEOUT";
+
+/// Default per-shard erasure write stall timeout in seconds.
+pub const DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT: u64 = 30;
+
+/// Environment variable for the absolute per-object erasure write cap (seconds).
+///
+/// Optional administrator backstop against a "slow-drip" peer that produces
+/// just enough forward progress to reset the per-shard stall timeout on every
+/// block while never converging. When set, the shard writers for one object are
+/// engaged for at most this long in aggregate before a stalled writer is failed
+/// and dropped. It is disabled by default because a legitimate large upload
+/// over a slow-but-honest link must not be killed on total time alone; the
+/// per-shard stall timeout is the primary guarantee.
+///
+/// Unit: seconds (u64). `0` (default) disables the absolute cap.
+pub const ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP: &str = "RUSTFS_OBJECT_DISK_WRITE_ABSOLUTE_CAP";
+
+/// Default absolute per-object erasure write cap in seconds (`0` = disabled).
+pub const DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP: u64 = 0;
+
 /// Environment variable for minimum GetObject timeout in seconds.
 ///
 /// When dynamic timeout calculation is enabled, this is the minimum timeout
@@ -79,6 +79,13 @@ pub const NATS_TLS_CLIENT_KEY: &str = "tls_client_key";
 pub const NATS_TLS_REQUIRED: &str = "tls_required";
 pub const NATS_QUEUE_DIR: &str = "queue_dir";
 pub const NATS_QUEUE_LIMIT: &str = "queue_limit";
+pub const NATS_JETSTREAM_ENABLE: &str = "jetstream_enable";
+pub const NATS_JETSTREAM_STREAM_NAME: &str = "jetstream_stream_name";
+pub const NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "jetstream_ack_timeout_secs";
+
+pub const NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS: u64 = 30;
+pub const NATS_JETSTREAM_ACK_TIMEOUT_MIN_SECS: u64 = 10;
+pub const NATS_JETSTREAM_ACK_TIMEOUT_MAX_SECS: u64 = 120;

 pub const PULSAR_BROKER: &str = "broker";
 pub const PULSAR_TOPIC: &str = "topic";
@@ -15,6 +15,8 @@
 #[cfg(feature = "constants")]
 pub mod constants;
 #[cfg(feature = "constants")]
+pub use constants::api::*;
+#[cfg(feature = "constants")]
 pub use constants::app::*;
 #[cfg(feature = "constants")]
 pub use constants::body_limits::*;
@@ -26,6 +26,9 @@ pub const NOTIFY_NATS_KEYS: &[&str] = &[
    crate::NATS_TLS_REQUIRED,
    crate::NATS_QUEUE_DIR,
    crate::NATS_QUEUE_LIMIT,
+    crate::NATS_JETSTREAM_ENABLE,
+    crate::NATS_JETSTREAM_STREAM_NAME,
+    crate::NATS_JETSTREAM_ACK_TIMEOUT_SECS,
    crate::COMMENT_KEY,
 ];

@@ -42,8 +45,11 @@ pub const ENV_NOTIFY_NATS_TLS_CLIENT_KEY: &str = "RUSTFS_NOTIFY_NATS_TLS_CLIENT_
 pub const ENV_NOTIFY_NATS_TLS_REQUIRED: &str = "RUSTFS_NOTIFY_NATS_TLS_REQUIRED";
 pub const ENV_NOTIFY_NATS_QUEUE_DIR: &str = "RUSTFS_NOTIFY_NATS_QUEUE_DIR";
 pub const ENV_NOTIFY_NATS_QUEUE_LIMIT: &str = "RUSTFS_NOTIFY_NATS_QUEUE_LIMIT";
+pub const ENV_NOTIFY_NATS_JETSTREAM_ENABLE: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_ENABLE";
+pub const ENV_NOTIFY_NATS_JETSTREAM_STREAM_NAME: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_STREAM_NAME";
+pub const ENV_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS: &str = "RUSTFS_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS";

-pub const ENV_NOTIFY_NATS_KEYS: &[&str; 13] = &[
+pub const ENV_NOTIFY_NATS_KEYS: &[&str; 16] = &[
    ENV_NOTIFY_NATS_ENABLE,
    ENV_NOTIFY_NATS_ADDRESS,
    ENV_NOTIFY_NATS_SUBJECT,
@@ -57,4 +63,7 @@ pub const ENV_NOTIFY_NATS_KEYS: &[&str; 13] = &[
    ENV_NOTIFY_NATS_TLS_REQUIRED,
    ENV_NOTIFY_NATS_QUEUE_DIR,
    ENV_NOTIFY_NATS_QUEUE_LIMIT,
+    ENV_NOTIFY_NATS_JETSTREAM_ENABLE,
+    ENV_NOTIFY_NATS_JETSTREAM_STREAM_NAME,
+    ENV_NOTIFY_NATS_JETSTREAM_ACK_TIMEOUT_SECS,
 ];
@@ -27,9 +27,9 @@ categories = ["web-programming", "development-tools", "data-structures", "securi
 [dependencies]
 base64-simd = { workspace = true }
 hmac = { workspace = true }
-rand = { workspace = true }
-serde = { workspace = true }
-serde_json.workspace = true
+rand = { workspace = true, features = ["serde"] }
+serde = { workspace = true, features = ["derive"] }
+serde_json = { workspace = true, features = ["raw_value"] }
 sha2 = { workspace = true }
 time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }

@@ -29,23 +29,23 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
 workspace = true

 [dependencies]
-aes-gcm = { workspace = true, optional = true }
+aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
 argon2 = { workspace = true, optional = true }
 chacha20poly1305 = { workspace = true, optional = true }
-jsonwebtoken = { workspace = true }
+jsonwebtoken = { workspace = true, features = ["aws_lc_rs"] }
 base64-simd = { workspace = true }
 pbkdf2 = { workspace = true, optional = true }
-rand = { workspace = true }
+rand = { workspace = true, features = ["serde"] }
 rsa = { workspace = true, features = ["sha2"] }
 serde = { workspace = true, features = ["derive"] }
 sha2 = { workspace = true, optional = true }
 thiserror.workspace = true
-serde_json.workspace = true
+serde_json = { workspace = true, features = ["raw_value"] }


 [dev-dependencies]
 test-case.workspace = true
-time.workspace = true
+time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }

 [features]
 default = ["crypto", "fips"]
@@ -28,7 +28,7 @@ categories = ["data-structures", "filesystem"]
 workspace = true

 [dependencies]
-serde = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
 path-clean = { workspace = true }
 rmp-serde = { workspace = true }
 async-trait = { workspace = true }
@@ -38,44 +38,47 @@ futures.workspace = true
 rustfs-lock.workspace = true
 rustfs-protos.workspace = true
 rmp-serde.workspace = true
-serde.workspace = true
-serde_json.workspace = true
-tonic = { workspace = true }
-tokio = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
+serde_json = { workspace = true, features = ["raw_value"] }
+tonic = { workspace = true, features = ["gzip", "deflate"] }
+tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tokio-stream = { workspace = true }
 rustfs-madmin.workspace = true
 rustfs-filemeta.workspace = true
-bytes.workspace = true
+bytes = { workspace = true, features = ["serde"] }
 serial_test = { workspace = true }
-aws-sdk-s3.workspace = true
+aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-config = { workspace = true }
-aws-smithy-http-client.workspace = true
+aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
 async-compression = { workspace = true, features = ["tokio", "bzip2", "xz"] }
 async-trait = { workspace = true }
 flate2.workspace = true
 http.workspace = true
-reqwest = { workspace = true }
+http-body-util.workspace = true
+hyper = { workspace = true, features = ["http2", "http1", "server"] }
+hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
+reqwest = { workspace = true, default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "multipart"] }
 rustfs-signer.workspace = true
 tracing = { workspace = true }
-tracing-subscriber = { workspace = true }
-uuid = { workspace = true }
+tracing-subscriber = { workspace = true, features = ["env-filter", "time"] }
+uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
 urlencoding.workspace = true
 walkdir.workspace = true
 base64 = { workspace = true }
-rand = { workspace = true }
-chrono = { workspace = true }
+rand = { workspace = true, features = ["serde"] }
+chrono = { workspace = true, features = ["serde"] }
 md5 = { workspace = true }
 sha2 = { workspace = true }
 astral-tokio-tar = { workspace = true }
-s3s = { workspace = true }
+s3s = { workspace = true, features = ["minio"] }
 zstd.workspace = true
-time.workspace = true
-suppaftp = { workspace = true, features = ["tokio", "rustls-aws-lc-rs"] }
+time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
+suppaftp = { workspace = true, features = ["rustls-aws-lc-rs", "tokio-rustls-aws-lc-rs"] }
 rcgen.workspace = true
 local-ip-address.workspace = true
 anyhow.workspace = true
-rustls.workspace = true
-russh = { workspace = true }
+rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
+russh = { workspace = true, features = ["serde"] }
 russh-sftp = { workspace = true }
 zip.workspace = true
-clap.workspace = true
+clap = { workspace = true, features = ["derive", "env"] }
@@ -0,0 +1,448 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Systematic HTTP e2e for the admin IAM management surface (backlog#1154
+//! peri-2, first batch): the full user / canned-policy / service-account CRUD
+//! lifecycle over real signed HTTP against a real binary, plus a non-admin
+//! denial probe for every management endpoint (reusing the sec-4 assertion
+//! pattern — the authorization gate itself is pinned by `admin_auth_test`).
+//!
+//! Before this suite the ~40 admin handler modules only had helper-level unit
+//! tests; no test proved that `mc admin user add` style flows work end to end
+//! (create -> attach policy -> the credential actually gains S3 access ->
+//! service account inherits it -> deletion revokes it).
+//!
+//! Later batches tracked on backlog#1154: config get/set, info, pools status,
+//! group lifecycle, import/export IAM.
+
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use aws_sdk_s3::config::{Credentials, Region};
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::{Client, Config};
+use http::header::{CONTENT_TYPE, HOST};
+use reqwest::StatusCode;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use serial_test::serial;
+use std::error::Error;
+use tokio::time::{Duration, sleep};
+
+type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
+type BoxError = Box<dyn Error + Send + Sync>;
+
+/// Signs and sends an admin HTTP request with the given credential, returning
+/// status and body. Native `/rustfs/admin/v3` requests and responses are plain
+/// JSON (the MinIO-compat encryption applies only to `/minio/admin/v3` paths).
+async fn admin_request(
+    base_url: &str,
+    method: http::Method,
+    path_and_query: &str,
+    body: Option<String>,
+    access_key: &str,
+    secret_key: &str,
+) -> Result<(StatusCode, String), BoxError> {
+    let url = format!("{base_url}{path_and_query}");
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
+    let mut builder = http::Request::builder()
+        .method(method.clone())
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if body.is_some() {
+        builder = builder.header(CONTENT_TYPE, "application/json");
+    }
+
+    let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
+    let signed = sign_v4(builder.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let mut request = local_http_client().request(reqwest_method, &url);
+    for (name, value) in signed.headers() {
+        request = request.header(name, value);
+    }
+    if let Some(body) = body {
+        request = request.body(body);
+    }
+    let response = request.send().await?;
+    let status = response.status();
+    let text = response.text().await.unwrap_or_default();
+    Ok((status, text))
+}
+
+/// Root-credential admin request that must succeed; returns the response body.
+async fn admin_ok(
+    env: &RustFSTestEnvironment,
+    method: http::Method,
+    path_and_query: &str,
+    body: Option<String>,
+) -> Result<String, BoxError> {
+    let (status, text) = admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
+    if !status.is_success() {
+        return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
+    }
+    Ok(text)
+}
+
+fn build_s3_client(url: &str, access_key: &str, secret_key: &str) -> Client {
+    let config = Config::builder()
+        .credentials_provider(Credentials::new(access_key, secret_key, None, None, "e2e-admin-crud"))
+        .region(Region::new("us-east-1"))
+        .endpoint_url(url)
+        .force_path_style(true)
+        .behavior_version_latest()
+        .build();
+    Client::from_conf(config)
+}
+
+/// Retries an S3 PUT until IAM propagation makes it succeed (bounded).
+async fn wait_for_s3_put(client: &Client, bucket: &str, key: &str, within: Duration) -> TestResult {
+    let deadline = tokio::time::Instant::now() + within;
+    loop {
+        match client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from_static(b"peri-2 probe"))
+            .send()
+            .await
+        {
+            Ok(_) => return Ok(()),
+            Err(err) => {
+                if tokio::time::Instant::now() >= deadline {
+                    return Err(format!("PUT {bucket}/{key} never succeeded: {err:?}").into());
+                }
+            }
+        }
+        sleep(Duration::from_millis(500)).await;
+    }
+}
+
+/// A canned policy granting full S3 access to one bucket.
+fn bucket_rw_policy(bucket: &str) -> String {
+    serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [{
+            "Effect": "Allow",
+            "Action": ["s3:*"],
+            "Resource": [
+                format!("arn:aws:s3:::{bucket}"),
+                format!("arn:aws:s3:::{bucket}/*")
+            ]
+        }]
+    })
+    .to_string()
+}
+
+/// Full user -> policy -> service-account lifecycle, proving each management
+/// call takes effect on the data plane, not just that the endpoint answers 200.
+#[tokio::test]
+#[serial]
+async fn test_admin_user_policy_service_account_crud_lifecycle() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let bucket = "peri2-crud";
+    let user = "peri2user";
+    let user_secret = "peri2usersecret";
+    let policy = "peri2-rw";
+
+    let root_client = env.create_s3_client();
+    root_client.create_bucket().bucket(bucket).send().await?;
+
+    // --- canned policy CRUD ---------------------------------------------------
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
+        Some(bucket_rw_policy(bucket)),
+    )
+    .await?;
+
+    let info = admin_ok(
+        &env,
+        http::Method::GET,
+        &format!("/rustfs/admin/v3/info-canned-policy?name={policy}"),
+        None,
+    )
+    .await?;
+    let info_json: serde_json::Value = serde_json::from_str(&info)?;
+    let statement_json = info_json.get("Policy").cloned().unwrap_or(info_json.clone());
+    assert!(
+        statement_json.to_string().contains("s3:*"),
+        "info-canned-policy must round-trip the policy document: {info}"
+    );
+
+    let listed = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/list-canned-policies", None).await?;
+    assert!(listed.contains(policy), "list-canned-policies must contain {policy}: {listed}");
+
+    // --- user CRUD --------------------------------------------------------------
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
+        Some(serde_json::json!({ "secretKey": user_secret, "status": "enabled" }).to_string()),
+    )
+    .await?;
+
+    let users = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/list-users", None).await?;
+    assert!(users.contains(user), "list-users must contain {user}: {users}");
+
+    let user_info = admin_ok(&env, http::Method::GET, &format!("/rustfs/admin/v3/user-info?accessKey={user}"), None).await?;
+    let user_info: rustfs_madmin::UserInfo = serde_json::from_str(&user_info)
+        .map_err(|e| format!("user-info response must decode as rustfs_madmin::UserInfo: {e}"))?;
+    assert_eq!(user_info.status, rustfs_madmin::AccountStatus::Enabled);
+
+    // The fresh user is authenticatable but unauthorized for the bucket.
+    let user_client = build_s3_client(&env.url, user, user_secret);
+    let denied = user_client
+        .put_object()
+        .bucket(bucket)
+        .key("before-attach")
+        .body(ByteStream::from_static(b"x"))
+        .send()
+        .await;
+    assert!(denied.is_err(), "user without a policy must not be able to write to {bucket}");
+
+    // --- attach policy: the credential actually gains S3 access -----------------
+    admin_ok(
+        &env,
+        http::Method::POST,
+        "/rustfs/admin/v3/idp/builtin/policy/attach",
+        Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
+    )
+    .await?;
+
+    wait_for_s3_put(&user_client, bucket, "after-attach.txt", Duration::from_secs(10)).await?;
+
+    let user_info = admin_ok(&env, http::Method::GET, &format!("/rustfs/admin/v3/user-info?accessKey={user}"), None).await?;
+    let user_info: rustfs_madmin::UserInfo = serde_json::from_str(&user_info)?;
+    assert!(
+        user_info.policy_name.as_deref().is_some_and(|p| p.contains(policy)),
+        "user-info must reflect the attached policy, got {:?}",
+        user_info.policy_name
+    );
+
+    // --- service account: create for the user, credential works, info/list pin it
+    let sa_resp = admin_ok(
+        &env,
+        http::Method::PUT,
+        "/rustfs/admin/v3/add-service-accounts",
+        Some(serde_json::json!({ "targetUser": user }).to_string()),
+    )
+    .await?;
+    let sa_json: serde_json::Value = serde_json::from_str(&sa_resp)?;
+    let sa_ak = sa_json["credentials"]["accessKey"]
+        .as_str()
+        .ok_or(format!("add-service-accounts response missing credentials.accessKey: {sa_resp}"))?
+        .to_string();
+    let sa_sk = sa_json["credentials"]["secretKey"]
+        .as_str()
+        .ok_or(format!("add-service-accounts response missing credentials.secretKey: {sa_resp}"))?
+        .to_string();
+
+    let sa_client = build_s3_client(&env.url, &sa_ak, &sa_sk);
+    wait_for_s3_put(&sa_client, bucket, "via-service-account.txt", Duration::from_secs(10)).await?;
+
+    let sa_info = admin_ok(
+        &env,
+        http::Method::GET,
+        &format!("/rustfs/admin/v3/info-service-account?accessKey={sa_ak}"),
+        None,
+    )
+    .await?;
+    let sa_info: serde_json::Value = serde_json::from_str(&sa_info)?;
+    assert_eq!(
+        sa_info["parentUser"].as_str(),
+        Some(user),
+        "info-service-account must name the parent user: {sa_info}"
+    );
+
+    let sa_list = admin_ok(
+        &env,
+        http::Method::GET,
+        &format!("/rustfs/admin/v3/list-service-accounts?user={user}"),
+        None,
+    )
+    .await?;
+    let sa_list: rustfs_madmin::ListServiceAccountsResp = serde_json::from_str(&sa_list)
+        .map_err(|e| format!("list-service-accounts must decode as rustfs_madmin::ListServiceAccountsResp: {e}"))?;
+    assert!(
+        sa_list.accounts.iter().any(|a| a.access_key == sa_ak),
+        "list-service-accounts must contain {sa_ak}"
+    );
+
+    // --- deletion revokes access -------------------------------------------------
+    admin_ok(
+        &env,
+        http::Method::DELETE,
+        &format!("/rustfs/admin/v3/delete-service-account?accessKey={sa_ak}"),
+        None,
+    )
+    .await?;
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
+    loop {
+        let revoked = sa_client
+            .put_object()
+            .bucket(bucket)
+            .key("after-sa-delete")
+            .body(ByteStream::from_static(b"x"))
+            .send()
+            .await;
+        if revoked.is_err() {
+            break;
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err("deleted service account credential still works".into());
+        }
+        sleep(Duration::from_millis(500)).await;
+    }
+
+    // Disable then remove the user; the credential must stop working.
+    admin_ok(
+        &env,
+        http::Method::PUT,
+        &format!("/rustfs/admin/v3/set-user-status?accessKey={user}&status=disabled"),
+        None,
+    )
+    .await?;
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
+    loop {
+        let disabled = user_client
+            .put_object()
+            .bucket(bucket)
+            .key("after-disable")
+            .body(ByteStream::from_static(b"x"))
+            .send()
+            .await;
+        if disabled.is_err() {
+            break;
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err("disabled user credential still works".into());
+        }
+        sleep(Duration::from_millis(500)).await;
+    }
+
+    admin_ok(
+        &env,
+        http::Method::DELETE,
+        &format!("/rustfs/admin/v3/remove-user?accessKey={user}"),
+        None,
+    )
+    .await?;
+    let users = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/list-users", None).await?;
+    assert!(!users.contains(user), "removed user must disappear from list-users: {users}");
+
+    admin_ok(
+        &env,
+        http::Method::DELETE,
+        &format!("/rustfs/admin/v3/remove-canned-policy?name={policy}"),
+        None,
+    )
+    .await?;
+    let (status, _) = admin_request(
+        &env.url,
+        http::Method::GET,
+        &format!("/rustfs/admin/v3/info-canned-policy?name={policy}"),
+        None,
+        &env.access_key,
+        &env.secret_key,
+    )
+    .await?;
+    assert!(!status.is_success(), "info-canned-policy must fail after remove-canned-policy");
+
+    env.stop_server();
+    Ok(())
+}
+
+/// Every management endpoint in the first batch must deny an authenticated but
+/// non-admin credential with 403 AccessDenied (sec-4 assertion pattern; the
+/// gate implementation itself is owned by sec-4 / admin_auth_test).
+#[tokio::test]
+#[serial]
+async fn test_admin_iam_endpoints_deny_non_admin_credential() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let limited = "peri2limited";
+    let limited_secret = "peri2limitedsecret";
+    let other = "peri2other";
+    for (user, secret) in [(limited, limited_secret), (other, "peri2othersecret")] {
+        admin_ok(
+            &env,
+            http::Method::PUT,
+            &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
+            Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
+        )
+        .await?;
+    }
+
+    // Admin-only management endpoints. Targeted probes aim at a DIFFERENT user
+    // (`other`): operations a principal performs on itself run in `deny_only`
+    // mode (see should_check_deny_only in rustfs/src/admin/handlers/user.rs)
+    // and would not be denied. Also deliberately excludes the self-service
+    // account endpoints (add/list service accounts), which any authenticated
+    // user may call for themselves.
+    let probes: Vec<(http::Method, String, Option<String>)> = vec![
+        (
+            http::Method::PUT,
+            "/rustfs/admin/v3/add-user?accessKey=peri2evil".to_string(),
+            Some(serde_json::json!({ "secretKey": "evilsecret123", "status": "enabled" }).to_string()),
+        ),
+        (http::Method::GET, "/rustfs/admin/v3/list-users".to_string(), None),
+        (http::Method::GET, format!("/rustfs/admin/v3/user-info?accessKey={other}"), None),
+        (
+            http::Method::PUT,
+            format!("/rustfs/admin/v3/set-user-status?accessKey={other}&status=disabled"),
+            None,
+        ),
+        (http::Method::DELETE, format!("/rustfs/admin/v3/remove-user?accessKey={other}"), None),
+        (
+            http::Method::PUT,
+            "/rustfs/admin/v3/add-canned-policy?name=peri2evilpolicy".to_string(),
+            Some(bucket_rw_policy("peri2-any")),
+        ),
+        (http::Method::GET, "/rustfs/admin/v3/list-canned-policies".to_string(), None),
+        (http::Method::GET, "/rustfs/admin/v3/info-canned-policy?name=readwrite".to_string(), None),
+        (
+            http::Method::DELETE,
+            "/rustfs/admin/v3/remove-canned-policy?name=readwrite".to_string(),
+            None,
+        ),
+        (
+            http::Method::POST,
+            "/rustfs/admin/v3/idp/builtin/policy/attach".to_string(),
+            Some(serde_json::json!({ "policies": ["readwrite"], "user": other }).to_string()),
+        ),
+    ];
+
+    for (method, path, body) in probes {
+        let (status, text) = admin_request(&env.url, method.clone(), &path, body, limited, limited_secret).await?;
+        assert_eq!(
+            status,
+            StatusCode::FORBIDDEN,
+            "non-admin credential must get 403 on {method} {path}, got {status}: {text}"
+        );
+        assert!(text.contains("AccessDenied"), "denial on {method} {path} must carry AccessDenied: {text}");
+    }
+
+    env.stop_server();
+    Ok(())
+}
@@ -0,0 +1,109 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! E2E coverage for the opt-in per-client S3 API rate limit (backlog#1191):
+//! the layer must be wired into the real server stack, reject over-limit
+//! clients with `429` + `Retry-After`, keep health probes exempt, and stay
+//! completely inert with default configuration.
+
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use serial_test::serial;
+use tracing::info;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+#[tokio::test]
+#[serial]
+async fn api_rate_limit_enforces_429_with_retry_after_when_enabled() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    // Burst 50 leaves headroom for the readiness-poll ListBuckets calls that
+    // share the loopback client IP; refill (60 rpm = 1/s) is slow enough that
+    // a rapid burst below reliably exhausts the bucket.
+    env.start_rustfs_server_with_env(
+        vec![],
+        &[
+            ("RUSTFS_API_RATE_LIMIT_ENABLE", "true"),
+            ("RUSTFS_API_RATE_LIMIT_RPM", "60"),
+            ("RUSTFS_API_RATE_LIMIT_BURST", "50"),
+        ],
+    )
+    .await?;
+
+    let client = local_http_client();
+    let list_buckets_url = format!("{}/", env.url);
+
+    let mut throttled = None;
+    let mut allowed = 0usize;
+    for _ in 0..80 {
+        let response = client.get(&list_buckets_url).send().await?;
+        if response.status() == reqwest::StatusCode::TOO_MANY_REQUESTS {
+            throttled = Some(response);
+            break;
+        }
+        allowed += 1;
+    }
+
+    let throttled = throttled.unwrap_or_else(|| panic!("no 429 within 80 rapid requests ({allowed} allowed) at burst 50"));
+    assert!(allowed > 0, "healthy traffic below the burst must not be throttled");
+    info!("rate limit engaged after {allowed} allowed requests");
+
+    let retry_after = throttled
+        .headers()
+        .get(reqwest::header::RETRY_AFTER)
+        .and_then(|v| v.to_str().ok())
+        .and_then(|v| v.parse::<u64>().ok())
+        .expect("429 must carry a numeric Retry-After header");
+    assert!(retry_after >= 1, "Retry-After must be at least one second, got {retry_after}");
+    assert_eq!(
+        throttled.headers().get("x-ratelimit-limit").and_then(|v| v.to_str().ok()),
+        Some("60"),
+        "429 must expose the configured limit"
+    );
+
+    let body = throttled.text().await?;
+    assert!(body.contains("<Code>TooManyRequests</Code>"), "S3-style error body expected: {body}");
+
+    // Health probes stay exempt even while the client budget is exhausted.
+    let health = client.get(format!("{}/health", env.url)).send().await?;
+    assert_ne!(
+        health.status(),
+        reqwest::StatusCode::TOO_MANY_REQUESTS,
+        "health probes must never be rate limited"
+    );
+
+    Ok(())
+}
+
+#[tokio::test]
+#[serial]
+async fn api_rate_limit_stays_inert_by_default() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+
+    let client = local_http_client();
+    let list_buckets_url = format!("{}/", env.url);
+
+    for i in 0..80 {
+        let response = client.get(&list_buckets_url).send().await?;
+        assert_ne!(
+            response.status(),
+            reqwest::StatusCode::TOO_MANY_REQUESTS,
+            "request {i} was throttled although rate limiting is disabled by default"
+        );
+    }
+
+    Ok(())
+}
@@ -19,8 +19,10 @@
 mod tests {
    use crate::common::{RustFSTestEnvironment, init_logging};
    use aws_sdk_s3::Client;
+    use aws_sdk_s3::config::{Credentials, Region, RequestChecksumCalculation};
    use aws_sdk_s3::primitives::ByteStream;
    use aws_sdk_s3::types::{ChecksumAlgorithm, ChecksumMode, CompletedMultipartUpload, CompletedPart};
+    use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
    use base64::Engine;
    use rustfs_rio::{Checksum, ChecksumType as RioChecksumType};
    use serial_test::serial;
@@ -31,6 +33,25 @@ mod tests {
        env.create_s3_client()
    }

+    /// Client with boto/SDK automatic checksum calculation disabled, so the ONLY
+    /// checksum on the wire is the one the test injects. Needed because the SDK's
+    /// default (crc32) would otherwise collide with the additional-algorithm header
+    /// we inject via `mutate_request` for XXHash/SHA-512/MD5 (which have no typed
+    /// SDK builder). Mirrors the boto3 `request_checksum_calculation=when_required`.
+    fn create_s3_client_no_auto_checksum(env: &RustFSTestEnvironment) -> Client {
+        let creds = Credentials::new(&env.access_key, &env.secret_key, None, None, "e2e-additional-checksum");
+        let config = aws_sdk_s3::Config::builder()
+            .credentials_provider(creds)
+            .region(Region::new("us-east-1"))
+            .endpoint_url(format!("http://{}", env.address))
+            .force_path_style(true)
+            .behavior_version_latest()
+            .request_checksum_calculation(RequestChecksumCalculation::WhenRequired)
+            .http_client(SmithyHttpClientBuilder::new().build_http())
+            .build();
+        Client::from_conf(config)
+    }
+
    async fn create_bucket(client: &Client, bucket: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
        match client.create_bucket().bucket(bucket).send().await {
            Ok(_) => {
@@ -459,4 +480,87 @@ mod tests {
            "Multipart object should report the same full-object CRC64NVME as direct upload"
        );
    }
+
+    /// Integration test for the AWS 2026-04 additional checksum algorithms
+    /// (XXHash3/64/128, SHA-512, MD5). aws_sdk_s3 has no typed builder for these, so the
+    /// `x-amz-checksum-<algo>` header is injected via `mutate_request` (value computed by
+    /// rustfs-rio, which is byte-for-byte identical to awscrt). Verifies the server
+    /// verifies-on-write: a correct value is accepted and the object stored; a wrong
+    /// value is rejected with BadDigest and nothing is stored. Full HEAD/GET header
+    /// echo round-trip is additionally exercised by the boto3+awscrt e2e.
+    #[tokio::test]
+    #[serial]
+    async fn test_additional_checksums_verify_on_write() {
+        init_logging();
+        info!("TEST: additional checksums (XXHash3/64/128, SHA-512, MD5) verify-on-write");
+
+        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
+        env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
+
+        let client = create_s3_client_no_auto_checksum(&env);
+        let bucket = "test-additional-checksums";
+        create_bucket(&client, bucket).await.expect("Failed to create bucket");
+
+        let content: &[u8] = b"additional-checksum verify-on-write payload for xxhash/sha512/md5";
+
+        for (ty, header) in [
+            (RioChecksumType::XXHASH3, "x-amz-checksum-xxhash3"),
+            (RioChecksumType::XXHASH64, "x-amz-checksum-xxhash64"),
+            (RioChecksumType::XXHASH128, "x-amz-checksum-xxhash128"),
+            (RioChecksumType::SHA512, "x-amz-checksum-sha512"),
+            (RioChecksumType::MD5, "x-amz-checksum-md5"),
+        ] {
+            // Correct checksum -> accepted, and the object is stored intact.
+            let good = Checksum::new_from_data(ty, content).expect("compute checksum").encoded;
+            let ok_key = format!("ok-{header}");
+            let put = client
+                .put_object()
+                .bucket(bucket)
+                .key(&ok_key)
+                .body(ByteStream::from_static(content))
+                .customize()
+                .mutate_request(move |req| {
+                    req.headers_mut().insert(header, good.clone());
+                })
+                .send()
+                .await;
+            assert!(put.is_ok(), "{header}: correct checksum must be accepted: {:?}", put.err());
+            let got = client
+                .get_object()
+                .bucket(bucket)
+                .key(&ok_key)
+                .send()
+                .await
+                .expect("GetObject");
+            let body = got.body.collect().await.expect("collect body").into_bytes();
+            assert_eq!(body.as_ref(), content, "{header}: stored body must match uploaded content");
+
+            // Wrong checksum -> rejected with BadDigest, and nothing is stored.
+            let bad = Checksum::new_from_data(ty, b"a totally different payload")
+                .expect("compute checksum")
+                .encoded;
+            let bad_key = format!("bad-{header}");
+            let put_bad = client
+                .put_object()
+                .bucket(bucket)
+                .key(&bad_key)
+                .body(ByteStream::from_static(content))
+                .customize()
+                .mutate_request(move |req| {
+                    req.headers_mut().insert(header, bad.clone());
+                })
+                .send()
+                .await;
+            assert!(put_bad.is_err(), "{header}: a mismatched checksum must be rejected");
+            let msg = format!("{:?}", put_bad.err().unwrap());
+            assert!(
+                msg.contains("BadDigest") || msg.to_lowercase().contains("digest") || msg.to_lowercase().contains("checksum"),
+                "{header}: expected a BadDigest/checksum error, got: {msg}"
+            );
+            let head = client.head_object().bucket(bucket).key(&bad_key).send().await;
+            assert!(head.is_err(), "{header}: nothing must be stored after a rejected PutObject");
+
+            info!("PASSED additional-checksum verify-on-write: {header}");
+        }
+    }
 }
@@ -0,0 +1,108 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Smoke tests for the multi-drive / multi-pool cluster harness.
+//!
+//! These belong to the nightly 4-node cluster lane (they spin up real RustFS
+//! processes, so they are excluded from the merge-gate `e2e-full` profile in
+//! `.config/nextest.toml`). They assert that:
+//!
+//! * `drivesPerNode > 1` produces a bootable single-pool cluster whose
+//!   `RUSTFS_VOLUMES` string enumerates every drive, and that a PUT/GET
+//!   round-trips through the multi-drive erasure layout.
+//! * A two-pool topology (one node per pool, `drivesPerNode` drives each) boots
+//!   with the ellipses `RUSTFS_VOLUMES` form and round-trips a PUT/GET.
+//!
+//! Readiness is established by the harness's `start()` handshake (TCP reachability
+//! plus an S3 `ListBuckets` poll) — there are no fixed sleeps.
+//!
+//! Out of scope for this block (tracked separately): network fault injection
+//! (toxiproxy / socket proxy) and 5GiB large-object budgets.
+
+use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
+use serial_test::serial;
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+const BUCKET: &str = "cluster-multidrive-pool-bucket";
+
+/// PUT an object then GET it back and assert the bytes round-trip.
+async fn put_get_roundtrip(cluster: &RustFSTestClusterEnvironment, key: &str, payload: &[u8]) -> TestResult {
+    let writer = cluster.create_s3_client(0)?;
+    writer
+        .put_object()
+        .bucket(BUCKET)
+        .key(key)
+        .body(bytes::Bytes::copy_from_slice(payload).into())
+        .send()
+        .await?;
+
+    // Read back from the last node to exercise the cross-node/cross-pool path.
+    let reader = cluster.create_s3_client(cluster.nodes.len() - 1)?;
+    let got = reader.get_object().bucket(BUCKET).key(key).send().await?;
+    let body = got.body.collect().await?.into_bytes();
+    assert_eq!(body.as_ref(), payload, "round-tripped object bytes must match");
+    Ok(())
+}
+
+/// 4 nodes x 2 drives, single pool: the multi-drive layout boots and round-trips.
+#[tokio::test]
+#[serial]
+async fn cluster_multidrive_single_pool_smoke() -> TestResult {
+    crate::common::init_logging();
+
+    let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 2)).await?;
+
+    // The single-pool multi-drive layout must list every (node, drive) endpoint
+    // explicitly (8 endpoints, no ellipses) so the server keeps one legacy pool.
+    let volumes = cluster.rustfs_volumes_arg();
+    assert_eq!(volumes.split(' ').count(), 8, "expected 8 explicit endpoints, got: {volumes}");
+    assert!(!volumes.contains('{'), "single-pool layout must not use ellipses: {volumes}");
+
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let payload = vec![0xA5u8; 512 * 1024];
+    put_get_roundtrip(&cluster, "multidrive/object", &payload).await?;
+    Ok(())
+}
+
+/// Two single-node pools, 2 drives each: the multi-pool layout boots and
+/// round-trips. Every pool is a distinct erasure pool (`pool_idx` 0 and 1).
+#[tokio::test]
+#[serial]
+async fn cluster_two_pool_smoke() -> TestResult {
+    crate::common::init_logging();
+
+    let mut cluster =
+        RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]])).await?;
+
+    // The two-pool layout must emit one ellipses argument per pool.
+    let volumes = cluster.rustfs_volumes_arg();
+    let args: Vec<&str> = volumes.split(' ').collect();
+    assert_eq!(args.len(), 2, "expected two pool arguments, got: {volumes}");
+    assert!(
+        args.iter().all(|a| a.contains("/drive{0...1}")),
+        "each pool arg must use the drive ellipses form: {volumes}"
+    );
+    assert_eq!(cluster.nodes[0].pool_idx, 0);
+    assert_eq!(cluster.nodes[1].pool_idx, 1);
+
+    cluster.start().await?;
+    cluster.create_test_bucket(BUCKET).await?;
+
+    let payload = vec![0x5Au8; 256 * 1024];
+    put_get_roundtrip(&cluster, "twopool/object", &payload).await?;
+    Ok(())
+}
@@ -446,6 +446,13 @@ impl RustFSTestEnvironment {
        self.start_rustfs_server_inner(extra_args, &[], false).await
    }

+    pub async fn start_rustfs_server_without_cleanup_with_env(
+        &mut self,
+        extra_env: &[(&str, &str)],
+    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        self.start_rustfs_server_inner(vec![], extra_env, false).await
+    }
+
    /// Wait for RustFS server to be ready.
    ///
    /// A listening TCP port is not sufficient here: the process may accept
@@ -514,6 +521,30 @@ impl RustFSTestEnvironment {
            }
        }
    }
+
+    /// Restart this server in place, preserving its on-disk data directory,
+    /// listen address, and credentials.
+    ///
+    /// Failure-recovery tests (backlog#1147 repl-5) need the *same* instance to
+    /// come back up after a crash/stop with its persisted state intact:
+    /// per-object replication status in `xl.meta`, the MRF recovery file, and
+    /// resync metadata all live under `temp_dir`, which this method keeps. The
+    /// address is reused too (the server sets `SO_REUSEADDR`/`SO_REUSEPORT`), so
+    /// existing S3 clients keep working without reconfiguration.
+    ///
+    /// The previous process is stopped first. `cleanup_existing` is false so a
+    /// sibling server sharing the harness (e.g. a replication target on another
+    /// port) is left untouched — only this instance is recycled. Pass the same
+    /// `extra_args` / `extra_env` the instance was originally started with;
+    /// background tasks read their env once at startup.
+    pub async fn restart_server_preserving_data(
+        &mut self,
+        extra_args: Vec<&str>,
+        extra_env: &[(&str, &str)],
+    ) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        self.stop_server();
+        self.start_rustfs_server_inner(extra_args, extra_env, false).await
+    }
 }

 impl Drop for RustFSTestEnvironment {
@@ -686,15 +717,166 @@ pub async fn awscurl_delete(
    execute_awscurl(url, "DELETE", None, access_key, secret_key).await
 }

+/// Cluster topology declaration for a `RustFSTestClusterEnvironment`.
+///
+/// Describes how many nodes to launch, how many data drives each node exposes,
+/// and how those nodes are grouped into erasure pools. The topology drives both
+/// on-disk directory creation and the `RUSTFS_VOLUMES` string assembled for the
+/// node processes.
+///
+/// # Single-host expressibility constraints
+///
+/// The harness runs every node on `127.0.0.1` with a distinct port, so the
+/// `RUSTFS_VOLUMES` string can only use the two forms the server parser accepts
+/// on a single machine (verified against `ecstore` `DisksLayout::from_volumes`):
+///
+/// * **Single pool** — every `(node, drive)` endpoint listed explicitly, no
+///   ellipses. This is the legacy form and yields exactly one pool spanning all
+///   nodes and drives (`DistErasure`). Any `drives_per_node >= 1` works.
+/// * **Multiple pools** — one ellipses arg per pool. A pool argument is a single
+///   URL template, so it can enumerate several drives on *one* host but cannot
+///   enumerate multiple distinct-port hosts. A pool that spanned two localhost
+///   nodes would need a host ellipses (`127.0.0.1:900{0...1}`), which forces the
+///   *same* on-disk path across those ports and collides physically. Therefore
+///   every pool in a multi-pool topology owns exactly one node. In addition, the
+///   parser rejects a single-drive ellipses pool (`drive{0...0}`), so a
+///   multi-pool topology requires `drives_per_node >= 2`.
+///
+/// Genuine multi-node pools (a pool striped across several hosts) require real
+/// multi-host infrastructure and are deferred to the nightly cluster CI lane
+/// (backlog #1313 / #1314); they are intentionally not expressible here.
+#[derive(Clone, Debug)]
+pub struct ClusterTopology {
+    /// Number of node processes to launch.
+    pub node_count: usize,
+    /// Number of independent data drives (directories) each node exposes.
+    pub drives_per_node: usize,
+    /// Pool membership as a list of node-index groups. An empty vector means a
+    /// single pool spanning every node.
+    pub pools: Vec<Vec<usize>>,
+}
+
+impl ClusterTopology {
+    /// Single pool, one drive per node — identical to the historical
+    /// `RustFSTestClusterEnvironment::new` layout.
+    pub fn single_pool(node_count: usize) -> Self {
+        Self {
+            node_count,
+            drives_per_node: 1,
+            pools: Vec::new(),
+        }
+    }
+
+    /// Single pool with `drives_per_node` drives on every node. The pool spans
+    /// all nodes and all drives (one `DistErasure` pool).
+    pub fn single_pool_multidrive(node_count: usize, drives_per_node: usize) -> Self {
+        Self {
+            node_count,
+            drives_per_node,
+            pools: Vec::new(),
+        }
+    }
+
+    /// Multi-pool topology where every pool owns exactly one node. `pools` lists
+    /// the node index backing each pool (e.g. `vec![vec![0], vec![1]]` for a
+    /// two-pool cluster). Requires `drives_per_node >= 2` (see the type-level
+    /// note on single-host expressibility).
+    pub fn per_node_pools(drives_per_node: usize, pools: Vec<Vec<usize>>) -> Self {
+        let node_count = pools.iter().flatten().copied().max().map(|m| m + 1).unwrap_or(0);
+        Self {
+            node_count,
+            drives_per_node,
+            pools,
+        }
+    }
+
+    /// Normalized pool membership: an empty `pools` becomes a single pool over
+    /// all node indices.
+    fn normalized_pools(&self) -> Vec<Vec<usize>> {
+        if self.pools.is_empty() {
+            vec![(0..self.node_count).collect()]
+        } else {
+            self.pools.clone()
+        }
+    }
+
+    /// Validate that the topology is expressible on a single localhost machine.
+    fn validate(&self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        if self.node_count == 0 {
+            return Err("Node count must be greater than zero".into());
+        }
+        if self.drives_per_node == 0 {
+            return Err("drives_per_node must be greater than zero".into());
+        }
+
+        let pools = self.normalized_pools();
+
+        // Every referenced node index must be in range.
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            if nodes.is_empty() {
+                return Err(format!("pool {pool_idx} has no nodes").into());
+            }
+            for &n in nodes {
+                if n >= self.node_count {
+                    return Err(format!("pool {pool_idx} references node {n} but node_count is {}", self.node_count).into());
+                }
+            }
+        }
+
+        // Each node must belong to exactly one pool.
+        let mut seen = vec![0usize; self.node_count];
+        for nodes in &pools {
+            for &n in nodes {
+                seen[n] += 1;
+            }
+        }
+        for (n, count) in seen.iter().enumerate() {
+            match count {
+                0 => return Err(format!("node {n} is not assigned to any pool").into()),
+                1 => {}
+                _ => return Err(format!("node {n} is assigned to more than one pool").into()),
+            }
+        }
+
+        // Multi-pool constraints imposed by the single-host RUSTFS_VOLUMES syntax.
+        if pools.len() > 1 {
+            if self.drives_per_node < 2 {
+                return Err(
+                    "multi-pool topology requires drives_per_node >= 2 (the server parser rejects a single-drive ellipses pool)"
+                        .into(),
+                );
+            }
+            for (pool_idx, nodes) in pools.iter().enumerate() {
+                if nodes.len() != 1 {
+                    return Err(format!(
+                        "pool {pool_idx} spans {} nodes; a pool striped across multiple localhost nodes is not expressible (needs host ellipses, which collides on shared paths). Use one node per pool, or the nightly multi-host lane (backlog #1313/#1314).",
+                        nodes.len()
+                    )
+                    .into());
+                }
+            }
+        }
+
+        Ok(())
+    }
+}
+
 /// Represents a single RustFS server instance in a test cluster.
 ///
 /// Each `ClusterNode` tracks the node's network address, base URL for
-/// S3-compatible requests, on-disk data directory, and the underlying
+/// S3-compatible requests, on-disk data directories, and the underlying
 /// child process handle when the node is running.
 pub struct ClusterNode {
    pub address: String,
    pub url: String,
+    /// Primary data directory for the node. For single-drive nodes this is the
+    /// node's only drive; for multi-drive nodes it is the first drive. Kept as a
+    /// stable field so existing single-drive tests continue to compile.
    pub data_dir: String,
+    /// All data drives exposed by this node (`data_dirs[0] == data_dir`).
+    pub data_dirs: Vec<String>,
+    /// Index of the pool this node belongs to.
+    pub pool_idx: usize,
    pub process: Option<Child>,
 }

@@ -710,6 +892,7 @@ pub struct RustFSTestClusterEnvironment {
    pub access_key: String,
    pub secret_key: String,
    pub extra_env: Vec<(String, String)>,
+    pub topology: ClusterTopology,
 }

 impl RustFSTestClusterEnvironment {
@@ -731,34 +914,82 @@ impl RustFSTestClusterEnvironment {
    /// * `Err(Box<dyn Error + Send + Sync>)` - An error if any step fails, such as temporary
    ///   directory creation failure or available port lookup failure.
    pub async fn new(node_count: usize) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
-        if node_count == 0 {
-            return Err("Node count must be greater than zero".into());
-        }
+        Self::with_topology(ClusterTopology::single_pool(node_count)).await
+    }
+
+    /// Create a RustFS test cluster environment from an explicit topology.
+    ///
+    /// Allocates a unique temporary root directory, an available TCP port per
+    /// node, and one data directory per drive. The topology is validated up front
+    /// (node/pool assignment, single-host multi-pool constraints); node processes
+    /// are not started at this stage.
+    ///
+    /// When the topology exposes more than one local drive on a node (multi-drive
+    /// or multi-pool layouts), `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` is added to
+    /// every node's environment: those drives live on the same temp filesystem, so
+    /// the server's distinct-physical-disk safety check would otherwise reject
+    /// startup. Single-drive single-pool clusters keep the historical environment
+    /// unchanged.
+    pub async fn with_topology(topology: ClusterTopology) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
+        topology.validate()?;
+
        let temp_dir = format!("/tmp/rustfs_cluster_test_{}", Uuid::new_v4());
        fs::create_dir_all(&temp_dir).await?;

-        let mut nodes = Vec::with_capacity(node_count);
-        for i in 0..node_count {
+        // Map every node to its owning pool index.
+        let pools = topology.normalized_pools();
+        let mut pool_of_node = vec![0usize; topology.node_count];
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            for &n in nodes {
+                pool_of_node[n] = pool_idx;
+            }
+        }
+
+        let multidrive = topology.drives_per_node > 1;
+
+        let mut nodes = Vec::with_capacity(topology.node_count);
+        for (i, &pool_idx) in pool_of_node.iter().enumerate() {
            let port = RustFSTestEnvironment::find_available_port().await?;
            let address = format!("127.0.0.1:{}", port);
            let url = format!("http://{}", address);
-            let data_dir = format!("{}/node{}", temp_dir, i);
-            fs::create_dir_all(&data_dir).await?;
+
+            // Single-drive nodes keep the historical `{temp}/node{i}` path so
+            // existing tests (and their on-disk assertions) are unaffected.
+            // Multi-drive nodes nest one `drive{d}` directory per drive so the
+            // ellipses form `drive{0...N-1}` can address them.
+            let data_dirs: Vec<String> = if multidrive {
+                (0..topology.drives_per_node)
+                    .map(|d| format!("{}/node{}/drive{}", temp_dir, i, d))
+                    .collect()
+            } else {
+                vec![format!("{}/node{}", temp_dir, i)]
+            };
+            for dir in &data_dirs {
+                fs::create_dir_all(dir).await?;
+            }

            nodes.push(ClusterNode {
                address,
                url,
-                data_dir,
+                data_dir: data_dirs[0].clone(),
+                data_dirs,
+                pool_idx,
                process: None,
            });
        }

+        let mut extra_env = Vec::new();
+        if multidrive {
+            extra_env.push(("RUSTFS_UNSAFE_BYPASS_DISK_CHECK".to_string(), "true".to_string()));
+        }
+
        Ok(Self {
            nodes,
            temp_dir,
            access_key: DEFAULT_ACCESS_KEY.to_string(),
            secret_key: DEFAULT_SECRET_KEY.to_string(),
-            extra_env: Vec::new(),
+            extra_env,
+            topology,
        })
    }

@@ -778,14 +1009,51 @@ impl RustFSTestClusterEnvironment {
        Ok(())
    }

-    /// Build the volumes argument string for RustFS binary (internal helper method).
+    /// Build the `RUSTFS_VOLUMES` argument string for the cluster's topology.
    ///
-    /// Concatenates the address and data directory of all cluster nodes into a single string
-    /// used as the `RUSTFS_VOLUMES` environment variable for RustFS node processes.
+    /// * **Single pool** — every `(node, drive)` endpoint is listed explicitly and
+    ///   space-joined. With no ellipses the server parser collapses them into one
+    ///   legacy pool spanning all nodes and drives.
+    /// * **Multiple pools** — each pool (exactly one node) contributes one ellipses
+    ///   argument `http://<addr><node-base>/drive{0...N-1}`; the space-separated
+    ///   arguments become one pool each.
+    ///
+    /// The server splits `RUSTFS_VOLUMES` on spaces (`value_delimiter = ' '`), which
+    /// this assembly matches, and the resulting layout is verified against the
+    /// `ecstore` parser in the unit tests below.
+    /// Public view of the assembled `RUSTFS_VOLUMES` string, so tests can assert
+    /// the pool/drive layout without starting node processes.
+    pub fn rustfs_volumes_arg(&self) -> String {
+        self.build_volumes_arg()
+    }
+
    fn build_volumes_arg(&self) -> String {
-        self.nodes
+        let pools = self.topology.normalized_pools();
+
+        if pools.len() <= 1 {
+            // Single pool: explicit enumeration of every drive on every node.
+            return self
+                .nodes
+                .iter()
+                .flat_map(|n| n.data_dirs.iter().map(move |dir| format!("http://{}{}", n.address, dir)))
+                .collect::<Vec<_>>()
+                .join(" ");
+        }
+
+        // Multi-pool: one ellipses argument per single-node pool. The drive
+        // directories are `<temp>/node{i}/drive{0..N-1}`, so the ellipses base is
+        // the shared parent of the node's drives.
+        pools
            .iter()
-            .map(|n| format!("http://{}{}", n.address, n.data_dir))
+            .map(|nodes| {
+                let node = &self.nodes[nodes[0]];
+                let base = node
+                    .data_dirs
+                    .first()
+                    .and_then(|d| d.rsplit_once('/').map(|(parent, _)| parent))
+                    .unwrap_or(&node.data_dir);
+                format!("http://{}{}/drive{{0...{}}}", node.address, base, self.topology.drives_per_node - 1)
+            })
            .collect::<Vec<_>>()
            .join(" ")
    }
@@ -1060,4 +1328,130 @@ mod tests {

        stdfs::remove_file(stamp_path).ok();
    }
+
+    /// Build a cluster environment struct in-memory (no ports, no processes) so
+    /// that `build_volumes_arg` can be exercised as a pure string builder. Node
+    /// directories mirror what `with_topology` would create for the topology.
+    fn fake_cluster(topology: ClusterTopology) -> RustFSTestClusterEnvironment {
+        let temp_dir = "/tmp/rustfs_cluster_test_FAKE".to_string();
+        let pools = topology.normalized_pools();
+        let mut pool_of_node = vec![0usize; topology.node_count];
+        for (pool_idx, nodes) in pools.iter().enumerate() {
+            for &n in nodes {
+                pool_of_node[n] = pool_idx;
+            }
+        }
+        let multidrive = topology.drives_per_node > 1;
+
+        let nodes = (0..topology.node_count)
+            .map(|i| {
+                let address = format!("127.0.0.1:{}", 9000 + i);
+                let data_dirs: Vec<String> = if multidrive {
+                    (0..topology.drives_per_node)
+                        .map(|d| format!("{}/node{}/drive{}", temp_dir, i, d))
+                        .collect()
+                } else {
+                    vec![format!("{}/node{}", temp_dir, i)]
+                };
+                ClusterNode {
+                    url: format!("http://{}", address),
+                    address,
+                    data_dir: data_dirs[0].clone(),
+                    data_dirs,
+                    pool_idx: pool_of_node[i],
+                    process: None,
+                }
+            })
+            .collect();
+
+        RustFSTestClusterEnvironment {
+            nodes,
+            temp_dir,
+            access_key: DEFAULT_ACCESS_KEY.to_string(),
+            secret_key: DEFAULT_SECRET_KEY.to_string(),
+            extra_env: Vec::new(),
+            topology,
+        }
+    }
+
+    #[test]
+    fn volumes_single_pool_single_drive_is_backward_compatible() {
+        // The historical layout: one explicit endpoint per node, space-joined,
+        // no ellipses, no `drive` sub-directory.
+        let env = fake_cluster(ClusterTopology::single_pool(4));
+        assert_eq!(
+            env.build_volumes_arg(),
+            "http://127.0.0.1:9000/tmp/rustfs_cluster_test_FAKE/node0 \
+             http://127.0.0.1:9001/tmp/rustfs_cluster_test_FAKE/node1 \
+             http://127.0.0.1:9002/tmp/rustfs_cluster_test_FAKE/node2 \
+             http://127.0.0.1:9003/tmp/rustfs_cluster_test_FAKE/node3"
+        );
+    }
+
+    #[test]
+    fn volumes_single_pool_multidrive_enumerates_every_drive() {
+        // 4 nodes x 2 drives -> 8 explicit endpoints, one legacy pool. No
+        // ellipses, so the server parser keeps this as a single DistErasure pool.
+        let env = fake_cluster(ClusterTopology::single_pool_multidrive(4, 2));
+        let expected = (0..4)
+            .flat_map(|i| {
+                (0..2).map(move |d| format!("http://127.0.0.1:{}/tmp/rustfs_cluster_test_FAKE/node{}/drive{}", 9000 + i, i, d))
+            })
+            .collect::<Vec<_>>()
+            .join(" ");
+        assert_eq!(env.build_volumes_arg(), expected);
+        assert_eq!(env.build_volumes_arg().split(' ').count(), 8);
+    }
+
+    #[test]
+    fn volumes_two_pool_uses_one_ellipses_arg_per_pool() {
+        // Two single-node pools, 2 drives each -> two ellipses arguments. The
+        // server parser treats each space-separated ellipses arg as its own pool.
+        let env = fake_cluster(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]));
+        assert_eq!(
+            env.build_volumes_arg(),
+            "http://127.0.0.1:9000/tmp/rustfs_cluster_test_FAKE/node0/drive{0...1} \
+             http://127.0.0.1:9001/tmp/rustfs_cluster_test_FAKE/node1/drive{0...1}"
+        );
+    }
+
+    #[test]
+    fn topology_rejects_multi_node_pool() {
+        // A pool striped across two localhost nodes is not expressible.
+        let err = ClusterTopology::per_node_pools(2, vec![vec![0, 1], vec![2, 3]])
+            .validate()
+            .unwrap_err()
+            .to_string();
+        assert!(err.contains("not expressible"), "unexpected error: {err}");
+    }
+
+    #[test]
+    fn topology_rejects_single_drive_multi_pool() {
+        // The server parser rejects a single-drive ellipses pool (`drive{0...0}`).
+        let err = ClusterTopology::per_node_pools(1, vec![vec![0], vec![1]])
+            .validate()
+            .unwrap_err()
+            .to_string();
+        assert!(err.contains("drives_per_node >= 2"), "unexpected error: {err}");
+    }
+
+    #[test]
+    fn topology_rejects_unassigned_and_duplicated_nodes() {
+        // Node 2 is never assigned to a pool.
+        let mut t = ClusterTopology::single_pool_multidrive(3, 2);
+        t.pools = vec![vec![0], vec![1]];
+        assert!(t.validate().unwrap_err().to_string().contains("not assigned"));
+
+        // Node 0 assigned twice.
+        let mut t = ClusterTopology::single_pool_multidrive(2, 2);
+        t.pools = vec![vec![0], vec![0]];
+        assert!(t.validate().unwrap_err().to_string().contains("more than one pool"));
+    }
+
+    #[test]
+    fn topology_single_pool_accepts_any_drive_count() {
+        assert!(ClusterTopology::single_pool(4).validate().is_ok());
+        assert!(ClusterTopology::single_pool_multidrive(4, 4).validate().is_ok());
+        assert!(ClusterTopology::single_pool_multidrive(1, 1).validate().is_ok());
+    }
 }
@@ -0,0 +1,191 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Over-the-wire smoke net for the embedded console listener (backlog#1154
+//! peri-4). The console is a second HTTP listener that serves the full S3/admin
+//! surface plus the unauthenticated console routes under `/rustfs/console`;
+//! before this test its behaviour was only covered by in-process router tests
+//! (`rustfs/src/admin/console.rs`), so a misconfiguration that exposed
+//! credentials through the public console endpoints or fail-opened the
+//! protected surface on the console port had no regression net.
+//!
+//! Pinned wire contract (real binary, real TCP):
+//!   * `/rustfs/console/version` and `/rustfs/console/license` answer 200 JSON
+//!     without authentication, with complete fields and no credential material.
+//!   * `/rustfs/console/license` exposes only the coarse `licensed` flag.
+//!   * the console SPA prefix dispatches to the static handler, never to the
+//!     S3 API (no S3 error XML), whether or not console assets are embedded.
+//!   * unauthenticated requests to the admin API and the S3 root on the console
+//!     listener are denied (403 AccessDenied) — the extra listener does not
+//!     fail-open the protected surface.
+//!   * a listener with the console disabled (the main S3 port here) does not
+//!     serve the unauthenticated console endpoints at all.
+
+use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use serial_test::serial;
+use std::error::Error;
+use tokio::time::{Duration, sleep};
+
+type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
+
+/// Polls the console version endpoint until the console listener accepts
+/// requests. The harness readiness check only covers the S3 listener; the
+/// console listener of the same process may come up moments later.
+async fn wait_for_console_ready(console_base: &str) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let client = local_http_client();
+    let url = format!("{console_base}/rustfs/console/version");
+    let mut last_err = String::new();
+    for _ in 0..40 {
+        match client.get(&url).send().await {
+            Ok(response) if response.status() == reqwest::StatusCode::OK => return Ok(response),
+            Ok(response) => last_err = format!("status {}", response.status()),
+            Err(err) => last_err = err.to_string(),
+        }
+        sleep(Duration::from_millis(250)).await;
+    }
+    Err(format!("console listener at {console_base} never became ready: {last_err}").into())
+}
+
+#[tokio::test]
+#[serial]
+async fn test_console_over_the_wire_smoke() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    // A unique secret makes the credential-leak assertions below sharp: any
+    // occurrence of it in a console response body is a genuine leak, not a
+    // collision with a common default string.
+    env.access_key = format!("peri4ak{}", uuid::Uuid::new_v4().simple());
+    env.secret_key = format!("peri4sk{}", uuid::Uuid::new_v4().simple());
+
+    let console_port = RustFSTestEnvironment::find_available_port().await?;
+    let console_address = format!("127.0.0.1:{console_port}");
+    let console_base = format!("http://{console_address}");
+    env.start_rustfs_server_with_env(
+        vec![],
+        &[
+            ("RUSTFS_CONSOLE_ENABLE", "true"),
+            ("RUSTFS_CONSOLE_ADDRESS", console_address.as_str()),
+        ],
+    )
+    .await?;
+
+    let client = local_http_client();
+
+    // --- /rustfs/console/version: 200 JSON, complete fields, no secrets ------
+    let version_response = wait_for_console_ready(&console_base).await?;
+    let content_type = version_response
+        .headers()
+        .get(reqwest::header::CONTENT_TYPE)
+        .and_then(|v| v.to_str().ok())
+        .unwrap_or_default()
+        .to_string();
+    assert!(content_type.starts_with("application/json"), "version content-type: {content_type}");
+    let version_body = version_response.text().await?;
+    let version_json: serde_json::Value = serde_json::from_str(&version_body)?;
+    for field in ["version", "version_info", "date"] {
+        assert!(
+            version_json[field].as_str().is_some_and(|v| !v.is_empty()),
+            "console version field {field} missing or empty: {version_body}"
+        );
+    }
+    assert!(
+        !version_body.contains(&env.secret_key) && !version_body.contains(&env.access_key),
+        "console version response leaks credentials: {version_body}"
+    );
+
+    // --- /rustfs/console/license: coarse licensed flag only ------------------
+    let license_response = client.get(format!("{console_base}/rustfs/console/license")).send().await?;
+    assert_eq!(license_response.status(), reqwest::StatusCode::OK);
+    let license_body = license_response.text().await?;
+    let license_json: serde_json::Value = serde_json::from_str(&license_body)?;
+    assert!(
+        license_json["licensed"].is_boolean(),
+        "license payload must expose a licensed bool: {license_body}"
+    );
+    let license_keys: Vec<&String> = license_json.as_object().map(|o| o.keys().collect()).unwrap_or_default();
+    assert_eq!(
+        license_keys,
+        vec!["licensed"],
+        "public license endpoint must not expose license metadata: {license_body}"
+    );
+    assert!(
+        !license_body.contains(&env.secret_key) && !license_body.contains(&env.access_key),
+        "console license response leaks credentials: {license_body}"
+    );
+
+    // --- console SPA prefix dispatches to the static handler, not the S3 API -
+    // With release console assets embedded this is 200 text/html; a from-source
+    // binary embeds an empty static dir and serves the handler's 404 fallback.
+    // Either way it must never fall through to an S3 handler (S3 error XML).
+    let spa_response = client.get(format!("{console_base}/rustfs/console/")).send().await?;
+    let spa_status = spa_response.status();
+    let spa_content_type = spa_response
+        .headers()
+        .get(reqwest::header::CONTENT_TYPE)
+        .and_then(|v| v.to_str().ok())
+        .unwrap_or_default()
+        .to_string();
+    let spa_body = spa_response.text().await?;
+    assert!(
+        !spa_body.contains("<Error>"),
+        "console SPA route fell through to the S3 API: status {spa_status}, body {spa_body}"
+    );
+    match spa_status {
+        reqwest::StatusCode::OK => {
+            assert!(
+                spa_content_type.starts_with("text/html"),
+                "embedded console index must be html, got {spa_content_type}"
+            );
+        }
+        reqwest::StatusCode::NOT_FOUND => {
+            assert!(
+                spa_body.contains("RustFS"),
+                "asset-less console 404 must come from the console static handler: {spa_body}"
+            );
+        }
+        other => panic!("console SPA route returned unexpected status {other}: {spa_body}"),
+    }
+
+    // --- protected surface on the console listener stays authenticated -------
+    let admin_response = client.get(format!("{console_base}/rustfs/admin/v3/info")).send().await?;
+    assert_eq!(
+        admin_response.status(),
+        reqwest::StatusCode::FORBIDDEN,
+        "unauthenticated admin API on the console listener must be denied"
+    );
+    let admin_body = admin_response.text().await?;
+    assert!(admin_body.contains("AccessDenied"), "admin denial must be AccessDenied: {admin_body}");
+
+    let s3_root_response = client.get(format!("{console_base}/")).send().await?;
+    assert_eq!(
+        s3_root_response.status(),
+        reqwest::StatusCode::FORBIDDEN,
+        "unauthenticated S3 root on the console listener must be denied"
+    );
+
+    // --- console disabled on a listener means no console surface at all ------
+    // The main S3 listener of this same process runs with the console disabled,
+    // so its console paths must not answer with the unauthenticated console
+    // payloads (they fall through to the authenticated S3 surface instead).
+    let s3_listener_console = client.get(format!("{}/rustfs/console/version", env.url)).send().await?;
+    assert_ne!(
+        s3_listener_console.status(),
+        reqwest::StatusCode::OK,
+        "console endpoints must not be served on a console-disabled listener"
+    );
+
+    env.stop_server();
+    Ok(())
+}
@@ -0,0 +1,11 @@
+# Programmable fake S3 target
+
+This module is the shared failure-injection boundary for replication end-to-end tests. It runs an in-process, path-style S3 endpoint backed by `s3s`; no production crate depends on it.
+
+`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
+
+Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
+
+Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
+
+The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.
@@ -0,0 +1,498 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! In-process, socket-level network fault-injection proxy for black-box
+//! cluster E2E tests (backlog#1325 network fault-injection block).
+//!
+//! `FaultProxy` binds a TCP listener on a random local port, forwards every
+//! accepted connection to a fixed target address, and lets a test flip the
+//! forwarding behaviour at runtime. It is the black-box counterpart to the
+//! in-process white-box hooks (rename barrier, disk call counters): those
+//! `#[cfg(test)]` primitives cannot reach across the process boundary into a
+//! spawned RustFS server, whereas this proxy sits on the wire between two
+//! nodes and needs no cooperation from the peer.
+//!
+//! Service targets:
+//! - **#1312 / #1319** lock-plane one-way partition and accept-then-blackhole
+//!   peer: run a node's lock/RPC port through the proxy, then
+//!   [`FaultMode::Partition`] one direction (data plane stays reachable via a
+//!   separate direct port) or [`FaultMode::Blackhole`] the whole connection.
+//! - **#1327** cross-process replay / tamper harness: the proxy is the wire
+//!   seam a replay/tamper filter can later hook into.
+//!
+//! Supported fault modes:
+//! - [`FaultMode::Pass`]: transparent byte forwarding (round-trip identical).
+//! - [`FaultMode::Latency`]: delay every forwarded chunk by a fixed duration.
+//! - [`FaultMode::Blackhole`]: accept the connection but forward nothing in
+//!   either direction and never respond (models an accepted-then-dead peer);
+//!   the client observes a read timeout, the proxy never panics.
+//! - [`FaultMode::Partition`]: block exactly one direction while the other
+//!   keeps flowing (models a lock-RPC one-way partition with a live data
+//!   plane).
+//!
+//! Wiring this into the cluster harness ("expose node port N through a proxy")
+//! is intentionally left as a follow-up: the harness multi-drive / 2-pool
+//! changes land separately (rustfs#4937), so this block ships the proxy tool
+//! plus its local self-tests against a loopback echo server and stays
+//! independent of that harness work.
+//!
+//! # Example
+//!
+//! ```ignore
+//! let proxy = FaultProxy::start(target_addr).await?;
+//! let via = proxy.local_addr(); // point the client here instead of `target_addr`
+//! proxy.set_mode(FaultMode::Latency(Duration::from_millis(50)));
+//! // ... exercise the client ...
+//! proxy.set_mode(FaultMode::Partition(Direction::ClientToServer));
+//! // ... assert the blocked direction is dead ...
+//! proxy.shutdown().await; // releases the listener port
+//! ```
+
+use std::io;
+use std::net::{Ipv4Addr, SocketAddr};
+use std::time::Duration;
+
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::net::{TcpListener, TcpStream};
+use tokio::sync::watch;
+use tokio::task::JoinHandle;
+
+/// A single logical direction of a proxied connection.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Direction {
+    /// Bytes travelling from the connecting client towards the target server
+    /// (the forward / request path).
+    ClientToServer,
+    /// Bytes travelling from the target server back to the client (the return
+    /// / response path).
+    ServerToClient,
+}
+
+/// Runtime-switchable forwarding behaviour of a [`FaultProxy`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum FaultMode {
+    /// Forward bytes transparently in both directions.
+    Pass,
+    /// Forward bytes in both directions, but delay every chunk by this
+    /// duration before it is written onward.
+    Latency(Duration),
+    /// Accept connections but forward nothing in either direction and never
+    /// respond. Bytes read from either side are drained and discarded so the
+    /// proxy never blocks or panics; the peer simply never hears back.
+    Blackhole,
+    /// Block exactly one direction (drop its bytes) while the other direction
+    /// keeps forwarding normally.
+    Partition(Direction),
+}
+
+/// An async TCP proxy that forwards a random local port to a fixed target and
+/// can inject network faults at runtime.
+///
+/// The proxy owns a background accept loop; each accepted connection is
+/// handled by its own task pair (one per direction). [`FaultProxy::shutdown`]
+/// stops the accept loop and drops the listener, releasing the bound port.
+pub struct FaultProxy {
+    listen_addr: SocketAddr,
+    target: SocketAddr,
+    mode_tx: watch::Sender<FaultMode>,
+    shutdown_tx: watch::Sender<bool>,
+    accept_task: JoinHandle<()>,
+}
+
+impl FaultProxy {
+    /// Bind a listener on `127.0.0.1:0` and start forwarding accepted
+    /// connections to `target`. Starts in [`FaultMode::Pass`].
+    pub async fn start(target: SocketAddr) -> io::Result<Self> {
+        let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
+        let listen_addr = listener.local_addr()?;
+
+        let (mode_tx, mode_rx) = watch::channel(FaultMode::Pass);
+        let (shutdown_tx, shutdown_rx) = watch::channel(false);
+
+        let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx));
+
+        Ok(Self {
+            listen_addr,
+            target,
+            mode_tx,
+            shutdown_tx,
+            accept_task,
+        })
+    }
+
+    /// Local address the proxy is listening on. Point clients here instead of
+    /// the real target to route their traffic through the proxy.
+    pub fn local_addr(&self) -> SocketAddr {
+        self.listen_addr
+    }
+
+    /// The fixed target address every connection is forwarded to.
+    pub fn target_addr(&self) -> SocketAddr {
+        self.target
+    }
+
+    /// The mode currently in effect.
+    pub fn mode(&self) -> FaultMode {
+        *self.mode_tx.borrow()
+    }
+
+    /// Switch the fault mode at runtime. Takes effect on the next chunk read by
+    /// each direction of every live and future connection.
+    pub fn set_mode(&self, mode: FaultMode) {
+        // A send only fails if every receiver has dropped, i.e. the accept loop
+        // and all connections have already ended; there is nothing to steer.
+        let _ = self.mode_tx.send(mode);
+    }
+
+    /// Stop the accept loop, signal live connections to wind down, and drop the
+    /// listener so the bound port is released. Awaits the accept loop so the
+    /// port is free once this returns.
+    pub async fn shutdown(self) {
+        let _ = self.shutdown_tx.send(true);
+        let _ = self.accept_task.await;
+    }
+}
+
+/// Accept loop: takes new connections until shutdown is signalled, then returns
+/// (dropping `listener`, which releases the port).
+async fn accept_loop(
+    listener: TcpListener,
+    target: SocketAddr,
+    mode_rx: watch::Receiver<FaultMode>,
+    mut shutdown_rx: watch::Receiver<bool>,
+) {
+    loop {
+        tokio::select! {
+            biased;
+            _ = shutdown_rx.changed() => {
+                if *shutdown_rx.borrow() {
+                    break;
+                }
+            }
+            accepted = listener.accept() => {
+                match accepted {
+                    Ok((client, _peer)) => {
+                        tokio::spawn(handle_connection(
+                            client,
+                            target,
+                            mode_rx.clone(),
+                            shutdown_rx.clone(),
+                        ));
+                    }
+                    // A transient accept error should not tear the proxy down.
+                    Err(_) => continue,
+                }
+            }
+        }
+    }
+}
+
+/// Bridge one accepted client connection to a fresh connection to the target,
+/// running an independent pump per direction.
+async fn handle_connection(
+    client: TcpStream,
+    target: SocketAddr,
+    mode_rx: watch::Receiver<FaultMode>,
+    shutdown_rx: watch::Receiver<bool>,
+) {
+    let server = match TcpStream::connect(target).await {
+        Ok(s) => s,
+        // Target unreachable: nothing to bridge; drop the client.
+        Err(_) => return,
+    };
+
+    let (client_rd, client_wr) = client.into_split();
+    let (server_rd, server_wr) = server.into_split();
+
+    let up = tokio::spawn(pump(
+        Direction::ClientToServer,
+        client_rd,
+        server_wr,
+        mode_rx.clone(),
+        shutdown_rx.clone(),
+    ));
+    let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx));
+
+    let _ = up.await;
+    let _ = down.await;
+}
+
+/// Copy bytes from `from` to `to` for a single direction, applying the current
+/// [`FaultMode`]. Returns when the source hits EOF/error, a write fails, or
+/// shutdown is signalled.
+async fn pump<R, W>(
+    dir: Direction,
+    mut from: R,
+    mut to: W,
+    mut mode_rx: watch::Receiver<FaultMode>,
+    mut shutdown_rx: watch::Receiver<bool>,
+) where
+    R: AsyncReadExt + Unpin,
+    W: AsyncWriteExt + Unpin,
+{
+    let mut buf = vec![0u8; 16 * 1024];
+    loop {
+        let n = tokio::select! {
+            biased;
+            _ = shutdown_rx.changed() => break,
+            read = from.read(&mut buf) => match read {
+                Ok(0) => break,        // clean EOF
+                Ok(n) => n,
+                Err(_) => break,       // reset / error: end this direction
+            },
+        };
+
+        // Snapshot the mode without holding the borrow across an await.
+        let mode = *mode_rx.borrow_and_update();
+        match mode {
+            // Whole connection is a black hole: drain and drop.
+            FaultMode::Blackhole => continue,
+            // This direction is partitioned off: drop its bytes; the peer keeps
+            // reading nothing while the other direction may still flow.
+            FaultMode::Partition(blocked) if blocked == dir => continue,
+            FaultMode::Latency(delay) => {
+                tokio::time::sleep(delay).await;
+                if to.write_all(&buf[..n]).await.is_err() {
+                    break;
+                }
+                if to.flush().await.is_err() {
+                    break;
+                }
+            }
+            FaultMode::Pass | FaultMode::Partition(_) => {
+                if to.write_all(&buf[..n]).await.is_err() {
+                    break;
+                }
+                if to.flush().await.is_err() {
+                    break;
+                }
+            }
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::Arc;
+    use std::time::Instant;
+    use tokio::sync::Mutex;
+
+    /// A loopback echo server that records every byte it receives, so tests can
+    /// distinguish "the server never got it" (client->server blocked) from "the
+    /// server got it but the reply never came back" (server->client blocked).
+    struct EchoServer {
+        addr: SocketAddr,
+        received: Arc<Mutex<Vec<u8>>>,
+        _task: JoinHandle<()>,
+    }
+
+    impl EchoServer {
+        async fn start() -> io::Result<Self> {
+            let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
+            let addr = listener.local_addr()?;
+            let received = Arc::new(Mutex::new(Vec::new()));
+            let received_task = received.clone();
+            let task = tokio::spawn(async move {
+                loop {
+                    let Ok((mut sock, _)) = listener.accept().await else {
+                        break;
+                    };
+                    let received_conn = received_task.clone();
+                    tokio::spawn(async move {
+                        let mut buf = vec![0u8; 16 * 1024];
+                        loop {
+                            match sock.read(&mut buf).await {
+                                Ok(0) | Err(_) => break,
+                                Ok(n) => {
+                                    received_conn.lock().await.extend_from_slice(&buf[..n]);
+                                    if sock.write_all(&buf[..n]).await.is_err() {
+                                        break;
+                                    }
+                                    let _ = sock.flush().await;
+                                }
+                            }
+                        }
+                    });
+                }
+            });
+            Ok(Self {
+                addr,
+                received,
+                _task: task,
+            })
+        }
+
+        async fn received(&self) -> Vec<u8> {
+            self.received.lock().await.clone()
+        }
+    }
+
+    /// Write `payload`, then try to read up to `want` bytes with a deadline.
+    /// Returns the bytes actually received before the timeout (possibly empty).
+    async fn write_then_read(stream: &mut TcpStream, payload: &[u8], want: usize, timeout: Duration) -> Vec<u8> {
+        stream.write_all(payload).await.expect("client write");
+        stream.flush().await.expect("client flush");
+
+        let mut out = Vec::new();
+        let mut buf = vec![0u8; want.max(1)];
+        let deadline = tokio::time::Instant::now() + timeout;
+        while out.len() < want {
+            let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
+            if remaining.is_zero() {
+                break;
+            }
+            match tokio::time::timeout(remaining, stream.read(&mut buf)).await {
+                Ok(Ok(0)) => break, // peer closed
+                Ok(Ok(n)) => out.extend_from_slice(&buf[..n]),
+                Ok(Err(_)) => break, // connection error
+                Err(_) => break,     // read timed out
+            }
+        }
+        out
+    }
+
+    #[tokio::test]
+    async fn pass_mode_round_trips_bytes() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        assert_eq!(proxy.mode(), FaultMode::Pass);
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"hello-fault-proxy";
+        let got = write_then_read(&mut client, payload, payload.len(), Duration::from_secs(5)).await;
+
+        assert_eq!(got, payload, "pass mode must forward bytes unchanged");
+        assert_eq!(echo.received().await, payload, "server must have received the bytes");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn latency_mode_delays_forwarding() {
+        // Loose lower bound only: sleep() guarantees *at least* the delay, so
+        // the assertion cannot flake on a slow machine, and we never assert an
+        // upper bound.
+        let delay = Duration::from_millis(200);
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Latency(delay));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"delayed";
+        let start = Instant::now();
+        let got = write_then_read(&mut client, payload, payload.len(), Duration::from_secs(5)).await;
+        let elapsed = start.elapsed();
+
+        assert_eq!(got, payload, "latency mode must still deliver the bytes");
+        // One delay on the request path plus one on the echo return path: the
+        // round trip must exceed at least a single configured delay.
+        assert!(elapsed >= delay, "round trip {elapsed:?} shorter than configured delay {delay:?}");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn blackhole_mode_yields_no_response_and_no_panic() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Blackhole);
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let got = write_then_read(&mut client, b"into-the-void", 1, Duration::from_millis(300)).await;
+
+        assert!(got.is_empty(), "blackhole mode must not return any bytes, got {got:?}");
+        assert!(echo.received().await.is_empty(), "blackhole must not forward to the server");
+
+        // Proxy is still alive and steerable after a blackholed connection.
+        proxy.set_mode(FaultMode::Pass);
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn partition_client_to_server_blocks_request_path() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Partition(Direction::ClientToServer));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let got = write_then_read(&mut client, b"blocked-request", 1, Duration::from_millis(300)).await;
+
+        assert!(got.is_empty(), "client->server partition must yield no echo");
+        assert!(
+            echo.received().await.is_empty(),
+            "client->server partition must stop bytes from reaching the server"
+        );
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn partition_server_to_client_blocks_only_return_path() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        proxy.set_mode(FaultMode::Partition(Direction::ServerToClient));
+
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let payload = b"one-way";
+        let got = write_then_read(&mut client, payload, 1, Duration::from_millis(300)).await;
+
+        // Client hears nothing back...
+        assert!(got.is_empty(), "server->client partition must block the reply");
+        // ...but the request path is live, so the server did receive the bytes.
+        // Poll briefly to avoid racing the forward direction.
+        let mut server_saw = Vec::new();
+        for _ in 0..30 {
+            server_saw = echo.received().await;
+            if !server_saw.is_empty() {
+                break;
+            }
+            tokio::time::sleep(Duration::from_millis(10)).await;
+        }
+        assert_eq!(server_saw, payload, "server->client partition must leave the request path intact");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn set_mode_switches_behaviour_at_runtime() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+
+        // Start passing: first exchange round-trips.
+        let mut client = TcpStream::connect(proxy.local_addr()).await.unwrap();
+        let first = write_then_read(&mut client, b"first", 5, Duration::from_secs(5)).await;
+        assert_eq!(first, b"first", "initial pass exchange must round-trip");
+
+        // Flip to blackhole on the same live connection: the next write gets no reply.
+        proxy.set_mode(FaultMode::Blackhole);
+        let second = write_then_read(&mut client, b"second", 1, Duration::from_millis(300)).await;
+        assert!(second.is_empty(), "after switching to blackhole the live connection must go silent");
+
+        proxy.shutdown().await;
+    }
+
+    #[tokio::test]
+    async fn shutdown_releases_the_listener_port() {
+        let echo = EchoServer::start().await.unwrap();
+        let proxy = FaultProxy::start(echo.addr).await.unwrap();
+        let addr = proxy.local_addr();
+
+        proxy.shutdown().await;
+
+        // The port must be free to bind again once shutdown returns.
+        let rebind = TcpListener::bind(addr).await;
+        assert!(rebind.is_ok(), "shutdown must release the listener port {addr}");
+    }
+}
@@ -23,6 +23,18 @@ pub mod common;
 #[cfg(test)]
 pub mod chaos;

+// Programmable S3 target for replication failure-path tests (backlog#1147 repl-8).
+#[cfg(test)]
+pub mod fake_s3_target;
+
+// Socket-level network fault-injection proxy for black-box cluster tests
+// (backlog#1325 network fault-injection block): latency / blackhole / one-way
+// partition on the wire between nodes. Serves #1312/#1319 (lock-plane one-way
+// partition, accept-then-blackhole peer) and #1327 (cross-process replay/tamper
+// seam); cluster-harness wiring is a follow-up (rustfs#4937).
+#[cfg(test)]
+pub mod fault_proxy;
+
 // Reliability tests built on the fault-injection harness
 #[cfg(test)]
 mod reliability_disk_fault_test;
@@ -67,6 +79,10 @@ mod bucket_policy_check_test;
 #[cfg(test)]
 mod security_boundary_test;

+// Opt-in per-client S3 API rate limiting (backlog#1191)
+#[cfg(test)]
+mod api_rate_limit_test;
+
 // Admin authorization gate: non-admin denial + root-credential lifecycle (backlog#1151 sec-4)
 #[cfg(test)]
 mod admin_auth_test;
@@ -143,6 +159,10 @@ mod object_lock;
 #[cfg(test)]
 mod cluster_concurrency_test;

+// Multi-drive (drivesPerNode) and 2-pool cluster harness smoke tests
+#[cfg(test)]
+mod cluster_multidrive_pool_test;
+
 // PutObject / MultipartUpload with checksum (Content-MD5, x-amz-checksum-*)
 #[cfg(test)]
 mod checksum_upload_test;
@@ -191,6 +211,30 @@ mod stale_multipart_cleanup_cluster_test;
 #[cfg(test)]
 mod object_lambda_test;

+// S3 event-notification webhook delivery end-to-end (backlog#1154 peri-1):
+// configure webhook target -> PutBucketNotificationConfiguration -> object
+// operation -> event delivered, plus filter negatives and store-queue redelivery.
+#[cfg(test)]
+mod notification_webhook_test;
+
+// TLS certificate hot-reload live-listener e2e (backlog#1154 peri-5): swap
+// certificates without a restart, existing sessions survive, bad material is
+// fail-safe (old certificate keeps serving, failure is logged).
+#[cfg(test)]
+mod tls_hot_reload_test;
+
+// Console listener over-the-wire smoke (backlog#1154 peri-4): public console
+// endpoints answer without credentials or leaks, the SPA prefix never falls
+// through to the S3 API, and the protected surface stays authenticated.
+#[cfg(test)]
+mod console_smoke_test;
+
+// Admin IAM management CRUD e2e (backlog#1154 peri-2): user / canned-policy /
+// service-account lifecycle over signed HTTP with data-plane effect assertions,
+// plus non-admin 403 probes per endpoint (sec-4 pattern).
+#[cfg(test)]
+mod admin_iam_crud_test;
+
 // Replication extension end-to-end regression tests
 #[cfg(test)]
 mod replication_extension_test;
@@ -0,0 +1,594 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! End-to-end regression net for the S3 event-notification pipeline
+//! (backlog#1154 peri-1). Proves the full "configure webhook target ->
+//! PutBucketNotificationConfiguration -> object operation -> event delivered"
+//! chain against a real rustfs binary and a real HTTP receiver, which no other
+//! test covers: the target-plugin suites stop at broker integration and the
+//! object-lambda suite only exercises the webhook target as a transform
+//! function, never as an S3 event sink.
+//!
+//! Coverage:
+//!   * PUT / multipart-complete / DELETE each deliver one event with the correct
+//!     eventName, bucket, key, versionId and eTag.
+//!   * prefix/suffix filters drop non-matching keys (rule-engine gate).
+//!   * an event queued while the target endpoint is unreachable is redelivered
+//!     from the on-disk store once the endpoint recovers (store-and-forward).
+
+use crate::common::{RustFSTestEnvironment, init_logging};
+use aws_sdk_s3::Client;
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{
+    BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, Event, FilterRule, FilterRuleName,
+    NotificationConfiguration, NotificationConfigurationFilter, QueueConfiguration, S3KeyFilter, VersioningConfiguration,
+};
+use http::header::{CONTENT_TYPE, HOST};
+use local_ip_address::local_ip;
+use reqwest::StatusCode;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use serde_json::Value;
+use serial_test::serial;
+use std::error::Error;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::net::TcpListener;
+use tokio::sync::mpsc;
+use tokio::task::JoinHandle;
+use tokio::time::{Duration, Instant, timeout};
+
+type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
+type BoxError = Box<dyn Error + Send + Sync>;
+
+/// Region embedded in the target ARN. Only the `id:name` tail of the ARN is used
+/// for target lookup (see `process_queue_configurations`), so the region is
+/// nominal, but it must be present for `ARN::parse` to succeed.
+const NOTIFY_REGION: &str = "us-east-1";
+
+/// Webhook targets are registered as `TargetID { id: <name>, name: "webhook" }`,
+/// so the ARN a notification rule references is
+/// `arn:rustfs:sqs:<region>:<name>:webhook`.
+fn target_arn(target_name: &str) -> String {
+    format!("arn:rustfs:sqs:{NOTIFY_REGION}:{target_name}:webhook")
+}
+
+// ---------------------------------------------------------------------------
+// In-test HTTP event receiver
+// ---------------------------------------------------------------------------
+
+/// Reads one HTTP request off the stream, returning its method and body. Handles
+/// the target's HEAD reachability probe (no body) and POST event deliveries.
+async fn read_http_message(stream: &mut tokio::net::TcpStream) -> Result<(String, Vec<u8>), BoxError> {
+    let mut buffer = Vec::new();
+    let mut chunk = [0_u8; 4096];
+
+    let header_end = loop {
+        let read = stream.read(&mut chunk).await?;
+        if read == 0 {
+            return Err("connection closed before request headers were complete".into());
+        }
+        buffer.extend_from_slice(&chunk[..read]);
+        if let Some(pos) = buffer.windows(4).position(|w| w == b"\r\n\r\n") {
+            break pos;
+        }
+    };
+
+    let header_text = std::str::from_utf8(&buffer[..header_end])?;
+    let mut lines = header_text.split("\r\n");
+    let request_line = lines.next().ok_or("missing request line")?;
+    let method = request_line.split_whitespace().next().ok_or("missing method")?.to_string();
+
+    let mut content_length = 0usize;
+    for line in lines {
+        if let Some((name, value)) = line.split_once(':')
+            && name.trim().eq_ignore_ascii_case("content-length")
+        {
+            content_length = value.trim().parse::<usize>().unwrap_or(0);
+        }
+    }
+
+    let body_offset = header_end + 4;
+    while buffer.len().saturating_sub(body_offset) < content_length {
+        let read = stream.read(&mut chunk).await?;
+        if read == 0 {
+            return Err("connection closed before request body was complete".into());
+        }
+        buffer.extend_from_slice(&chunk[..read]);
+    }
+
+    Ok((method, buffer[body_offset..body_offset + content_length].to_vec()))
+}
+
+/// Drives an accepted listener: answers every request 200 OK (so the target's
+/// HEAD reachability probe reports it online) and forwards each non-empty POST
+/// body, parsed as the S3 event envelope JSON, to `tx`.
+fn serve_event_collector(listener: TcpListener, tx: mpsc::UnboundedSender<Value>) -> JoinHandle<()> {
+    tokio::spawn(async move {
+        loop {
+            let Ok((mut stream, _)) = listener.accept().await else {
+                return;
+            };
+            let tx = tx.clone();
+            tokio::spawn(async move {
+                if let Ok(Ok((method, body))) = timeout(Duration::from_secs(5), read_http_message(&mut stream)).await {
+                    let _ = stream
+                        .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
+                        .await;
+                    let _ = stream.shutdown().await;
+                    if method == "POST"
+                        && !body.is_empty()
+                        && let Ok(value) = serde_json::from_slice::<Value>(&body)
+                    {
+                        let _ = tx.send(value);
+                    }
+                }
+            });
+        }
+    })
+}
+
+/// Binds a fresh collector on a random port. Returns its `/events`
+/// endpoint URL, a receiver of parsed event envelopes, and the serving task.
+async fn spawn_event_collector() -> Result<(String, mpsc::UnboundedReceiver<Value>, JoinHandle<()>), BoxError> {
+    let listener = TcpListener::bind("0.0.0.0:0").await?;
+    let port = listener.local_addr()?.port();
+    let endpoint_ip = local_ip()?;
+    let (tx, rx) = mpsc::unbounded_channel();
+    let handle = serve_event_collector(listener, tx);
+    Ok((format!("http://{endpoint_ip}.nip.io:{port}/events"), rx, handle))
+}
+
+/// Decoded object key of the first record in an event envelope.
+fn event_key(envelope: &Value) -> Option<String> {
+    let raw = envelope["Records"][0]["s3"]["object"]["key"].as_str()?;
+    Some(
+        urlencoding::decode(raw)
+            .map(|d| d.into_owned())
+            .unwrap_or_else(|_| raw.to_string()),
+    )
+}
+
+/// Waits until an event targeting `key` whose `EventName` starts with
+/// `event_prefix` arrives, returning the full envelope. Other envelopes are
+/// discarded, so a lingering duplicate of an earlier event (delivery is
+/// at-least-once) or the create event of a key that is later deleted never
+/// satisfies a wait for the opposite event type.
+async fn wait_for_event(
+    rx: &mut mpsc::UnboundedReceiver<Value>,
+    key: &str,
+    event_prefix: &str,
+    within: Duration,
+) -> Result<Value, BoxError> {
+    let deadline = Instant::now() + within;
+    loop {
+        let remaining = deadline.saturating_duration_since(Instant::now());
+        if remaining.is_zero() {
+            return Err(format!("no {event_prefix}* event for key {key} within {within:?}").into());
+        }
+        match timeout(remaining, rx.recv()).await {
+            Ok(Some(envelope)) => {
+                let key_matches = event_key(&envelope).as_deref() == Some(key);
+                let name_matches = envelope["EventName"].as_str().is_some_and(|n| n.starts_with(event_prefix));
+                if key_matches && name_matches {
+                    return Ok(envelope);
+                }
+            }
+            Ok(None) => return Err("event collector channel closed".into()),
+            Err(_) => return Err(format!("no {event_prefix}* event for key {key} within {within:?}").into()),
+        }
+    }
+}
+
+/// Collects every event until `stop_key` is seen (plus a short grace window to
+/// catch stragglers), or until `max_wait` elapses. Used to prove a negative:
+/// non-matching keys must never appear even though a matching sentinel does.
+async fn collect_until(
+    rx: &mut mpsc::UnboundedReceiver<Value>,
+    stop_key: &str,
+    max_wait: Duration,
+    grace: Duration,
+) -> Vec<Value> {
+    let hard_deadline = Instant::now() + max_wait;
+    let mut soft_deadline: Option<Instant> = None;
+    let mut collected = Vec::new();
+    loop {
+        let deadline = soft_deadline.unwrap_or(hard_deadline).min(hard_deadline);
+        let remaining = deadline.saturating_duration_since(Instant::now());
+        if remaining.is_zero() {
+            break;
+        }
+        match timeout(remaining, rx.recv()).await {
+            Ok(Some(envelope)) => {
+                let matched = event_key(&envelope).as_deref() == Some(stop_key);
+                collected.push(envelope);
+                if matched && soft_deadline.is_none() {
+                    soft_deadline = Some(Instant::now() + grace);
+                }
+            }
+            Ok(None) => break,
+            Err(_) => break,
+        }
+    }
+    collected
+}
+
+// ---------------------------------------------------------------------------
+// Admin target configuration (signed admin HTTP)
+// ---------------------------------------------------------------------------
+
+async fn signed_admin_request(
+    env: &RustFSTestEnvironment,
+    method: http::Method,
+    url: &str,
+    body: Option<Vec<u8>>,
+) -> Result<reqwest::Response, BoxError> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
+    let mut builder = http::Request::builder()
+        .method(method.clone())
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if body.is_some() {
+        builder = builder.header(CONTENT_TYPE, "application/json");
+    }
+
+    let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
+    let signed = sign_v4(
+        builder.body(Body::empty())?,
+        content_len,
+        &env.access_key,
+        &env.secret_key,
+        "",
+        "us-east-1",
+    );
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let mut request = crate::common::local_http_client().request(reqwest_method, url);
+    for (name, value) in signed.headers() {
+        request = request.header(name, value);
+    }
+    if let Some(body) = body {
+        request = request.body(body);
+    }
+    Ok(request.send().await?)
+}
+
+async fn enable_notify_module(env: &RustFSTestEnvironment) -> TestResult {
+    let payload = serde_json::json!({
+        "notify_enabled": true,
+        "audit_enabled": false,
+    });
+    let url = format!("{}/rustfs/admin/v3/module-switches", env.url);
+    let response = signed_admin_request(env, http::Method::PUT, &url, Some(payload.to_string().into_bytes())).await?;
+    if response.status() != StatusCode::OK {
+        let status = response.status();
+        let text = response.text().await.unwrap_or_default();
+        return Err(format!("failed to enable notify module: {status} {text}").into());
+    }
+    Ok(())
+}
+
+/// Registers a webhook notification target with a persistent queue directory, so
+/// delivery goes through the durable store-and-forward path.
+async fn configure_webhook_target(env: &RustFSTestEnvironment, target_name: &str, endpoint: &str) -> TestResult {
+    let queue_dir = format!("{}/notify-queue-{target_name}", env.temp_dir);
+    tokio::fs::create_dir_all(&queue_dir).await?;
+    let payload = serde_json::json!({
+        "key_values": [
+            { "key": "endpoint", "value": endpoint },
+            { "key": "queue_dir", "value": queue_dir },
+        ]
+    });
+    let url = format!("{}/rustfs/admin/v3/target/notify_webhook/{target_name}", env.url);
+    let response = signed_admin_request(env, http::Method::PUT, &url, Some(payload.to_string().into_bytes())).await?;
+    let status = response.status();
+    if status != StatusCode::OK {
+        let text = response.text().await.unwrap_or_default();
+        return Err(format!("failed to configure webhook target {target_name}: {status} {text}").into());
+    }
+    Ok(())
+}
+
+/// Polls the admin target ARN list until the freshly configured target is
+/// registered in the runtime, so notification rules and object events do not
+/// race target activation.
+async fn wait_for_target_registered(env: &RustFSTestEnvironment, target_name: &str) -> TestResult {
+    let suffix = format!(":{target_name}:webhook");
+    let url = format!("{}/rustfs/admin/v3/target/arns", env.url);
+    for _ in 0..80 {
+        let response = signed_admin_request(env, http::Method::GET, &url, None).await?;
+        if response.status() == StatusCode::OK {
+            let arns: Vec<String> = serde_json::from_slice(&response.bytes().await?)?;
+            if arns.iter().any(|arn| arn.ends_with(&suffix)) {
+                return Ok(());
+            }
+        }
+        tokio::time::sleep(Duration::from_millis(250)).await;
+    }
+    Err(format!("target {target_name} was not registered in admin ARNs").into())
+}
+
+/// Binds a bucket to a webhook target for ObjectCreated:*/ObjectRemoved:* events,
+/// filtered to `prefix` + `suffix`.
+async fn put_notification_config(client: &Client, bucket: &str, target_name: &str, prefix: &str, suffix: &str) -> TestResult {
+    let key_filter = S3KeyFilter::builder()
+        .filter_rules(FilterRule::builder().name(FilterRuleName::Prefix).value(prefix).build())
+        .filter_rules(FilterRule::builder().name(FilterRuleName::Suffix).value(suffix).build())
+        .build();
+    let queue = QueueConfiguration::builder()
+        .id(format!("{target_name}-rule"))
+        .queue_arn(target_arn(target_name))
+        .events(Event::from("s3:ObjectCreated:*"))
+        .events(Event::from("s3:ObjectRemoved:*"))
+        .filter(NotificationConfigurationFilter::builder().key(key_filter).build())
+        .build()?;
+    let config = NotificationConfiguration::builder().queue_configurations(queue).build();
+    client
+        .put_bucket_notification_configuration()
+        .bucket(bucket)
+        .notification_configuration(config)
+        .send()
+        .await?;
+    Ok(())
+}
+
+async fn enable_versioning(client: &Client, bucket: &str) -> TestResult {
+    client
+        .put_bucket_versioning()
+        .bucket(bucket)
+        .versioning_configuration(
+            VersioningConfiguration::builder()
+                .status(BucketVersioningStatus::Enabled)
+                .build(),
+        )
+        .send()
+        .await?;
+    Ok(())
+}
+
+fn trimmed_etag(value: Option<&str>) -> Option<String> {
+    value.map(|e| e.trim_matches('"').to_string())
+}
+
+// ---------------------------------------------------------------------------
+// Tests
+// ---------------------------------------------------------------------------
+
+/// PUT / multipart-complete / DELETE each deliver one event with correct fields,
+/// and the prefix/suffix filter drops non-matching keys.
+#[tokio::test]
+#[serial]
+async fn test_webhook_event_delivery_and_filtering() -> TestResult {
+    init_logging();
+
+    let (endpoint, mut rx, handle) = spawn_event_collector().await?;
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    enable_notify_module(&env).await?;
+
+    let bucket = "peri1-events";
+    let target = "peri1events";
+    let client = env.create_s3_client();
+    client.create_bucket().bucket(bucket).send().await?;
+    enable_versioning(&client, bucket).await?;
+
+    configure_webhook_target(&env, target, &endpoint).await?;
+    wait_for_target_registered(&env, target).await?;
+    put_notification_config(&client, bucket, target, "uploads/", ".dat").await?;
+
+    // --- PUT: ObjectCreated:Put with exact eTag + versionId ------------------
+    let put_key = "uploads/report.dat";
+    let put = client
+        .put_object()
+        .bucket(bucket)
+        .key(put_key)
+        .body(ByteStream::from_static(b"peri-1 put body"))
+        .send()
+        .await?;
+    let put_version = put
+        .version_id()
+        .ok_or("PUT response missing versionId (versioning not enabled?)")?;
+
+    let created = wait_for_event(&mut rx, put_key, "s3:ObjectCreated:", Duration::from_secs(20)).await?;
+    let record = &created["Records"][0];
+    let object = &record["s3"]["object"];
+    assert_eq!(
+        created["EventName"].as_str(),
+        Some("s3:ObjectCreated:Put"),
+        "envelope EventName: {created}"
+    );
+    assert!(
+        record["eventName"]
+            .as_str()
+            .is_some_and(|n| n.starts_with("s3:ObjectCreated:")),
+        "record eventName: {record}"
+    );
+    assert_eq!(record["s3"]["bucket"]["name"].as_str(), Some(bucket), "bucket in event: {record}");
+    assert_eq!(object["versionId"].as_str(), Some(put_version), "versionId in event: {object}");
+    assert_eq!(
+        trimmed_etag(object["eTag"].as_str()),
+        trimmed_etag(put.e_tag()),
+        "eTag in event: {object}"
+    );
+
+    // --- Multipart complete: ObjectCreated:CompleteMultipartUpload -----------
+    let mp_key = "uploads/multi.dat";
+    let created_mp = client.create_multipart_upload().bucket(bucket).key(mp_key).send().await?;
+    let upload_id = created_mp.upload_id().ok_or("missing upload id")?;
+    let part = client
+        .upload_part()
+        .bucket(bucket)
+        .key(mp_key)
+        .upload_id(upload_id)
+        .part_number(1)
+        .body(ByteStream::from_static(b"peri-1 multipart part body"))
+        .send()
+        .await?;
+    let complete = client
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(mp_key)
+        .upload_id(upload_id)
+        .multipart_upload(
+            CompletedMultipartUpload::builder()
+                .parts(
+                    CompletedPart::builder()
+                        .part_number(1)
+                        .e_tag(part.e_tag().unwrap_or_default())
+                        .build(),
+                )
+                .build(),
+        )
+        .send()
+        .await?;
+
+    let mp_event = wait_for_event(&mut rx, mp_key, "s3:ObjectCreated:", Duration::from_secs(20)).await?;
+    let mp_record = &mp_event["Records"][0];
+    assert_eq!(
+        mp_event["EventName"].as_str(),
+        Some("s3:ObjectCreated:CompleteMultipartUpload"),
+        "multipart EventName: {mp_event}"
+    );
+    assert_eq!(mp_record["s3"]["bucket"]["name"].as_str(), Some(bucket));
+    assert_eq!(
+        trimmed_etag(mp_record["s3"]["object"]["eTag"].as_str()),
+        trimmed_etag(complete.e_tag()),
+        "multipart eTag in event: {mp_record}"
+    );
+
+    // --- Filter: non-matching prefix and suffix must never be delivered ------
+    let wrong_prefix = "logs/report.dat"; // right suffix, wrong prefix
+    let wrong_suffix = "uploads/report.txt"; // right prefix, wrong suffix
+    let sentinel = "uploads/keep.dat";
+    for key in [wrong_prefix, wrong_suffix, sentinel] {
+        client
+            .put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from_static(b"filter probe"))
+            .send()
+            .await?;
+    }
+    let collected = collect_until(&mut rx, sentinel, Duration::from_secs(20), Duration::from_secs(2)).await;
+    let seen: Vec<String> = collected.iter().filter_map(event_key).collect();
+    assert!(
+        seen.iter().any(|k| k == sentinel),
+        "matching sentinel {sentinel} was not delivered; saw {seen:?}"
+    );
+    assert!(
+        !seen.iter().any(|k| k == wrong_prefix),
+        "wrong-prefix key {wrong_prefix} bypassed the filter; saw {seen:?}"
+    );
+    assert!(
+        !seen.iter().any(|k| k == wrong_suffix),
+        "wrong-suffix key {wrong_suffix} bypassed the filter; saw {seen:?}"
+    );
+
+    // --- DELETE on a versioned bucket: ObjectRemoved:* with delete-marker version
+    let delete = client.delete_object().bucket(bucket).key(put_key).send().await?;
+    let removed = wait_for_event(&mut rx, put_key, "s3:ObjectRemoved:", Duration::from_secs(20)).await?;
+    let removed_record = &removed["Records"][0];
+    assert!(
+        removed["EventName"]
+            .as_str()
+            .is_some_and(|n| n.starts_with("s3:ObjectRemoved:")),
+        "delete EventName: {removed}"
+    );
+    if let Some(marker_version) = delete.version_id() {
+        assert_eq!(
+            removed_record["s3"]["object"]["versionId"].as_str(),
+            Some(marker_version),
+            "delete-marker versionId in event: {removed_record}"
+        );
+    }
+
+    env.stop_server();
+    handle.abort();
+    Ok(())
+}
+
+/// An event queued while the target endpoint is unreachable survives on the
+/// durable store and is redelivered once the endpoint comes back.
+#[tokio::test]
+#[serial]
+#[ignore = "FAILING deterministically on main since it landed (#4821): the target is created but never appears in /rustfs/admin/v3/target/arns, so wait_for_target_registered times out. Quarantined per the flake policy; remove with the fix for rustfs#4852"]
+async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
+    init_logging();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    enable_notify_module(&env).await?;
+
+    let bucket = "peri1-redeliver";
+    let target = "peri1redeliver";
+    let client = env.create_s3_client();
+    client.create_bucket().bucket(bucket).send().await?;
+
+    // Configure the target while its endpoint is reachable so activation and
+    // ARN registration complete deterministically. Registering against a dead
+    // endpoint stalls behind the reachability probe's timeout and flakes the
+    // registration wait on loaded runners (observed on the merge run of
+    // rustfs#4821).
+    let listener = TcpListener::bind("0.0.0.0:0").await?;
+    let port = listener.local_addr()?.port();
+    let endpoint_ip = local_ip()?;
+    let endpoint = format!("http://{endpoint_ip}.nip.io:{port}/events");
+    let (setup_tx, _setup_rx) = mpsc::unbounded_channel();
+    let setup_handle = serve_event_collector(listener, setup_tx);
+
+    configure_webhook_target(&env, target, &endpoint).await?;
+    wait_for_target_registered(&env, target).await?;
+    put_notification_config(&client, bucket, target, "uploads/", ".dat").await?;
+
+    // Take the endpoint down (drops the listener, so connections are refused —
+    // a retryable NotConnected), then PUT: the event cannot be delivered and
+    // must survive on the durable queue store.
+    setup_handle.abort();
+    let _ = setup_handle.await;
+
+    let key = "uploads/redeliver.dat";
+    client
+        .put_object()
+        .bucket(bucket)
+        .key(key)
+        .body(ByteStream::from_static(b"queued while target down"))
+        .send()
+        .await?;
+
+    // Hold the endpoint down long enough for at least one replay attempt to
+    // fail (the replay worker scans the store every 500ms), so recovery below
+    // exercises real redelivery rather than a first-attempt success.
+    tokio::time::sleep(Duration::from_secs(2)).await;
+
+    // Bring the endpoint back on the same port; the replay worker retries with
+    // exponential backoff and delivers the queued event.
+    let listener = TcpListener::bind(("0.0.0.0", port)).await?;
+    let (tx, mut rx) = mpsc::unbounded_channel();
+    let handle = serve_event_collector(listener, tx);
+
+    let redelivered = wait_for_event(&mut rx, key, "s3:ObjectCreated:", Duration::from_secs(45)).await?;
+    assert_eq!(
+        redelivered["EventName"].as_str(),
+        Some("s3:ObjectCreated:Put"),
+        "redelivered EventName: {redelivered}"
+    );
+    assert_eq!(redelivered["Records"][0]["s3"]["bucket"]["name"].as_str(), Some(bucket));
+
+    env.stop_server();
+    handle.abort();
+    Ok(())
+}
@@ -22,3 +22,4 @@ mod lifecycle;
 mod lock;
 mod node_interact_test;
 mod sql;
+mod tiering;
@@ -0,0 +1,380 @@
+#![cfg(test)]
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Hermetic ILM transition main-path end-to-end test (backlog#1148 ilm-7).
+//!
+//! Two embedded `rustfs` servers, each with independent credentials, port and
+//! data directory:
+//!
+//! * `cold` — a second RustFS server wired as a [`TierType::RustFS`] remote tier.
+//! * `hot` — the source server that transitions objects to `cold`.
+//!
+//! There are no containers, no external S3 backend and no `awscurl`: the
+//! `AddTier` admin call is signed in-process with `rustfs_signer`, exactly like
+//! the other admin-API e2e suites in this crate. The RustFS warm backend has no
+//! loopback/SSRF restriction (that guard is replication-only), so `hot` can tier
+//! to `cold` over `http://127.0.0.1:<port>`.
+//!
+//! A single test drives the full transition main path and pins the chain
+//! required by ilm-7:
+//!   1. `AddTier(RustFS)` on `hot` targeting `cold` — the real connectivity /
+//!      in-use probe runs (no `force`), so this also proves the tier is reachable.
+//!   2. A `Transition Days=0` rule installed before a multipart PUT transitions
+//!      the object immediately (the completion path enqueues it; the 1s scanner
+//!      cycle is only a backstop).
+//!   3. `HEAD` reports `x-amz-storage-class == <tier name>` and no `x-amz-restore`.
+//!   4. `GET` streams byte-identical data back through the warm backend, and the
+//!      content-type and user metadata survive the round trip (rustfs#2246).
+//!   5. Range `GET` within a part and across the part boundary read the correct
+//!      bytes from the tier (backlog#807).
+//!   6. The remote object is present in the cold-tier bucket after transition.
+//!   7. `DeleteObject` on `hot` drives free-version cleanup: the cold-tier copy
+//!      eventually disappears and the hot object is gone (no local residue).
+
+use crate::common::{RustFSTestEnvironment, local_http_client};
+use aws_sdk_s3::Client;
+use aws_sdk_s3::error::ProvideErrorMetadata;
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{
+    BucketLifecycleConfiguration, CompletedMultipartUpload, CompletedPart, ExpirationStatus, LifecycleRule, LifecycleRuleFilter,
+    Transition, TransitionStorageClass,
+};
+use http::Method;
+use http::header::HOST;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
+use std::time::{Duration as StdDuration, Instant};
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+const TIER_NAME: &str = "COLDTIER";
+const TIER_BUCKET: &str = "ilm7-cold-tier";
+const TIER_PREFIX: &str = "tiered";
+const SOURCE_BUCKET: &str = "ilm7-hot";
+const OBJECT_KEY: &str = "tier/report.bin";
+const CONTENT_TYPE: &str = "application/x-ilm7";
+const USER_META_KEY: &str = "ilm7-origin";
+const USER_META_VAL: &str = "hermetic-transition";
+
+/// 5 MiB — the S3 minimum size for a non-final multipart part; the object's only
+/// internal part boundary sits at this offset.
+const PART0_SIZE: usize = 5 * 1024 * 1024;
+/// 1 MiB tail so the completed object is genuinely multipart (two parts).
+const PART1_SIZE: usize = 1024 * 1024;
+const OBJECT_SIZE: usize = PART0_SIZE + PART1_SIZE;
+
+/// Deterministic, position-dependent payload: adjacent offsets differ, so a
+/// misaligned range read is caught.
+fn payload() -> Vec<u8> {
+    (0..OBJECT_SIZE).map(|i| (i % 251) as u8).collect()
+}
+
+/// Sign and send an admin request in-process (no `awscurl`).
+///
+/// Mirrors the shared admin-API e2e pattern: the SigV4 signature is computed
+/// over `UNSIGNED_PAYLOAD`, so the JSON body rides on the wire without being
+/// pre-hashed. Returns the response status and body text.
+async fn signed_admin_request(
+    base_url: &str,
+    method: Method,
+    path: &str,
+    body: Option<&str>,
+    access_key: &str,
+    secret_key: &str,
+) -> Result<(reqwest::StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
+    let url = format!("{base_url}{path}");
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+    let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
+
+    let request = http::Request::builder()
+        .method(method.clone())
+        .uri(uri)
+        .header(HOST, authority)
+        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
+
+    let client = local_http_client();
+    let mut request_builder = client.request(method, url.as_str());
+    for (name, value) in signed.headers() {
+        request_builder = request_builder.header(name, value);
+    }
+    if !body_bytes.is_empty() {
+        request_builder = request_builder.body(body_bytes);
+    }
+    let response = request_builder.send().await?;
+    let status = response.status();
+    let text = response.text().await?;
+    Ok((status, text))
+}
+
+/// Wire `hot` -> `cold` as a `TierType::RustFS` remote tier via `AddTier`.
+///
+/// No `force`, so the server runs the real in-use / connectivity probe against
+/// `cold` (which requires the tier bucket to already exist there).
+async fn add_rustfs_tier(hot: &RustFSTestEnvironment, cold: &RustFSTestEnvironment) -> TestResult {
+    let body = serde_json::json!({
+        "type": "rustfs",
+        "rustfs": {
+            "name": TIER_NAME,
+            "endpoint": cold.url.as_str(),
+            "accessKey": cold.access_key.as_str(),
+            "secretKey": cold.secret_key.as_str(),
+            "bucket": TIER_BUCKET,
+            "prefix": TIER_PREFIX,
+            "region": "us-east-1",
+            "storageClass": ""
+        }
+    })
+    .to_string();
+
+    let (status, resp) = signed_admin_request(
+        &hot.url,
+        Method::PUT,
+        "/rustfs/admin/v3/tier",
+        Some(&body),
+        &hot.access_key,
+        &hot.secret_key,
+    )
+    .await?;
+    if !status.is_success() {
+        return Err(format!("AddTier(RustFS) failed: status={status}, body={resp}").into());
+    }
+    Ok(())
+}
+
+/// A current-version `Transition Days=0` rule scoped to the object's prefix.
+fn transition_rule() -> Result<LifecycleRule, Box<dyn std::error::Error + Send + Sync>> {
+    Ok(LifecycleRule::builder()
+        .id("ilm7-transition")
+        .filter(LifecycleRuleFilter::builder().prefix("tier/").build())
+        .transitions(
+            Transition::builder()
+                .days(0)
+                .storage_class(TransitionStorageClass::from(TIER_NAME))
+                .build(),
+        )
+        .status(ExpirationStatus::Enabled)
+        .build()?)
+}
+
+/// Upload `data` as a two-part multipart object with a content-type and one
+/// user-metadata entry.
+async fn put_multipart_object(client: &Client, bucket: &str, key: &str, data: &[u8]) -> TestResult {
+    let create = client
+        .create_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .content_type(CONTENT_TYPE)
+        .metadata(USER_META_KEY, USER_META_VAL)
+        .send()
+        .await?;
+    let upload_id = create
+        .upload_id()
+        .ok_or("CreateMultipartUpload returned no upload id")?
+        .to_string();
+
+    let mut completed = Vec::new();
+    for (idx, chunk) in [&data[..PART0_SIZE], &data[PART0_SIZE..]].into_iter().enumerate() {
+        let part_number = (idx + 1) as i32;
+        let uploaded = client
+            .upload_part()
+            .bucket(bucket)
+            .key(key)
+            .upload_id(&upload_id)
+            .part_number(part_number)
+            .body(ByteStream::from(chunk.to_vec()))
+            .send()
+            .await?;
+        completed.push(
+            CompletedPart::builder()
+                .part_number(part_number)
+                .e_tag(uploaded.e_tag().unwrap_or_default())
+                .build(),
+        );
+    }
+
+    client
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed)).build())
+        .send()
+        .await?;
+    Ok(())
+}
+
+/// Number of objects currently stored in the cold-tier bucket.
+async fn cold_tier_object_count(cold_client: &Client) -> Result<usize, Box<dyn std::error::Error + Send + Sync>> {
+    let resp = cold_client.list_objects_v2().bucket(TIER_BUCKET).send().await?;
+    Ok(resp.contents().len())
+}
+
+/// Poll `HEAD` until the object's storage class is the tier name (transition
+/// complete), or fail after `deadline`.
+async fn wait_for_transition(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
+    let start = Instant::now();
+    loop {
+        let head = client.head_object().bucket(bucket).key(key).send().await?;
+        if head.storage_class().map(|sc| sc.as_str()) == Some(TIER_NAME) {
+            return Ok(());
+        }
+        if start.elapsed() >= deadline {
+            return Err(format!(
+                "object {bucket}/{key} was not transitioned to {TIER_NAME} within {}s (storage_class={:?})",
+                deadline.as_secs(),
+                head.storage_class()
+            )
+            .into());
+        }
+        tokio::time::sleep(StdDuration::from_millis(500)).await;
+    }
+}
+
+/// Poll until the cold-tier bucket is empty (remote free-version cleanup done),
+/// or fail after `deadline`.
+async fn wait_for_cold_tier_empty(cold_client: &Client, deadline: StdDuration) -> TestResult {
+    let start = Instant::now();
+    loop {
+        let count = cold_tier_object_count(cold_client).await?;
+        if count == 0 {
+            return Ok(());
+        }
+        if start.elapsed() >= deadline {
+            return Err(format!(
+                "cold-tier bucket still holds {count} object(s) {}s after DeleteObject; \
+                 free-version remote cleanup did not converge",
+                deadline.as_secs()
+            )
+            .into());
+        }
+        tokio::time::sleep(StdDuration::from_millis(500)).await;
+    }
+}
+
+/// GET `bytes=start-end` (inclusive) and assert it equals `data[start..=end]`.
+async fn assert_range(client: &Client, start: usize, end: usize, data: &[u8]) -> TestResult {
+    let range = format!("bytes={start}-{end}");
+    let resp = client
+        .get_object()
+        .bucket(SOURCE_BUCKET)
+        .key(OBJECT_KEY)
+        .range(&range)
+        .send()
+        .await?;
+    let got = resp.body.collect().await?.into_bytes();
+    let expected = &data[start..=end];
+    assert_eq!(got.len(), expected.len(), "range {range}: length mismatch");
+    assert_eq!(got.as_ref(), expected, "range {range}: bytes mismatch reading from the tier");
+    Ok(())
+}
+
+/// Full ilm-7 hermetic transition main path across two embedded RustFS servers.
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn test_hermetic_transition_main_path() -> TestResult {
+    // Cold-tier server (independent credentials). It is a passive tier target,
+    // so it needs no lifecycle/scanner configuration.
+    let mut cold = RustFSTestEnvironment::new().await?;
+    cold.access_key = "coldtieradmin".to_string();
+    cold.secret_key = "coldtiersecret".to_string();
+    cold.start_rustfs_server_without_cleanup(vec![]).await?;
+    let cold_client = cold.create_s3_client();
+    cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
+
+    // Hot/source server. A 1s scanner cycle is a backstop; transition is
+    // primarily driven immediately by the multipart completion path.
+    let mut hot = RustFSTestEnvironment::new().await?;
+    hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1")])
+        .await?;
+    let hot_client = hot.create_s3_client();
+
+    // Wire the RustFS remote tier (real connectivity probe, no force).
+    add_rustfs_tier(&hot, &cold).await?;
+
+    // Source bucket + Days=0 transition rule installed BEFORE the write so the
+    // completion path enqueues the transition immediately.
+    hot_client.create_bucket().bucket(SOURCE_BUCKET).send().await?;
+    let lifecycle = BucketLifecycleConfiguration::builder().rules(transition_rule()?).build()?;
+    hot_client
+        .put_bucket_lifecycle_configuration()
+        .bucket(SOURCE_BUCKET)
+        .lifecycle_configuration(lifecycle)
+        .send()
+        .await?;
+
+    let data = payload();
+    put_multipart_object(&hot_client, SOURCE_BUCKET, OBJECT_KEY, &data).await?;
+
+    // 1) Transition completes: HEAD reports the tier name and no restore state.
+    wait_for_transition(&hot_client, SOURCE_BUCKET, OBJECT_KEY, StdDuration::from_secs(90)).await?;
+    let head = hot_client.head_object().bucket(SOURCE_BUCKET).key(OBJECT_KEY).send().await?;
+    assert_eq!(
+        head.storage_class().map(|sc| sc.as_str()),
+        Some(TIER_NAME),
+        "transitioned object must report the tier name as its storage class"
+    );
+    assert!(
+        head.restore().is_none(),
+        "a freshly transitioned object must not advertise x-amz-restore, got {:?}",
+        head.restore()
+    );
+
+    // 2) The remote object now lives in the cold-tier bucket.
+    assert!(
+        cold_tier_object_count(&cold_client).await? >= 1,
+        "cold-tier bucket must hold the transitioned object"
+    );
+
+    // 3) GET streams identical bytes back through the warm backend; content-type
+    //    and user metadata survive the transition round trip (rustfs#2246).
+    let get = hot_client.get_object().bucket(SOURCE_BUCKET).key(OBJECT_KEY).send().await?;
+    assert_eq!(get.content_type(), Some(CONTENT_TYPE), "content-type must survive transition");
+    assert_eq!(
+        get.metadata().and_then(|m| m.get(USER_META_KEY)).map(String::as_str),
+        Some(USER_META_VAL),
+        "user metadata must survive transition"
+    );
+    let body = get.body.collect().await?.into_bytes();
+    assert_eq!(body.len(), data.len(), "full transitioned GET length mismatch");
+    assert_eq!(body.as_ref(), data.as_slice(), "full transitioned GET must be byte-identical");
+
+    // 4) Range GET within a single part and across the part boundary
+    //    (backlog#807): both must read the correct bytes from the tier.
+    assert_range(&hot_client, 1000, 1099, &data).await?;
+    assert_range(&hot_client, PART0_SIZE - 5, PART0_SIZE + 4, &data).await?;
+
+    // 5) DeleteObject drives free-version cleanup. The local object is gone
+    //    immediately; the remote copy is removed asynchronously.
+    hot_client
+        .delete_object()
+        .bucket(SOURCE_BUCKET)
+        .key(OBJECT_KEY)
+        .send()
+        .await?;
+
+    let get_after = hot_client.get_object().bucket(SOURCE_BUCKET).key(OBJECT_KEY).send().await;
+    let err = get_after.expect_err("hot object must be gone immediately after delete");
+    assert_eq!(
+        err.as_service_error().and_then(|e| e.code()),
+        Some("NoSuchKey"),
+        "hot GET after delete must be NoSuchKey, got {err:?}"
+    );
+
+    wait_for_cold_tier_empty(&cold_client, StdDuration::from_secs(90)).await?;
+
+    Ok(())
+}
@@ -0,0 +1,306 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Live-listener TLS certificate hot-reload e2e (backlog#1154 peri-5).
+//!
+//! `crates/tls-runtime` had ~20 unit tests but no test ever rotated a
+//! certificate under a listening HTTPS server, so the operational promise
+//! "swap certificates without a restart" (RUSTFS_TLS_RELOAD_ENABLE) had no
+//! automated proof. This suite pins, against a real binary over real TLS
+//! handshakes:
+//!
+//!   * the server starts with certificate A and serves its fingerprint;
+//!   * after the on-disk material is replaced with certificate B, new
+//!     connections receive B within the reload interval — no restart;
+//!   * a connection established under A keeps working across the swap
+//!     (existing sessions are not torn down);
+//!   * replacing the material with garbage does not take down the listener or
+//!     the previously loaded certificate (fail-safe, no fail-open) and leaves
+//!     an assertable `tls_reload_failed` log event.
+
+use crate::common::{RustFSTestEnvironment, init_logging};
+use rcgen::generate_simple_self_signed;
+use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
+use rustls::pki_types::{CertificateDer, ServerName, UnixTime};
+use rustls::{ClientConfig, ClientConnection, DigitallySignedStruct, Error as RustlsError, SignatureScheme, StreamOwned};
+use serial_test::serial;
+use sha2::{Digest, Sha256};
+use std::error::Error;
+use std::io::{Read, Write};
+use std::net::TcpStream;
+use std::sync::Arc;
+use tokio::time::{Duration, sleep};
+
+type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
+type BoxError = Box<dyn Error + Send + Sync>;
+
+const CERT_FILE: &str = "rustfs_cert.pem";
+const KEY_FILE: &str = "rustfs_key.pem";
+/// Minimum value the server accepts for RUSTFS_TLS_RELOAD_INTERVAL (seconds).
+const RELOAD_INTERVAL_SECS: u64 = 5;
+
+#[derive(Debug)]
+struct AcceptAnyServerCertVerifier;
+
+impl ServerCertVerifier for AcceptAnyServerCertVerifier {
+    fn verify_server_cert(
+        &self,
+        _end_entity: &CertificateDer<'_>,
+        _intermediates: &[CertificateDer<'_>],
+        _server_name: &ServerName<'_>,
+        _ocsp_response: &[u8],
+        _now: UnixTime,
+    ) -> Result<ServerCertVerified, RustlsError> {
+        Ok(ServerCertVerified::assertion())
+    }
+
+    fn verify_tls12_signature(
+        &self,
+        _message: &[u8],
+        _cert: &CertificateDer<'_>,
+        _dss: &DigitallySignedStruct,
+    ) -> Result<HandshakeSignatureValid, RustlsError> {
+        Ok(HandshakeSignatureValid::assertion())
+    }
+
+    fn verify_tls13_signature(
+        &self,
+        _message: &[u8],
+        _cert: &CertificateDer<'_>,
+        _dss: &DigitallySignedStruct,
+    ) -> Result<HandshakeSignatureValid, RustlsError> {
+        Ok(HandshakeSignatureValid::assertion())
+    }
+
+    fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
+        rustls::crypto::aws_lc_rs::default_provider()
+            .signature_verification_algorithms
+            .supported_schemes()
+    }
+}
+
+fn tls_client_config() -> Arc<ClientConfig> {
+    Arc::new(
+        ClientConfig::builder()
+            .dangerous()
+            .with_custom_certificate_verifier(Arc::new(AcceptAnyServerCertVerifier))
+            .with_no_client_auth(),
+    )
+}
+
+/// A live TLS session over a blocking TCP stream, kept open across reloads.
+struct TlsSession {
+    stream: StreamOwned<ClientConnection, TcpStream>,
+    fingerprint: String,
+    host: String,
+}
+
+/// Completes a TLS handshake with `addr` and returns the session together with
+/// the SHA-256 fingerprint of the served leaf certificate.
+fn tls_connect(addr: &str) -> Result<TlsSession, BoxError> {
+    let host = addr.split(':').next().unwrap_or("127.0.0.1").to_string();
+    let server_name = ServerName::try_from(host.clone())?;
+    let mut conn = ClientConnection::new(tls_client_config(), server_name)?;
+    let mut tcp = TcpStream::connect(addr)?;
+    tcp.set_read_timeout(Some(std::time::Duration::from_secs(10)))?;
+    tcp.set_write_timeout(Some(std::time::Duration::from_secs(10)))?;
+
+    // Drive the handshake to completion so peer_certificates() is populated.
+    while conn.is_handshaking() {
+        conn.complete_io(&mut tcp)?;
+    }
+
+    let leaf = conn
+        .peer_certificates()
+        .and_then(|certs| certs.first())
+        .ok_or("server presented no certificate")?;
+    let fingerprint = Sha256::digest(leaf.as_ref())
+        .iter()
+        .map(|b| format!("{b:02x}"))
+        .collect::<String>();
+
+    Ok(TlsSession {
+        stream: StreamOwned::new(conn, tcp),
+        fingerprint,
+        host,
+    })
+}
+
+/// Sends a keep-alive HTTP request on the session and reads the response head,
+/// proving the TLS session is still fully functional.
+fn http_roundtrip(session: &mut TlsSession) -> Result<String, BoxError> {
+    let request = format!("GET / HTTP/1.1\r\nHost: {}\r\nConnection: keep-alive\r\n\r\n", session.host);
+    session.stream.write_all(request.as_bytes())?;
+    session.stream.flush()?;
+
+    // Read up to the end of the headers plus a body chunk; anonymous ListBuckets
+    // answers a small error payload, which is fine — we only need a valid
+    // HTTP status line over the existing TLS session.
+    let mut buf = vec![0_u8; 8192];
+    let read = session.stream.read(&mut buf)?;
+    if read == 0 {
+        return Err("connection closed by server".into());
+    }
+    let head = String::from_utf8_lossy(&buf[..read]).to_string();
+    if !head.starts_with("HTTP/1.1") {
+        return Err(format!("unexpected response head: {head}").into());
+    }
+    Ok(head)
+}
+
+async fn write_cert_pair(tls_dir: &std::path::Path, sans: Vec<String>) -> Result<(), BoxError> {
+    let cert = generate_simple_self_signed(sans)?;
+    tokio::fs::write(tls_dir.join(CERT_FILE), cert.cert.pem()).await?;
+    tokio::fs::write(tls_dir.join(KEY_FILE), cert.signing_key.serialize_pem()).await?;
+    Ok(())
+}
+
+/// Polls new TLS connections until the served fingerprint changes away from
+/// `old_fingerprint`, returning the new one.
+async fn wait_for_new_fingerprint(addr: &str, old_fingerprint: &str, within: Duration) -> Result<String, BoxError> {
+    let deadline = tokio::time::Instant::now() + within;
+    loop {
+        let addr_owned = addr.to_string();
+        let observed = tokio::task::spawn_blocking(move || tls_connect(&addr_owned).map(|s| s.fingerprint)).await?;
+        if let Ok(fp) = observed
+            && fp != old_fingerprint
+        {
+            return Ok(fp);
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!("served certificate never rotated away from {old_fingerprint} within {within:?}").into());
+        }
+        sleep(Duration::from_secs(1)).await;
+    }
+}
+
+/// Starts the rustfs binary with HTTPS + hot reload enabled, capturing its
+/// stdout/stderr to `log_path`. The harness's own start path is unusable here:
+/// its readiness probe drives the AWS SDK over the environment URL, and the SDK
+/// rejects the self-signed test certificate.
+async fn start_https_server_with_reload(
+    env: &mut RustFSTestEnvironment,
+    tls_dir: &std::path::Path,
+    log_path: &str,
+) -> TestResult {
+    let log_file = std::fs::OpenOptions::new().create(true).append(true).open(log_path)?;
+    let log_file_err = log_file.try_clone()?;
+    let process = std::process::Command::new(crate::common::rustfs_binary_path())
+        .env("RUST_LOG", "rustfs=info")
+        .env("RUSTFS_CONSOLE_ENABLE", "false")
+        .env("RUSTFS_TLS_PATH", tls_dir)
+        .env("RUSTFS_TLS_RELOAD_ENABLE", "true")
+        .env("RUSTFS_TLS_RELOAD_INTERVAL", RELOAD_INTERVAL_SECS.to_string())
+        .stdout(std::process::Stdio::from(log_file))
+        .stderr(std::process::Stdio::from(log_file_err))
+        .args([
+            "--address",
+            &env.address,
+            "--access-key",
+            &env.access_key,
+            "--secret-key",
+            &env.secret_key,
+            &env.temp_dir,
+        ])
+        .spawn()?;
+    env.process = Some(process);
+
+    // Readiness = a TLS handshake completes on the listener.
+    let addr = env.address.clone();
+    for attempt in 0..60 {
+        let addr_clone = addr.clone();
+        if tokio::task::spawn_blocking(move || tls_connect(&addr_clone)).await?.is_ok() {
+            return Ok(());
+        }
+        if attempt == 59 {
+            return Err("HTTPS server never completed a TLS handshake within 60s".into());
+        }
+        sleep(Duration::from_secs(1)).await;
+    }
+    Ok(())
+}
+
+/// Runs `http_roundtrip` on a session inside the blocking pool, handing the
+/// session back for later reuse (the point: the same TLS session survives).
+async fn roundtrip_and_return(mut session: TlsSession) -> Result<TlsSession, BoxError> {
+    tokio::task::spawn_blocking(move || {
+        http_roundtrip(&mut session)?;
+        Ok::<_, BoxError>(session)
+    })
+    .await?
+}
+
+#[tokio::test]
+#[serial]
+async fn test_tls_certificate_hot_reload_live_listener() -> TestResult {
+    init_logging();
+    // Install the process-wide rustls crypto provider (idempotent).
+    let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    let tls_dir = std::path::PathBuf::from(format!("{}/tls", env.temp_dir));
+    tokio::fs::create_dir_all(&tls_dir).await?;
+    write_cert_pair(&tls_dir, vec!["localhost".into(), "127.0.0.1".into()]).await?;
+
+    let log_path = format!("{}/server.log", env.temp_dir);
+    start_https_server_with_reload(&mut env, &tls_dir, &log_path).await?;
+
+    let addr = env.address.clone();
+
+    // --- certificate A is served, and the session stays usable ---------------
+    let session_a = {
+        let addr = addr.clone();
+        tokio::task::spawn_blocking(move || tls_connect(&addr)).await??
+    };
+    let fingerprint_a = session_a.fingerprint.clone();
+    let session_a = roundtrip_and_return(session_a).await?;
+
+    // --- swap to certificate B: new connections pick it up without restart ---
+    write_cert_pair(&tls_dir, vec!["localhost".into(), "127.0.0.1".into()]).await?;
+    let reload_budget = Duration::from_secs(RELOAD_INTERVAL_SECS * 4 + 10);
+    let fingerprint_b = wait_for_new_fingerprint(&addr, &fingerprint_a, reload_budget).await?;
+    assert_ne!(fingerprint_a, fingerprint_b);
+
+    // The connection opened under certificate A must survive the rotation.
+    let _session_a = roundtrip_and_return(session_a).await?;
+
+    // --- garbage material: fail-safe, keep serving B, log the failure --------
+    tokio::fs::write(tls_dir.join(CERT_FILE), b"not a certificate").await?;
+    tokio::fs::write(tls_dir.join(KEY_FILE), b"not a key").await?;
+
+    // Give the reload loop at least two ticks to observe the bad material.
+    sleep(Duration::from_secs(RELOAD_INTERVAL_SECS * 2 + 2)).await;
+
+    let after_bad = {
+        let addr = addr.clone();
+        tokio::task::spawn_blocking(move || tls_connect(&addr)).await??
+    };
+    assert_eq!(
+        after_bad.fingerprint, fingerprint_b,
+        "bad on-disk material must not change or drop the served certificate"
+    );
+
+    let log = tokio::fs::read_to_string(&log_path).await.unwrap_or_default();
+    assert!(
+        log.contains("tls_reload_failed") || log.contains("TLS reload failed"),
+        "a failed reload must leave an assertable log event; captured log did not contain one"
+    );
+
+    // The process must still be alive and serving.
+    let final_session = tokio::task::spawn_blocking(move || tls_connect(&addr)).await??;
+    let _ = roundtrip_and_return(final_session).await?;
+
+    env.stop_server();
+    Ok(())
+}
@@ -49,7 +49,7 @@ rustfs-signer.workspace = true
 rustfs-storage-api.workspace = true
 rustfs-tls-runtime.workspace = true
 rustfs-checksums.workspace = true
-rustfs-config = { workspace = true, features = ["constants", "notify", "audit", "server-config-model"] }
+rustfs-config = { workspace = true, features = ["notify", "audit", "server-config-model"] }
 rustfs-concurrency.workspace = true
 rustfs-credentials = { workspace = true }
 rustfs-common.workspace = true
@@ -62,9 +62,9 @@ rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
 rustfs-object-capacity.workspace = true
 async-trait.workspace = true
-bytes.workspace = true
+bytes = { workspace = true, features = ["serde"] }
 byteorder = { workspace = true }
-chrono.workspace = true
+chrono = { workspace = true, features = ["serde"] }
 glob = { workspace = true }
 thiserror.workspace = true
 flatbuffers.workspace = true
@@ -72,22 +72,22 @@ futures.workspace = true
 futures-util.workspace = true
 tracing.workspace = true
 tracing-opentelemetry.workspace = true
-serde.workspace = true
-time.workspace = true
+serde = { workspace = true, features = ["derive"] }
+time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
 bytesize.workspace = true
-serde_json.workspace = true
+serde_json = { workspace = true, features = ["raw_value"] }
 quick-xml = { workspace = true, features = ["serialize", "async-tokio"] }
-s3s.workspace = true
+s3s = { workspace = true, features = ["minio"] }
 http.workspace = true
 opentelemetry.workspace = true
 http-body = { workspace = true }
 http-body-util.workspace = true
 url.workspace = true
-uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
-reed-solomon-erasure = { workspace = true }
+uuid = { workspace = true, features = ["v4", "fast-rng", "serde", "macro-diagnostics"] }
+reed-solomon-erasure = { workspace = true, features = ["simd-accel"] }
 reed-solomon-simd = { workspace = true }
 lazy_static.workspace = true
-moka = { workspace = true }
+moka = { workspace = true, features = ["future"] }
 rustfs-lock.workspace = true
 rustfs-io-metrics.workspace = true
 regex = { workspace = true }
@@ -101,38 +101,38 @@ sha1 = { workspace = true }
 sha2 = { workspace = true }
 hex-simd = { workspace = true }
 tempfile.workspace = true
-hyper.workspace = true
-hyper-util.workspace = true
-hyper-rustls.workspace = true
-rustls.workspace = true
+hyper = { workspace = true, features = ["http2", "http1", "server"] }
+hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
+hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
+rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
 rustls-pki-types.workspace = true
-tokio = { workspace = true, features = ["io-util", "sync", "signal"] }
-tonic.workspace = true
+tokio = { workspace = true, features = ["io-util", "sync", "signal", "fs", "rt-multi-thread"] }
+tonic = { workspace = true, features = ["gzip", "deflate"] }
 xxhash-rust = { workspace = true, features = ["xxh64", "xxh3"] }
-tower.workspace = true
+tower = { workspace = true, features = ["timeout"] }
 async-channel.workspace = true
 enumset = { workspace = true }
 num_cpus = { workspace = true }
-rand.workspace = true
+rand = { workspace = true, features = ["serde"] }
 pin-project-lite.workspace = true
 md-5.workspace = true
 memmap2 = { workspace = true }
 libc.workspace = true
 # "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate
 # (backlog#1178); "fs" comes from the workspace default.
-rustix = { workspace = true, features = ["process"] }
+rustix = { workspace = true, features = ["process", "fs"] }
 rustfs-madmin.workspace = true
-reqwest = { workspace = true }
-aes-gcm.workspace = true
+reqwest = { workspace = true, default-features = false, features = ["rustls", "http2", "system-proxy"] }
+aes-gcm = { workspace = true, features = ["rand_core"] }
 chacha20poly1305.workspace = true
-aws-sdk-s3 = { workspace = true }
+aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 urlencoding = { workspace = true }
-smallvec = { workspace = true }
-shadow-rs.workspace = true
+smallvec = { workspace = true, features = ["serde"] }
+shadow-rs = { workspace = true, default-features = false }
 async-recursion.workspace = true
 aws-credential-types = { workspace = true }
 aws-smithy-types = { workspace = true }
-aws-smithy-runtime-api = { workspace = true }
+aws-smithy-runtime-api = { workspace = true, features = ["http-1x"] }
 parking_lot = { workspace = true }
 base64-simd.workspace = true
 serde_urlencoded.workspace = true
@@ -141,7 +141,7 @@ google-cloud-auth = { workspace = true }
 aws-config = { workspace = true }
 faster-hex = { workspace = true }
 ratelimit = { workspace = true }
-aws-smithy-http-client.workspace = true
+aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }

 # Observability and Metrics
 metrics = { workspace = true }
@@ -153,19 +153,19 @@ metrics = { workspace = true }
 rustfs-uring = "0.2.1"

 [dev-dependencies]
-tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util"] }
+tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }
 criterion = { workspace = true, features = ["html_reports"] }
 temp-env = { workspace = true, features = ["async_closure"] }
-tracing-subscriber = { workspace = true, features = ["json"] }
+tracing-subscriber = { workspace = true, features = ["json", "env-filter", "time"] }
 serial_test = { workspace = true }
-opentelemetry_sdk = { workspace = true }
+opentelemetry_sdk = { workspace = true, features = ["rt-tokio"] }
 proptest = "1"
 rcgen.workspace = true
-insta = { workspace = true }
+insta = { workspace = true, features = ["yaml", "json"] }
 rustfs-crypto = { workspace = true }

 [build-dependencies]
-shadow-rs = { workspace = true, features = ["build", "metadata"] }
+shadow-rs = { workspace = true, default-features = false, features = ["build", "metadata"] }

 [[bench]]
 name = "erasure_benchmark"
@@ -134,7 +134,7 @@ pub mod bucket {
    }

    pub mod quota {
-        pub use crate::bucket::quota::{BucketQuota, QuotaError, QuotaOperation};
+        pub use crate::bucket::quota::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};

        pub mod checker {
            pub use crate::bucket::quota::checker::QuotaChecker;
@@ -1023,20 +1023,49 @@ fn build_insecure_aws_s3_http_client() -> SharedHttpClient {
    http_client_fn(move |_settings, _components| connector.clone())
 }

-fn build_aws_s3_http_client_from_target_ca_pem(ca_cert_pem: &str) -> Result<SharedHttpClient, BucketTargetError> {
-    let certs = rustls_pki_types::CertificateDer::pem_slice_iter(ca_cert_pem.as_bytes())
+fn validate_ca_pem_bundle(ca_cert_pem: &[u8]) -> Result<(), String> {
+    let certs = rustls_pki_types::CertificateDer::pem_slice_iter(ca_cert_pem)
        .collect::<Result<Vec<_>, _>>()
-        .map_err(|err| BucketTargetError::Io(std::io::Error::other(format!("invalid target CA PEM: {err}"))))?;
+        .map_err(|err| format!("invalid PEM encoding: {err}"))?;

    if certs.is_empty() {
-        return Err(BucketTargetError::Io(std::io::Error::other(
-            "invalid target CA PEM: no certificates found",
-        )));
+        return Err("no certificates found".to_string());
    }

-    let mut trust_store = smithy_tls::TrustStore::empty();
-    trust_store.add_pem_certificate(ca_cert_pem.as_bytes());
+    // Smithy's rustls adapter defers parsing custom certificates and assumes
+    // they are valid when the HTTPS connector is built. Validate every DER
+    // certificate first so malformed configuration is reported rather than
+    // reaching an `expect` in the dependency.
+    let mut validation_store = rustls::RootCertStore::empty();
+    for cert in certs {
+        validation_store
+            .add(cert)
+            .map_err(|err| format!("invalid X.509 certificate: {err}"))?;
+    }

+    Ok(())
+}
+
+fn validate_target_ca_pem(ca_cert_pem: &str) -> Result<(), BucketTargetError> {
+    validate_ca_pem_bundle(ca_cert_pem.as_bytes())
+        .map_err(|err| BucketTargetError::Io(std::io::Error::other(format!("invalid target CA PEM: {err}"))))
+}
+
+fn compose_replication_trust_store(certificate_bundles: impl IntoIterator<Item = Vec<u8>>) -> (smithy_tls::TrustStore, usize) {
+    // `TrustStore::default()` keeps the platform-native roots enabled. Target
+    // and RUSTFS_TLS_PATH certificates extend that baseline instead of
+    // replacing it with a target-specific trust island.
+    let mut trust_store = smithy_tls::TrustStore::default();
+    let mut custom_bundle_count = 0;
+    for pem in certificate_bundles {
+        trust_store.add_pem_certificate(pem);
+        custom_bundle_count += 1;
+    }
+
+    (trust_store, custom_bundle_count)
+}
+
+fn build_aws_s3_http_client_with_trust_store(trust_store: smithy_tls::TrustStore) -> Result<SharedHttpClient, BucketTargetError> {
    let tls_context = smithy_tls::TlsContext::builder()
        .with_trust_store(trust_store)
        .build()
@@ -1048,6 +1077,60 @@ fn build_aws_s3_http_client_from_target_ca_pem(ca_cert_pem: &str) -> Result<Shar
        .build_https())
 }

+async fn load_tls_path_ca_bundles(tls_dir: &Path, trust_leaf_cert_as_ca: bool) -> Vec<Vec<u8>> {
+    let mut certificate_bundles = Vec::new();
+
+    let ca_path = tls_dir.join(RUSTFS_CA_CERT);
+    match tokio::fs::read(&ca_path).await {
+        Ok(pem) => match validate_ca_pem_bundle(&pem) {
+            Ok(()) => certificate_bundles.push(pem),
+            Err(err) => warn!("ignoring invalid custom CA bundle {:?} for replication client: {}", ca_path, err),
+        },
+        Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
+        Err(e) => warn!("failed to read custom CA bundle {:?} for replication client: {}", ca_path, e),
+    }
+
+    if trust_leaf_cert_as_ca {
+        let leaf_cert_path = tls_dir.join(RUSTFS_TLS_CERT);
+        match tokio::fs::read(&leaf_cert_path).await {
+            Ok(pem) => match validate_ca_pem_bundle(&pem) {
+                Ok(()) => certificate_bundles.push(pem),
+                Err(err) => warn!(
+                    "ignoring invalid leaf certificate {:?} for replication client trust store: {}",
+                    leaf_cert_path, err
+                ),
+            },
+            Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
+            Err(e) => warn!("failed to read leaf cert {:?} for replication client trust store: {}", leaf_cert_path, e),
+        }
+    }
+
+    certificate_bundles
+}
+
+async fn load_configured_tls_ca_bundles() -> Vec<Vec<u8>> {
+    let tls_path = rustfs_utils::get_env_str(rustfs_config::ENV_RUSTFS_TLS_PATH, rustfs_config::DEFAULT_RUSTFS_TLS_PATH);
+    if tls_path.is_empty() {
+        return Vec::new();
+    }
+
+    load_tls_path_ca_bundles(
+        Path::new(&tls_path),
+        rustfs_utils::get_env_bool(ENV_TRUST_LEAF_CERT_AS_CA, DEFAULT_TRUST_LEAF_CERT_AS_CA),
+    )
+    .await
+}
+
+async fn build_aws_s3_http_client_from_target_ca_pem(ca_cert_pem: &str) -> Result<SharedHttpClient, BucketTargetError> {
+    validate_target_ca_pem(ca_cert_pem)?;
+
+    let mut certificate_bundles = load_configured_tls_ca_bundles().await;
+    certificate_bundles.push(ca_cert_pem.as_bytes().to_vec());
+    let (trust_store, _) = compose_replication_trust_store(certificate_bundles);
+
+    build_aws_s3_http_client_with_trust_store(trust_store)
+}
+
 async fn build_aws_s3_http_client_for_target(target: &BucketTarget) -> Result<Option<SharedHttpClient>, BucketTargetError> {
    if !target.secure {
        return Ok(None);
@@ -1058,62 +1141,28 @@ async fn build_aws_s3_http_client_for_target(target: &BucketTarget) -> Result<Op
    }

    if has_custom_ca_pem(target) {
-        return build_aws_s3_http_client_from_target_ca_pem(&target.ca_cert_pem).map(Some);
+        return build_aws_s3_http_client_from_target_ca_pem(&target.ca_cert_pem)
+            .await
+            .map(Some);
    }

    Ok(build_aws_s3_http_client_from_tls_path().await)
 }

 async fn build_aws_s3_http_client_from_tls_path() -> Option<aws_sdk_s3::config::SharedHttpClient> {
-    let tls_path = rustfs_utils::get_env_str(rustfs_config::ENV_RUSTFS_TLS_PATH, rustfs_config::DEFAULT_RUSTFS_TLS_PATH);
-    if tls_path.is_empty() {
+    let certificate_bundles = load_configured_tls_ca_bundles().await;
+    if certificate_bundles.is_empty() {
        return None;
    }

-    let tls_dir = Path::new(&tls_path);
-    let mut trust_store = smithy_tls::TrustStore::empty();
-    let mut has_custom_certs = false;
-
-    let ca_path = tls_dir.join(RUSTFS_CA_CERT);
-    match tokio::fs::read(&ca_path).await {
-        Ok(pem) => {
-            trust_store.add_pem_certificate(pem);
-            has_custom_certs = true;
-        }
-        Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
-        Err(e) => warn!("failed to read custom CA bundle {:?} for replication client: {}", ca_path, e),
-    }
-
-    if rustfs_utils::get_env_bool(ENV_TRUST_LEAF_CERT_AS_CA, DEFAULT_TRUST_LEAF_CERT_AS_CA) {
-        let leaf_cert_path = tls_dir.join(RUSTFS_TLS_CERT);
-        match tokio::fs::read(&leaf_cert_path).await {
-            Ok(pem) => {
-                trust_store.add_pem_certificate(pem);
-                has_custom_certs = true;
-            }
-            Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
-            Err(e) => warn!("failed to read leaf cert {:?} for replication client trust store: {}", leaf_cert_path, e),
-        }
-    }
-
-    if !has_custom_certs {
-        return None;
-    }
-
-    let tls_context = match smithy_tls::TlsContext::builder().with_trust_store(trust_store).build() {
-        Ok(ctx) => ctx,
+    let (trust_store, _) = compose_replication_trust_store(certificate_bundles);
+    match build_aws_s3_http_client_with_trust_store(trust_store) {
+        Ok(client) => Some(client),
        Err(e) => {
            warn!("failed to build AWS SDK TLS context for replication client: {}", e);
-            return None;
+            None
        }
-    };
-
-    Some(
-        SmithyHttpClientBuilder::new()
-            .tls_provider(smithy_tls::Provider::rustls(smithy_tls::rustls_provider::CryptoMode::AwsLc))
-            .tls_context(tls_context)
-            .build_https(),
-    )
+    }
 }

 fn should_force_path_style(target: &BucketTarget) -> bool {
@@ -1920,6 +1969,63 @@ mod tests {
    use super::*;
    use rcgen::generate_simple_self_signed;

+    fn spawn_single_request_https_server(cert: &rcgen::CertifiedKey<rcgen::KeyPair>) -> (u16, std::thread::JoinHandle<()>) {
+        use std::io::{Read, Write};
+
+        ensure_rustls_crypto_provider();
+        let listener = std::net::TcpListener::bind(("127.0.0.1", 0)).expect("test TLS listener should bind");
+        let port = listener
+            .local_addr()
+            .expect("test TLS listener should have an address")
+            .port();
+        let server_config = rustls::ServerConfig::builder()
+            .with_no_client_auth()
+            .with_single_cert(
+                vec![cert.cert.der().clone()],
+                rustls_pki_types::PrivateKeyDer::try_from(cert.signing_key.serialize_der())
+                    .expect("test TLS private key should convert"),
+            )
+            .expect("test TLS server config should build");
+
+        let handle = std::thread::spawn(move || {
+            let (stream, _) = listener.accept().expect("test TLS client should connect");
+            stream
+                .set_read_timeout(Some(Duration::from_secs(10)))
+                .expect("test TLS read timeout should configure");
+            stream
+                .set_write_timeout(Some(Duration::from_secs(10)))
+                .expect("test TLS write timeout should configure");
+            let connection = rustls::ServerConnection::new(Arc::new(server_config)).expect("test TLS connection should build");
+            let mut stream = rustls::StreamOwned::new(connection, stream);
+            let mut request = [0_u8; 8192];
+            let _ = stream.read(&mut request).expect("test TLS request should be readable");
+            stream
+                .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
+                .expect("test TLS response should be written");
+            stream.flush().expect("test TLS response should flush");
+        });
+
+        (port, handle)
+    }
+
+    fn s3_client_with_http_client(port: u16, http_client: SharedHttpClient) -> S3Client {
+        let credentials = SdkCredentials::builder()
+            .access_key_id("test-access")
+            .secret_access_key("test-secret")
+            .provider_name("bucket_target_tls_test")
+            .build();
+        let config = S3Config::builder()
+            .endpoint_url(format!("https://localhost:{port}"))
+            .credentials_provider(SharedCredentialsProvider::new(credentials))
+            .region(SdkRegion::new("us-east-1"))
+            .force_path_style(true)
+            .behavior_version(aws_sdk_s3::config::BehaviorVersion::latest())
+            .http_client(http_client)
+            .build();
+
+        S3Client::from_conf(config)
+    }
+
    #[test]
    fn replication_target_versioning_enabled_requires_enabled_status() {
        let enabled = BucketVersioningStatus::Enabled;
@@ -2243,6 +2349,78 @@ mod tests {
        assert_eq!(client.endpoint, "https://192.168.1.10:9000");
    }

+    #[tokio::test]
+    async fn replication_trust_store_composes_system_global_and_target_roots_for_real_tls() {
+        let tls_dir = tempfile::tempdir().expect("temporary TLS directory should be created");
+        let global_ca =
+            generate_simple_self_signed(vec!["localhost".to_string()]).expect("global CA certificate should generate");
+        let target_ca =
+            generate_simple_self_signed(vec!["localhost".to_string()]).expect("target CA certificate should generate");
+
+        tokio::fs::write(tls_dir.path().join(RUSTFS_CA_CERT), global_ca.cert.pem())
+            .await
+            .expect("global CA bundle should be written");
+
+        let mut certificate_bundles = load_tls_path_ca_bundles(tls_dir.path(), false).await;
+        certificate_bundles.push(target_ca.cert.pem().into_bytes());
+        let (trust_store, _) = compose_replication_trust_store(certificate_bundles);
+        assert!(
+            format!("{trust_store:?}").contains("enable_native_roots: true"),
+            "per-target trust must retain the SDK's platform-native roots"
+        );
+        let http_client = build_aws_s3_http_client_with_trust_store(trust_store).expect("composed TLS client should build");
+
+        let (global_port, global_server) = spawn_single_request_https_server(&global_ca);
+        s3_client_with_http_client(global_port, http_client.clone())
+            .head_bucket()
+            .bucket("test-bucket")
+            .send()
+            .await
+            .expect("global RUSTFS_TLS_PATH CA should authenticate its TLS server");
+        global_server.join().expect("global CA TLS server should finish");
+
+        let (target_port, target_server) = spawn_single_request_https_server(&target_ca);
+        s3_client_with_http_client(target_port, http_client)
+            .head_bucket()
+            .bucket("test-bucket")
+            .send()
+            .await
+            .expect("per-target CA should authenticate its TLS server alongside global roots");
+        target_server.join().expect("target CA TLS server should finish");
+    }
+
+    #[tokio::test]
+    async fn tls_path_leaf_trust_remains_opt_in() {
+        let tls_dir = tempfile::tempdir().expect("temporary TLS directory should be created");
+        let global_ca =
+            generate_simple_self_signed(vec!["global-ca.example".to_string()]).expect("global CA certificate should generate");
+        let trusted_leaf =
+            generate_simple_self_signed(vec!["leaf.example".to_string()]).expect("trusted leaf certificate should generate");
+        tokio::fs::write(tls_dir.path().join(RUSTFS_CA_CERT), global_ca.cert.pem())
+            .await
+            .expect("global CA bundle should be written");
+        tokio::fs::write(tls_dir.path().join(RUSTFS_TLS_CERT), trusted_leaf.cert.pem())
+            .await
+            .expect("trusted leaf certificate should be written");
+
+        assert_eq!(load_tls_path_ca_bundles(tls_dir.path(), true).await.len(), 2);
+        assert_eq!(load_tls_path_ca_bundles(tls_dir.path(), false).await.len(), 1);
+    }
+
+    #[tokio::test]
+    async fn skip_tls_verify_takes_priority_over_invalid_custom_ca_pem() {
+        let client = build_aws_s3_http_client_for_target(&BucketTarget {
+            secure: true,
+            skip_tls_verify: true,
+            ca_cert_pem: "not a pem".to_string(),
+            ..Default::default()
+        })
+        .await
+        .expect("skip verification should bypass custom CA parsing");
+
+        assert!(client.is_some(), "secure targets with skip verification need a custom HTTP client");
+    }
+
    #[tokio::test]
    async fn get_remote_target_client_internal_rejects_invalid_custom_ca_pem() {
        let sys = BucketTargetSys::default();
@@ -2267,6 +2445,31 @@ mod tests {
        assert!(err.to_string().contains("invalid target CA PEM"));
    }

+    #[test]
+    fn target_ca_rejects_pem_wrapped_invalid_der_before_smithy_builds() {
+        let err = validate_target_ca_pem("-----BEGIN CERTIFICATE-----\nAQID\n-----END CERTIFICATE-----\n")
+            .expect_err("PEM-wrapped invalid DER must be rejected");
+
+        assert!(err.to_string().contains("invalid target CA PEM"));
+        assert!(err.to_string().contains("invalid X.509 certificate"));
+    }
+
+    #[tokio::test]
+    async fn invalid_global_ca_is_ignored_without_reaching_smithy() {
+        let tls_dir = tempfile::tempdir().expect("temporary TLS directory should be created");
+        tokio::fs::write(
+            tls_dir.path().join(RUSTFS_CA_CERT),
+            b"-----BEGIN CERTIFICATE-----\nAQID\n-----END CERTIFICATE-----\n",
+        )
+        .await
+        .expect("invalid global CA fixture should be written");
+
+        assert!(
+            load_tls_path_ca_bundles(tls_dir.path(), false).await.is_empty(),
+            "invalid global CA must fall back to default roots instead of reaching Smithy's panic path"
+        );
+    }
+
    // backlog#806-16 regression tests for the rolling one-minute latency window.

    #[test]
@@ -836,6 +836,12 @@ pub struct TransitionState {
    compensation_scheduled_tasks: AtomicI64,
    compensation_running_tasks: AtomicI64,
    compensation_buckets: Arc<Mutex<HashSet<String>>>,
+    // (bucket, object, version) currently queued or being transitioned. A single
+    // PUT can enqueue the same object twice (immediate transition + startup
+    // compensation backfill); without this guard the duplicates run concurrently,
+    // and the loser races the winner's source cleanup — reading data the winner
+    // already removed and logging a spurious NotFound failure (rustfs/backlog#1268).
+    in_flight_transitions: Arc<Mutex<HashSet<(String, String, String)>>>,
    last_day_stats: Arc<Mutex<HashMap<String, LastDayTierStats>>>,
 }

@@ -869,10 +875,39 @@ impl TransitionState {
            compensation_scheduled_tasks: AtomicI64::new(0),
            compensation_running_tasks: AtomicI64::new(0),
            compensation_buckets: Arc::new(Mutex::new(HashSet::new())),
+            in_flight_transitions: Arc::new(Mutex::new(HashSet::new())),
            last_day_stats: Arc::new(Mutex::new(HashMap::new())),
        })
    }

+    fn transition_key(oi: &ObjectInfo) -> (String, String, String) {
+        (
+            oi.bucket.clone(),
+            oi.name.clone(),
+            oi.version_id.map(|v| v.to_string()).unwrap_or_default(),
+        )
+    }
+
+    /// Try to claim a transition for this exact (bucket, object, version). Returns
+    /// `false` when one is already queued or running, so the caller can skip the
+    /// duplicate enqueue (rustfs/backlog#1268). The claim is released in the
+    /// worker once the transition finishes, or here if the enqueue fails.
+    fn reserve_transition(&self, oi: &ObjectInfo) -> bool {
+        let key = Self::transition_key(oi);
+        match self.in_flight_transitions.lock() {
+            Ok(mut set) => set.insert(key),
+            Err(poisoned) => poisoned.into_inner().insert(key),
+        }
+    }
+
+    fn release_transition(&self, oi: &ObjectInfo) {
+        let key = Self::transition_key(oi);
+        match self.in_flight_transitions.lock() {
+            Ok(mut set) => set.remove(&key),
+            Err(poisoned) => poisoned.into_inner().remove(&key),
+        };
+    }
+
    fn reserve_bucket_compensation(&self, bucket: &str) -> bool {
        let inserted = match self.compensation_buckets.lock() {
            Ok(mut scheduled) => scheduled.insert(bucket.to_string()),
@@ -1046,6 +1081,15 @@ impl TransitionState {
            return false;
        }

+        // Deduplicate concurrent enqueues of the same object version. The claim is
+        // released by the worker after the transition finishes (rustfs/backlog#1268).
+        // This is a no-op, not a new enqueue, so it must not touch the enqueue
+        // counters (the first enqueue already counted this object).
+        if !self.reserve_transition(oi) {
+            self.record_scanner_transition_state();
+            return true;
+        }
+
        let task = TransitionTask {
            obj_info: oi.clone(),
            src: src.clone(),
@@ -1084,6 +1128,9 @@ impl TransitionState {
                    self.handle_immediate_enqueue_failure(oi, src, ImmediateEnqueueFailure::QueueClosed { timeout_ms: None });
                }
            }
+            if !queued {
+                self.release_transition(oi);
+            }
            record_scanner_transition_enqueue_result(src, 1, queued);
            self.record_scanner_transition_state();
            return queued;
@@ -1113,6 +1160,9 @@ impl TransitionState {
                );
            }
        }
+        if !queued {
+            self.release_transition(oi);
+        }
        record_scanner_transition_enqueue_result(src, 1, queued);
        self.record_scanner_transition_state();
        queued
@@ -1239,6 +1289,9 @@ impl TransitionState {

                            emit_transition_complete_event(obj_info_for_event);
                        }
+                        // Release the dedup claim so a later lifecycle pass can
+                        // re-transition this object if needed (rustfs/backlog#1268).
+                        transition_state.release_transition(&task.obj_info);
                        TransitionState::add_counter(&transition_state.active_tasks, -1);
                        transition_state.record_scanner_transition_state();
                    }
@@ -3244,8 +3297,16 @@ mod tests {
            ..Default::default()
        };

+        // A distinct object fills past the capacity-1 queue and is reported as a
+        // missed enqueue. (Re-enqueuing the same object would instead be deduped;
+        // see transition_reserve_dedupes_same_object_version.)
+        let other = ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object-2".to_string(),
+            ..Default::default()
+        };
        let first = state.queue_transition_task(&object, &event, &LcEventSrc::Scanner).await;
-        let second = state.queue_transition_task(&object, &event, &LcEventSrc::Scanner).await;
+        let second = state.queue_transition_task(&other, &event, &LcEventSrc::Scanner).await;

        assert!(first);
        assert!(!second);
@@ -3267,8 +3328,13 @@ mod tests {
            ..Default::default()
        };

+        let other = ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object-2".to_string(),
+            ..Default::default()
+        };
        let first = state.queue_transition_task(&object, &event, &LcEventSrc::Scanner).await;
-        let second = state.queue_transition_task(&object, &event, &LcEventSrc::Scanner).await;
+        let second = state.queue_transition_task(&other, &event, &LcEventSrc::Scanner).await;

        assert!(first);
        assert!(!second);
@@ -3769,6 +3835,62 @@ mod tests {
        assert_eq!(state.scanner_transition_state_update().compensation_pending, 1);
    }

+    #[test]
+    fn transition_reserve_dedupes_same_object_version() {
+        // Regression for rustfs/backlog#1268: a single object version can be
+        // enqueued twice (immediate transition + compensation backfill). The
+        // second claim must be rejected until the first is released, so the two
+        // do not run concurrently and race the source cleanup.
+        let state = TransitionState::new_with_capacity(4);
+
+        let oi = ObjectInfo {
+            bucket: "foo".to_string(),
+            name: "payload.bin".to_string(),
+            version_id: None,
+            ..Default::default()
+        };
+
+        assert!(state.reserve_transition(&oi), "first claim must succeed");
+        assert!(!state.reserve_transition(&oi), "duplicate claim must be rejected");
+
+        // A different object is independent.
+        let other = ObjectInfo {
+            bucket: "foo".to_string(),
+            name: "other.bin".to_string(),
+            version_id: None,
+            ..Default::default()
+        };
+        assert!(state.reserve_transition(&other), "distinct object must claim independently");
+
+        // After release, the same object can be re-claimed (later lifecycle pass).
+        state.release_transition(&oi);
+        assert!(state.reserve_transition(&oi), "re-claim after release must succeed");
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn queue_transition_task_dedupes_same_object_without_second_enqueue() {
+        // Capacity 4 leaves room, so a rejected second enqueue can only be the
+        // dedup guard, not queue pressure (rustfs/backlog#1268).
+        let state = TransitionState::new_with_capacity(4);
+        let object = ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object".to_string(),
+            ..Default::default()
+        };
+        let event = crate::bucket::lifecycle::lifecycle::Event {
+            action: IlmAction::TransitionAction,
+            ..Default::default()
+        };
+
+        let first = state.queue_transition_task(&object, &event, &LcEventSrc::Scanner).await;
+        let second = state.queue_transition_task(&object, &event, &LcEventSrc::Scanner).await;
+
+        assert!(first, "first enqueue must succeed");
+        assert!(second, "duplicate enqueue is reported handled (already queued)");
+        assert_eq!(state.transition_rx.len(), 1, "only one task must actually be queued");
+    }
+
    #[tokio::test]
    #[serial]
    async fn transition_state_init_honors_runtime_configured_worker_count() {
@@ -260,7 +260,9 @@ pub(crate) fn replication_put_object_header_size(put_options: &PutObjectOptions)

 fn replication_source_object(object_info: &ObjectInfo) -> ReplicationSourceObject<'_> {
    ReplicationSourceObject {
-        mod_time: object_info.mod_time,
+        mod_time: object_info
+            .mod_time
+            .map(|mod_time| OffsetDateTime::from_unix_timestamp(mod_time.unix_timestamp()).unwrap_or(mod_time)),
        version_id: object_info.version_id.map(|version_id| version_id.to_string()),
        etag: object_info.etag.as_deref(),
        actual_size: object_info.get_actual_size().unwrap_or_default(),
@@ -463,6 +465,35 @@ mod tests {
        );
    }

+    #[test]
+    fn replication_action_for_target_head_compares_http_date_precision() {
+        for (source_nanos, target_secs, expected) in [
+            (10_123_456_789, 10, ReplicationAction::None),
+            (-10_876_543_211, -11, ReplicationAction::None),
+            (10_600_000_000, 11, ReplicationAction::All),
+        ] {
+            let mod_time = OffsetDateTime::from_unix_timestamp_nanos(source_nanos).expect("valid timestamp");
+            let object_info = ObjectInfo {
+                mod_time: Some(mod_time),
+                version_id: Some(Uuid::new_v4()),
+                etag: Some("abc123".to_string()),
+                size: 10,
+                ..Default::default()
+            };
+            let target = HeadObjectOutput::builder()
+                .last_modified(DateTime::from_secs(target_secs))
+                .version_id(object_info.version_id.expect("version ID").to_string())
+                .e_tag("abc123")
+                .content_length(10)
+                .build();
+
+            assert_eq!(
+                replication_action_for_target_head(&object_info, &target, ReplicationType::Object),
+                expected
+            );
+        }
+    }
+
    #[test]
    fn replication_remove_options_mark_replication_requests() {
        let mtime = OffsetDateTime::UNIX_EPOCH + Duration::seconds(10);
@@ -585,4 +616,43 @@ mod tests {

        assert!(err.to_string().contains(ERR_REPLICATION_MANAGED_SSE_UNSUPPORTED));
    }
+
+    // T3 (#1264): the outbound replication path forwards a stored object checksum into
+    // user_metadata via decrypt_checksums, which is algorithm-agnostic. This locks that
+    // the AWS 2026-04 additional algorithms (XXHash3/64/128, SHA-512, MD5) are forwarded
+    // identically to the classic five — i.e. replication treats the new algorithms
+    // consistently, with no new-algorithm-specific gap on the outbound side.
+    #[test]
+    fn replication_put_object_options_forwards_new_algorithm_checksums_like_classic() {
+        use rustfs_rio::{Checksum, ChecksumType};
+
+        let payload = b"replication checksum consistency payload";
+        let cases = [
+            // classic five (baseline)
+            ("CRC32", ChecksumType::CRC32),
+            ("SHA256", ChecksumType::SHA256),
+            // AWS 2026-04 additional algorithms
+            ("XXHASH3", ChecksumType::XXHASH3),
+            ("XXHASH64", ChecksumType::XXHASH64),
+            ("XXHASH128", ChecksumType::XXHASH128),
+            ("SHA512", ChecksumType::SHA512),
+            ("MD5", ChecksumType::MD5),
+        ];
+
+        for (name, ty) in cases {
+            let checksum = Checksum::new_from_data(ty, payload).expect("compute checksum");
+            let object_info = ObjectInfo {
+                checksum: Some(checksum.to_bytes(&[])),
+                ..Default::default()
+            };
+
+            let (opts, _is_multipart) = replication_put_object_options("", &object_info).expect("build replication put options");
+
+            assert_eq!(
+                opts.user_metadata.get(name),
+                Some(&checksum.encoded),
+                "replication must forward the {name} checksum into user_metadata identically to the classic algorithms"
+            );
+        }
+    }
 }
@@ -120,7 +120,11 @@ impl Default for PutObjectOptions {
            storage_class: "".to_string(),
            website_redirect_location: "".to_string(),
            part_size: 0,
-            legalhold: ObjectLockLegalHoldStatus::from_static(ObjectLockLegalHoldStatus::OFF),
+            // Empty, not OFF: `header()` emits x-amz-object-lock-legal-hold for
+            // any non-empty status, and CompleteMultipartUpload rejects requests
+            // that carry object-lock headers, breaking multipart transitions
+            // (rustfs/rustfs#4811). Only send the header when a status is set.
+            legalhold: ObjectLockLegalHoldStatus::from_static(""),
            send_content_md5: false,
            disable_content_sha256: false,
            disable_multipart: false,
@@ -18,6 +18,7 @@
 #![allow(clippy::all)]

 use http::{HeaderMap, HeaderName, StatusCode};
+use http_body_util::BodyExt;
 use hyper::body::Bytes;
 use s3s::S3ErrorCode;
 use std::collections::HashMap;
@@ -244,7 +245,23 @@ impl TransitionClient {
            )));
        }
        //}
-        let initiate_multipart_upload_result = InitiateMultipartUploadResult::default();
+        // Parse the CreateMultipartUpload response for the UploadId. Returning a
+        // default (empty) result here made every multipart transition fail at the
+        // first UploadPart with "UploadID cannot be empty" (rustfs/rustfs#4811).
+        let mut body_vec = Vec::new();
+        let mut body = resp.into_body();
+        while let Some(frame) = body.frame().await {
+            let frame = frame.map_err(|e| std::io::Error::other(e.to_string()))?;
+            if let Some(data) = frame.data_ref() {
+                body_vec.extend_from_slice(data);
+            }
+        }
+        let initiate_multipart_upload_result =
+            quick_xml::de::from_str::<InitiateMultipartUploadResult>(&String::from_utf8_lossy(&body_vec))
+                .map_err(|e| std::io::Error::other(format!("failed to parse CreateMultipartUpload response: {e}")))?;
+        if initiate_multipart_upload_result.upload_id.is_empty() {
+            return Err(std::io::Error::other("CreateMultipartUpload response missing UploadId"));
+        }
        Ok(initiate_multipart_upload_result)
    }

@@ -432,3 +449,56 @@ pub struct UploadPartParams {
    pub custom_header: HeaderMap,
    pub trailer: HeaderMap,
 }
+
+#[cfg(test)]
+mod tests {
+    use crate::client::api_s3_datatypes::{CompleteMultipartUpload, CompletePart, InitiateMultipartUploadResult};
+
+    #[test]
+    fn complete_multipart_upload_serializes_s3_part_elements() {
+        // Regression for rustfs/rustfs#4811: without serde renames quick-xml emits
+        // <parts>/<part_num>/<etag>, so the remote parses zero <Part> elements and
+        // completes a 0-byte object (while still returning 200). The body must use
+        // S3 element names, and MD5-only transitions must not emit empty checksum
+        // elements.
+        let complete = CompleteMultipartUpload {
+            parts: vec![
+                CompletePart {
+                    part_num: 1,
+                    etag: "etag-one".to_string(),
+                    ..Default::default()
+                },
+                CompletePart {
+                    part_num: 2,
+                    etag: "etag-two".to_string(),
+                    ..Default::default()
+                },
+            ],
+        };
+        let xml = complete.marshal_msg().expect("marshal");
+        assert!(xml.contains("<Part>"), "missing <Part>: {xml}");
+        assert!(xml.contains("<PartNumber>1</PartNumber>"), "missing PartNumber: {xml}");
+        assert!(xml.contains("<ETag>etag-one</ETag>"), "missing ETag: {xml}");
+        assert!(xml.contains("<PartNumber>2</PartNumber>"), "missing part 2: {xml}");
+        assert!(!xml.contains("part_num"), "leaked rust field name: {xml}");
+        assert!(!xml.contains("<ChecksumCRC32>"), "emitted empty checksum: {xml}");
+    }
+
+    #[test]
+    fn parses_create_multipart_upload_response() {
+        // Regression for rustfs/rustfs#4811: `initiate_multipart_upload` used to
+        // return a default (empty) result, so every multipart transition failed
+        // the first UploadPart with "UploadID cannot be empty". The UploadId must
+        // be parsed out of the S3 XML response.
+        let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
+<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+  <Bucket>bar</Bucket>
+  <Key>foo/payload.bin</Key>
+  <UploadId>a1b2c3-d4e5-f6</UploadId>
+</InitiateMultipartUploadResult>"#;
+        let parsed: InitiateMultipartUploadResult = quick_xml::de::from_str(xml).expect("parse");
+        assert_eq!(parsed.upload_id, "a1b2c3-d4e5-f6");
+        assert_eq!(parsed.bucket, "bar");
+        assert_eq!(parsed.key, "foo/payload.bin");
+    }
+}
@@ -25,6 +25,7 @@ use std::io::Error;
 use std::sync::RwLock;
 use std::{collections::HashMap, sync::Arc};
 use time::{OffsetDateTime, format_description};
+use tokio::io::AsyncReadExt;
 use tokio::{select, sync::mpsc};
 use tokio_util::sync::CancellationToken;
 use tracing::warn;
@@ -45,6 +46,33 @@ use crate::client::utils::base64_encode;
 use rustfs_utils::path::trim_etag;
 use s3s::header::{X_AMZ_EXPIRATION, X_AMZ_VERSION_ID};

+/// Read exactly `want` bytes for a single multipart part, or fewer if the reader
+/// reaches EOF first. Advances the reader so the next call returns the following
+/// part. Replaces the previous per-part `read_all()`/`to_vec()`, which drained
+/// the entire source into the first part and left later parts empty
+/// (rustfs/rustfs#4811).
+async fn read_multipart_part(reader: &mut ReaderImpl, want: usize) -> Result<Vec<u8>, std::io::Error> {
+    match reader {
+        ReaderImpl::Body(content_body) => {
+            let take = content_body.len().min(want);
+            Ok(content_body.split_to(take).to_vec())
+        }
+        ReaderImpl::ObjectBody(content_body) => {
+            let mut buf = vec![0u8; want];
+            let mut filled = 0;
+            while filled < want {
+                let n = content_body.read(&mut buf[filled..]).await?;
+                if n == 0 {
+                    break;
+                }
+                filled += n;
+            }
+            buf.truncate(filled);
+            Ok(buf)
+        }
+    }
+}
+
 pub struct UploadedPartRes {
    pub error: std::io::Error,
    pub part_num: i64,
@@ -133,7 +161,6 @@ impl TransitionClient {
        let mut total_uploaded_size: i64 = 0;

        let mut parts_info = HashMap::<i64, ObjectPart>::new();
-        let mut buf = Vec::<u8>::with_capacity(part_size as usize);

        let mut md5_base64: String = "".to_string();
        for part_number in 1..=total_parts_count {
@@ -141,14 +168,12 @@ impl TransitionClient {
                part_size = lastpart_size;
            }

-            match &mut reader {
-                ReaderImpl::Body(content_body) => {
-                    buf = content_body.to_vec();
-                }
-                ReaderImpl::ObjectBody(content_body) => {
-                    buf = content_body.read_all().await?;
-                }
-            }
+            // Read exactly this part's bytes. Using `read_all()`/`to_vec()` here
+            // drained the whole source into the first part and left every later
+            // part empty, silently corrupting any multipart upload of a streamed
+            // (`ObjectBody`) source — e.g. ILM transitions of >128 MiB objects,
+            // which split into 128 MiB parts (rustfs/rustfs#4811).
+            let buf = read_multipart_part(&mut reader, part_size as usize).await?;
            let length = buf.len();

            if opts.send_content_md5 {
@@ -159,7 +184,7 @@ impl TransitionClient {
                };
                let hash = md5_hash.hash_encode(&buf[..length]);
                md5_base64 = base64_encode(hash.as_ref());
-            } else {
+            } else if opts.auto_checksum.is_set() {
                let mut crc = opts.auto_checksum.hasher()?;
                crc.update(&buf[..length]);
                let csum = crc.finalize();
@@ -174,6 +199,10 @@ impl TransitionClient {
                    warn!("Invalid header name: {}", opts.auto_checksum.key());
                }
            }
+            // else: neither MD5 nor a concrete additional checksum was requested,
+            // so upload the part without a per-part checksum header. Guarding the
+            // branch on `is_set()` avoids calling `hasher()` on `ChecksumNone`,
+            // which errors with "unsupported checksum type" (rustfs/rustfs#4811).

            let hooked = ReaderImpl::Body(Bytes::from(buf)); //newHook(BufferReader::new(buf), opts.progress);
            let mut p = UploadPartParams {
@@ -183,7 +212,9 @@ impl TransitionClient {
                reader: hooked,
                part_number,
                md5_base64: md5_base64.clone(),
-                size: part_size,
+                // Use the bytes actually read, not the planned part_size, so the
+                // uploaded Content-Length matches the body even on a short read.
+                size: length as i64,
                //sse: opts.server_side_encryption,
                stream_sha256: !opts.disable_content_sha256,
                custom_header: custom_header.clone(),
@@ -194,7 +225,7 @@ impl TransitionClient {

            parts_info.entry(part_number).or_insert(obj_part);

-            total_uploaded_size += part_size as i64;
+            total_uploaded_size += length as i64;
        }

        if size > 0 && total_uploaded_size != size {
@@ -593,9 +624,79 @@ fn collect_complete_parts(parts_info: &HashMap<i64, ObjectPart>, total_parts_cou

 #[cfg(test)]
 mod tests {
-    use super::{ObjectPart, collect_complete_parts};
+    use super::{ObjectPart, ReaderImpl, collect_complete_parts, read_multipart_part};
+    use crate::object_api::GetObjectReader;
+    use bytes::Bytes;
    use std::collections::HashMap;

+    // Drive a reader through the same per-part loop the multipart stream uses and
+    // collect the size of every part. Regression for rustfs/rustfs#4811: the old
+    // `read_all()` per part drained the whole source into part 1.
+    async fn collect_part_sizes(mut reader: ReaderImpl, total: usize, part_size: usize, last_part_size: usize) -> Vec<usize> {
+        let parts = total.div_ceil(part_size);
+        let mut sizes = Vec::new();
+        for part_number in 1..=parts {
+            let want = if part_number == parts { last_part_size } else { part_size };
+            let buf = read_multipart_part(&mut reader, want).await.unwrap();
+            sizes.push(buf.len());
+        }
+        // Nothing must remain after the planned parts are consumed.
+        assert!(read_multipart_part(&mut reader, part_size).await.unwrap().is_empty());
+        sizes
+    }
+
+    #[tokio::test]
+    async fn read_multipart_part_splits_streamed_object_body_evenly() {
+        // 250 bytes at part_size 100 -> parts [100, 100, 50], mirroring the
+        // >128 MiB / 128 MiB split from the bug report on a small deterministic
+        // stream.
+        let total = 250usize;
+        let (mut w, r) = tokio::io::duplex(64);
+        tokio::spawn(async move {
+            use tokio::io::AsyncWriteExt;
+            let data: Vec<u8> = (0..total).map(|i| i as u8).collect();
+            w.write_all(&data).await.unwrap();
+        });
+        let reader = ReaderImpl::ObjectBody(GetObjectReader {
+            stream: Box::new(r),
+            object_info: Default::default(),
+            buffered_body: None,
+            body_source: Default::default(),
+        });
+
+        let sizes = collect_part_sizes(reader, total, 100, 50).await;
+        assert_eq!(sizes, vec![100, 100, 50]);
+    }
+
+    #[tokio::test]
+    async fn read_multipart_part_splits_in_memory_body_evenly() {
+        let total = 250usize;
+        let data: Vec<u8> = (0..total).map(|i| i as u8).collect();
+        let reader = ReaderImpl::Body(Bytes::from(data));
+
+        let sizes = collect_part_sizes(reader, total, 100, 50).await;
+        assert_eq!(sizes, vec![100, 100, 50]);
+    }
+
+    #[tokio::test]
+    async fn read_multipart_part_stops_at_eof_without_overrun() {
+        // Reader shorter than the requested part size must return only what is
+        // available, not block or pad.
+        let (mut w, r) = tokio::io::duplex(64);
+        tokio::spawn(async move {
+            use tokio::io::AsyncWriteExt;
+            w.write_all(&[1u8; 30]).await.unwrap();
+        });
+        let mut reader = ReaderImpl::ObjectBody(GetObjectReader {
+            stream: Box::new(r),
+            object_info: Default::default(),
+            buffered_body: None,
+            body_source: Default::default(),
+        });
+        let buf = read_multipart_part(&mut reader, 100).await.unwrap();
+        assert_eq!(buf.len(), 30);
+    }
+
    fn parts_map(n: i64) -> HashMap<i64, ObjectPart> {
        let mut m = HashMap::new();
        for i in 1..=n {
@@ -209,7 +209,8 @@ pub struct ListObjectPartsResult {
    pub encoding_type: String,
 }

-#[derive(Debug, Default)]
+#[derive(Debug, Default, Serialize, Deserialize)]
+#[serde(default, rename_all = "PascalCase")]
 pub struct InitiateMultipartUploadResult {
    pub bucket: String,
    pub key: String,
@@ -229,15 +230,28 @@ pub struct CompleteMultipartUploadResult {
    pub checksum_crc64nvme: String,
 }

+// Field renames drive the CompleteMultipartUpload XML body sent to the remote.
+// Without them quick-xml emits the Rust field names (<part_num>/<etag>), so the
+// remote parses zero <Part> elements and completes a 0-byte object while still
+// returning 200 — the multipart transition silently produces an empty object
+// (rustfs/rustfs#4811). Empty checksum fields are skipped so an MD5-only
+// transition does not emit blank <ChecksumCRC32/> elements.
 #[derive(Clone, Debug, Default, PartialEq, Eq, PartialOrd, Ord, serde::Serialize)]
 pub struct CompletePart {
    //api has
-    pub etag: String,
+    #[serde(rename = "PartNumber")]
    pub part_num: i64,
+    #[serde(rename = "ETag")]
+    pub etag: String,
+    #[serde(rename = "ChecksumCRC32", skip_serializing_if = "String::is_empty")]
    pub checksum_crc32: String,
+    #[serde(rename = "ChecksumCRC32C", skip_serializing_if = "String::is_empty")]
    pub checksum_crc32c: String,
+    #[serde(rename = "ChecksumSHA1", skip_serializing_if = "String::is_empty")]
    pub checksum_sha1: String,
+    #[serde(rename = "ChecksumSHA256", skip_serializing_if = "String::is_empty")]
    pub checksum_sha256: String,
+    #[serde(rename = "ChecksumCRC64NVME", skip_serializing_if = "String::is_empty")]
    pub checksum_crc64nvme: String,
 }

@@ -272,7 +286,9 @@ pub struct CopyObjectPartResult {
 }

 #[derive(Debug, Default, serde::Serialize)]
+#[serde(rename = "CompleteMultipartUpload")]
 pub struct CompleteMultipartUpload {
+    #[serde(rename = "Part")]
    pub parts: Vec<CompletePart>,
 }

@@ -71,7 +71,11 @@ impl ChecksumMode {
            8_u8 => ChecksumMode::ChecksumCRC32,
            16_u8 => ChecksumMode::ChecksumCRC32C,
            32_u8 => ChecksumMode::ChecksumCRC64NVME,
-            _ => panic!("enum err."),
+            // Fail closed: any mode without a concrete base algorithm (e.g. a
+            // bare ChecksumFullObject flag) is treated as "no checksum" rather
+            // than panicking. Callers already gate real work behind
+            // is_set()/can_composite()/hasher(), so this only removes a crash.
+            _ => ChecksumMode::ChecksumNone,
        }
    }

@@ -177,6 +181,17 @@ impl ChecksumMode {
    }

    pub fn is_set(&self) -> bool {
+        // `ChecksumNone` is the zeroth enum variant, so it occupies bit 0 of the
+        // `EnumSet` repr and a naive `len() == 1` check reports "no checksum" as a
+        // configured checksum. A checksum is only "set" when a concrete algorithm
+        // (one with a real hasher) is selected; the bare `ChecksumFullObject` flag
+        // has no base algorithm and is likewise not set. Treating `ChecksumNone`
+        // as set made ILM transitions of >128 MiB objects fail with
+        // "unsupported checksum type" (rustfs/rustfs#4811): the multipart put path
+        // took the checksum branch and called `ChecksumNone.hasher()`.
+        if matches!(self, ChecksumMode::ChecksumNone) {
+            return false;
+        }
        let s = EnumSet::from(*self).intersection(*C_ChecksumMask);
        s.len() == 1
    }
@@ -272,6 +287,70 @@ impl ChecksumMode {
    }
 }

+#[cfg(test)]
+mod tests {
+    use super::ChecksumMode;
+
+    #[test]
+    fn test_base_is_fail_closed_and_never_panics() {
+        // Every mode must resolve to a concrete base without panicking. The bare
+        // ChecksumFullObject flag has no base algorithm and must fall back to
+        // ChecksumNone instead of crashing (previously `panic!("enum err.")`).
+        assert_eq!(ChecksumMode::ChecksumFullObject.base(), ChecksumMode::ChecksumNone);
+        assert_eq!(ChecksumMode::ChecksumNone.base(), ChecksumMode::ChecksumNone);
+        assert_eq!(ChecksumMode::ChecksumCRC32.base(), ChecksumMode::ChecksumCRC32);
+        assert_eq!(ChecksumMode::ChecksumCRC32C.base(), ChecksumMode::ChecksumCRC32C);
+        assert_eq!(ChecksumMode::ChecksumSHA1.base(), ChecksumMode::ChecksumSHA1);
+        assert_eq!(ChecksumMode::ChecksumSHA256.base(), ChecksumMode::ChecksumSHA256);
+        assert_eq!(ChecksumMode::ChecksumCRC64NVME.base(), ChecksumMode::ChecksumCRC64NVME);
+    }
+
+    #[test]
+    fn test_hasher_fails_closed_for_unsupported_mode() {
+        // Modes without a real hasher must return an error, not panic.
+        assert!(ChecksumMode::ChecksumNone.hasher().is_err());
+        assert!(ChecksumMode::ChecksumFullObject.hasher().is_err());
+        assert!(ChecksumMode::ChecksumCRC32.hasher().is_ok());
+    }
+
+    #[test]
+    fn test_is_set_is_false_for_none_and_bare_full_object() {
+        // Regression for rustfs/rustfs#4811: `ChecksumNone` must NOT be reported as
+        // a configured checksum. It is the zeroth enum variant (bit 0 of the
+        // EnumSet repr), so the old `len() == 1` check treated it as set and drove
+        // the multipart put path into `ChecksumNone.hasher()` → "unsupported
+        // checksum type". Every mode reported as set must also have a real hasher.
+        assert!(!ChecksumMode::ChecksumNone.is_set());
+        assert!(!ChecksumMode::ChecksumFullObject.is_set());
+
+        for mode in [
+            ChecksumMode::ChecksumCRC32,
+            ChecksumMode::ChecksumCRC32C,
+            ChecksumMode::ChecksumSHA1,
+            ChecksumMode::ChecksumSHA256,
+            ChecksumMode::ChecksumCRC64NVME,
+        ] {
+            assert!(mode.is_set(), "{mode:?} should be set");
+            assert!(mode.hasher().is_ok(), "{mode:?} reported set but has no hasher");
+        }
+    }
+
+    #[test]
+    fn test_set_default_upgrades_none() {
+        // With `is_set()` fixed, `set_default` must upgrade an unset mode to the
+        // provided default (previously `ChecksumNone` was seen as set and never
+        // upgraded).
+        let mut mode = ChecksumMode::ChecksumNone;
+        mode.set_default(ChecksumMode::ChecksumCRC32C);
+        assert_eq!(mode, ChecksumMode::ChecksumCRC32C);
+
+        // An already-set mode is left untouched.
+        let mut existing = ChecksumMode::ChecksumSHA256;
+        existing.set_default(ChecksumMode::ChecksumCRC32C);
+        assert_eq!(existing, ChecksumMode::ChecksumSHA256);
+    }
+}
+
 #[derive(Default)]
 pub struct Checksum {
    checksum_type: ChecksumMode,
@@ -121,8 +121,11 @@ pub fn new_getobjectreader<'a>(
    let is_compressed = false; //oi.is_compressed_ok();

    let rs_;
-    if rs.is_none() && opts.part_number.is_some() && opts.part_number.expect("operation should succeed") > 0 {
-        rs_ = part_number_to_rangespec(oi.clone(), opts.part_number.expect("operation should succeed"));
+    if rs.is_none()
+        && let Some(part_number) = opts.part_number
+        && part_number > 0
+    {
+        rs_ = part_number_to_rangespec(oi.clone(), part_number);
    } else {
        rs_ = rs.clone();
    }
@@ -158,6 +161,16 @@ pub fn new_getobjectreader<'a>(

        return Ok((get_fn, off as i64, length as i64));
    }
+    if rs.is_none() && opts.part_number.is_none() && oi.size >= 0 {
+        get_fn = Arc::new(move |input_reader: BufReader<Cursor<Vec<u8>>>, _: HeaderMap| GetObjectReader {
+            object_info: oi.clone(),
+            stream: Box::new(input_reader),
+            buffered_body: None,
+            body_source: Default::default(),
+        });
+
+        return Ok((get_fn, 0, oi.size));
+    }
    Err(ErrorResponse {
        code: S3ErrorCode::InvalidRange,
        message: "Invalid range".to_string(),
@@ -198,6 +211,57 @@ pub fn get_raw_etag(metadata: &HashMap<String, String>) -> String {
 #[cfg(test)]
 mod tests {
    use super::*;
+    use std::sync::Arc;
+
+    fn multipart_object_info() -> ObjectInfo {
+        ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object".to_string(),
+            size: 6 * 1024 * 1024,
+            actual_size: 6 * 1024 * 1024,
+            parts: Arc::new(vec![
+                ObjectPartInfo {
+                    number: 1,
+                    size: 5 * 1024 * 1024,
+                    actual_size: 5 * 1024 * 1024,
+                    ..Default::default()
+                },
+                ObjectPartInfo {
+                    number: 2,
+                    size: 1024 * 1024,
+                    actual_size: 1024 * 1024,
+                    ..Default::default()
+                },
+            ]),
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn test_new_getobjectreader_uses_full_range_for_unranged_multipart_object() {
+        let oi = multipart_object_info();
+        let opts = ObjectOptions::default();
+
+        let result = new_getobjectreader(&None, &oi, &opts, &HeaderMap::new())
+            .expect("unranged multipart object should build a full-object reader");
+
+        assert_eq!(result.1, 0);
+        assert_eq!(result.2, oi.size);
+    }
+
+    #[test]
+    fn test_new_getobjectreader_keeps_part_number_range_for_multipart_object() {
+        let oi = multipart_object_info();
+        let opts = ObjectOptions {
+            part_number: Some(2),
+            ..Default::default()
+        };
+
+        let result = new_getobjectreader(&None, &oi, &opts, &HeaderMap::new()).expect("part number should build that part range");
+
+        assert_eq!(result.1, 5 * 1024 * 1024);
+        assert_eq!(result.2, 1024 * 1024);
+    }

    #[test]
    fn test_to_s3s_etag() {
@@ -91,10 +91,41 @@ pub fn is_minio_header(header_key: &str) -> bool {
    header_key.to_lowercase().starts_with("x-minio-")
 }

+/// Standard base64 (with `+`/`/` and `=` padding). Every base64 value this
+/// transition client emits or parses — `Content-MD5`, `x-amz-checksum-*`, and
+/// checksum digests in request/response bodies — is S3 wire format, which is
+/// standard base64. The URL-safe, unpadded alphabet used previously made remotes
+/// reject `Content-MD5` with "Invalid content MD5: Base64Error" and could not
+/// even decode a padded checksum coming back from the peer (rustfs/rustfs#4811).
 pub fn base64_encode(input: &[u8]) -> String {
-    base64_simd::URL_SAFE_NO_PAD.encode_to_string(input)
+    base64_simd::STANDARD.encode_to_string(input)
 }

 pub fn base64_decode(input: &[u8]) -> Result<Vec<u8>, base64_simd::Error> {
-    base64_simd::URL_SAFE_NO_PAD.decode_to_vec(input)
+    base64_simd::STANDARD.decode_to_vec(input)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{base64_decode, base64_encode};
+
+    #[test]
+    fn base64_encode_is_standard_s3_wire_format() {
+        // S3 reads Content-MD5 / checksum values with a standard base64 decoder,
+        // so the encoder must emit '+'/'/' and '=' padding and round-trip through
+        // one. Regression for rustfs/rustfs#4811 ("Invalid content MD5:
+        // Base64Error"). 16-byte MD5-length input chosen to force '=' padding.
+        let digest: [u8; 16] = [
+            0xfb, 0xff, 0xff, 0xef, 0x10, 0x20, 0x30, 0x40, 0x50, 0x60, 0x70, 0x80, 0x90, 0xa0, 0xb0, 0xc0,
+        ];
+        let encoded = base64_encode(&digest);
+        assert!(encoded.ends_with('='), "16-byte input must be padded: {encoded}");
+        assert!(!encoded.contains(['-', '_']), "must use the standard alphabet: {encoded}");
+        let via_standard = base64_simd::STANDARD
+            .decode_to_vec(encoded.as_bytes())
+            .expect("standard decode");
+        assert_eq!(via_standard, digest);
+        // Our own decoder must accept the same wire format it produces.
+        assert_eq!(base64_decode(encoded.as_bytes()).expect("round-trip"), digest);
+    }
 }
@@ -23,17 +23,19 @@ use rustfs_config::{
    MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
    MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
    MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
-    MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT,
-    NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING,
-    POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT,
-    POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR,
-    PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC,
-    PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS,
-    REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT,
-    REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT,
-    REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA,
-    WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR,
-    WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY,
+    MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS,
+    NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR,
+    NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
+    NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
+    POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
+    PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
+    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
+    REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
+    REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
+    REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
+    REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY,
+    WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
+    WEBHOOK_SKIP_TLS_VERIFY,
 };
 use std::sync::LazyLock;

@@ -350,6 +352,21 @@ pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
            value: DEFAULT_LIMIT.to_string(),
            hidden_if_empty: false,
        },
+        KV {
+            key: NATS_JETSTREAM_ENABLE.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: NATS_JETSTREAM_STREAM_NAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(),
+            value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
+            hidden_if_empty: false,
+        },
        KV {
            key: COMMENT_KEY.to_owned(),
            value: "".to_owned(),
@@ -23,16 +23,18 @@ use rustfs_config::{
    MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
    MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
    MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
-    MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT,
-    NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING,
-    POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT,
-    POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR,
-    PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC,
-    PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS,
-    REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT,
-    REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT,
-    REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA,
-    WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
+    MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS,
+    NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR,
+    NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
+    NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
+    POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
+    PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
+    PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
+    REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
+    REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
+    REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
+    REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT,
+    WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
 };
 use std::sync::LazyLock;

@@ -328,6 +330,21 @@ pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
            value: DEFAULT_LIMIT.to_string(),
            hidden_if_empty: false,
        },
+        KV {
+            key: NATS_JETSTREAM_ENABLE.to_owned(),
+            value: EnableState::Off.to_string(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: NATS_JETSTREAM_STREAM_NAME.to_owned(),
+            value: "".to_owned(),
+            hidden_if_empty: false,
+        },
+        KV {
+            key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(),
+            value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
+            hidden_if_empty: false,
+        },
        KV {
            key: COMMENT_KEY.to_owned(),
            value: "".to_owned(),
@@ -0,0 +1,146 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Deterministic crash-point injection for erasure-coded commit sequences.
+//!
+//! Each [`CrashPoint`] names one instant inside a multi-step commit at which a
+//! `#[cfg(test)]` scenario can simulate a hard power loss: the commit stops dead
+//! at the armed step with **no** cleanup, leaving the on-disk state exactly as
+//! the preceding steps left it. The test then reopens the disk (or rebuilds the
+//! erasure set) and asserts the raw state is coherent — the object reads back as
+//! either the whole old version or the whole new version, never a torn mix — and
+//! that any staged leftovers are reclaimable and the operation is safely
+//! retryable. This is the crash-consistency counterpart to the graceful
+//! `should_fail_*` failpoints, which exercise in-process rollback rather than an
+//! abrupt loss.
+//!
+//! Unlike the graceful failpoints, a crash point runs no rollback: it models the
+//! process disappearing, so the assertion is purely over the bytes left on disk.
+//!
+//! Test-only: the arming registry and the real [`should_crash_at`] are
+//! `#[cfg(test)]`. In every other build `should_crash_at` is a const-`false`
+//! `#[inline(always)]` no-op, so the injection points on the real commit paths
+//! (`rename_data`, `write_all_meta`, `complete_multipart_upload`) compile away to
+//! nothing and add no branch to production writes. The [`CrashPoint`] variants
+//! are still constructed at those call sites in every build, so no variant is
+//! dead code.
+//!
+//! Coverage plan: rustfs/backlog#935 / rustfs/backlog#896 (rename_data),
+//! rustfs/backlog#864 (fault-interrupted overwrite/delete must not mutate a
+//! committed object), rustfs/backlog#878 (old-or-new-never-mixed).
+
+/// A named instant inside an erasure-coded commit sequence at which a test can
+/// simulate a hard power loss.
+///
+/// All points share one keyed registry (each arm is matched on `(point, key)`),
+/// so a scenario arms exactly the window under test and every other commit path
+/// runs untouched.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub(crate) enum CrashPoint {
+    /// `rename_data`: after the data dir is renamed into its destination but
+    /// before the old-metadata rollback backup is written. Pre-commit — xl.meta
+    /// is untouched, so a crash here must leave the object readable as the old
+    /// version (or absent when there was none).
+    RenameAfterDataRename,
+    /// `rename_data`: after the rollback backup is durable, immediately before
+    /// the xl.meta commit rename that makes the new version visible. Still
+    /// pre-commit, so a crash here must also leave the old version readable.
+    RenameAfterBackupBeforeMetaCommit,
+    /// `rename_data`: after the xl.meta commit rename has made the new version
+    /// visible, before the durability fsync and with **no** rollback. The commit
+    /// rename already landed, so a crash here must leave the object readable as
+    /// the new version (rustfs/backlog#878, new-version side).
+    RenameAfterMetaCommit,
+    /// `write_all_meta` (the atomic temp+rename behind `update_metadata` and
+    /// `write_metadata`): after the replacement xl.meta is staged in the tmp
+    /// bucket but before the rename that publishes it. Pre-commit — the
+    /// destination xl.meta is untouched, so a crash here must leave the object's
+    /// metadata byte-for-byte the old version (rustfs/backlog#864); the staged
+    /// tmp file is a harmless orphan swept by tmp-bucket GC.
+    MetaWriteAfterTmpBeforeRename,
+    /// `complete_multipart_upload`: after the upload is fully staged and locked
+    /// but before the authoritative `rename_data` commit. Pre-commit — no disk
+    /// has moved staged data, so a crash here must leave any prior committed
+    /// version byte-for-byte intact (rustfs/backlog#864) and the upload fully
+    /// retryable.
+    MultipartBeforeCommitRename,
+    /// `complete_multipart_upload`: after the `rename_data` commit succeeds but
+    /// before the stale `part.N.meta` cleanup. Post-commit — the new version is
+    /// durably committed and visible, so a crash here must leave the object
+    /// readable as the new version; the un-reclaimed staging parts are swept by
+    /// a retried completion or upload GC (rustfs/backlog#946).
+    MultipartAfterCommitBeforePartsCleanup,
+}
+
+#[cfg(test)]
+pub(crate) use armed::{arm, disarm, should_crash_at};
+
+/// Production no-op: the compiler inlines this to `false`, so armed crash points
+/// exist only in `#[cfg(test)]` builds and never add a branch to real commits.
+#[cfg(not(test))]
+#[inline(always)]
+pub(crate) fn should_crash_at(_point: CrashPoint, _key: &str) -> bool {
+    false
+}
+
+#[cfg(test)]
+mod armed {
+    use super::CrashPoint;
+    use std::sync::{Mutex, MutexGuard};
+
+    // A keyed multiset of pending arms, not a single slot: the rename_data /
+    // xl.meta scenarios serialize via `durability_mode_override` while the
+    // multipart scenarios use `#[serial]`, so the two groups can overlap under
+    // `cargo test`'s shared-process threads. Keying every arm on a unique object
+    // path lets concurrent scenarios coexist without clobbering each other; a
+    // match on `(point, key)` is unambiguous. `#[serial]` still gives the
+    // multipart tests their own serialization for unrelated global state.
+    static ARMED: Mutex<Vec<(CrashPoint, String)>> = Mutex::new(Vec::new());
+
+    fn guard() -> MutexGuard<'static, Vec<(CrashPoint, String)>> {
+        // A poisoned lock only means a prior test panicked mid-scenario; recover
+        // the registry rather than cascade the panic into unrelated tests.
+        ARMED.lock().unwrap_or_else(|poisoned| poisoned.into_inner())
+    }
+
+    /// Arm a one-shot crash: the next operation that reaches `point` while
+    /// committing `key` stops there. Consumed on the first match so it never
+    /// leaks into an unrelated commit. Arming the same `(point, key)` twice is
+    /// idempotent.
+    pub(crate) fn arm(point: CrashPoint, key: &str) {
+        let mut armed = guard();
+        if !armed.iter().any(|(p, k)| *p == point && k == key) {
+            armed.push((point, key.to_string()));
+        }
+    }
+
+    /// Clear the pending arm for exactly this `(point, key)`. A scenario retries
+    /// the same key after the crash, so teardown disarms in case the injection
+    /// was somehow never reached and would otherwise fire inside the retry.
+    pub(crate) fn disarm(point: CrashPoint, key: &str) {
+        guard().retain(|(p, k)| !(*p == point && k == key));
+    }
+
+    /// Returns `true` (consuming the arm) iff a crash is armed for exactly this
+    /// `point` and `key`.
+    pub(crate) fn should_crash_at(point: CrashPoint, key: &str) -> bool {
+        let mut armed = guard();
+        if let Some(idx) = armed.iter().position(|(p, k)| *p == point && k == key) {
+            armed.swap_remove(idx);
+            true
+        } else {
+            false
+        }
+    }
+}
@@ -217,6 +217,27 @@ pub fn get_object_disk_read_timeout() -> Duration {
    )
 }

+/// Per-shard erasure write stall budget: a shard write (or shutdown) that makes
+/// no forward progress for this long is failed and its disk dropped before
+/// commit. Re-armed on every shard write, so it bounds a stall rather than the
+/// whole transfer. `0` disables the deadline (wait indefinitely).
+pub fn get_object_disk_write_stall_timeout() -> Duration {
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT,
+        rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT,
+    ))
+}
+
+/// Optional absolute per-object erasure write cap (administrator slow-drip
+/// backstop). `0` (default) disables the cap; the per-shard stall timeout is the
+/// primary guarantee.
+pub fn get_object_disk_write_absolute_cap() -> Duration {
+    Duration::from_secs(rustfs_utils::get_env_u64(
+        rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP,
+        rustfs_config::DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP,
+    ))
+}
+
 pub fn get_drive_active_check_interval() -> Duration {
    Duration::from_secs(rustfs_utils::get_env_u64(
        rustfs_config::ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS,
@@ -1653,6 +1674,32 @@ mod tests {
        });
    }

+    #[test]
+    fn object_disk_write_stall_timeout_default_and_override() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, || {
+            assert_eq!(
+                get_object_disk_write_stall_timeout(),
+                Duration::from_secs(rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT)
+            );
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("9"), || {
+            assert_eq!(get_object_disk_write_stall_timeout(), Duration::from_secs(9));
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("0"), || {
+            assert!(get_object_disk_write_stall_timeout().is_zero(), "0 disables the stall deadline");
+        });
+    }
+
+    #[test]
+    fn object_disk_write_absolute_cap_defaults_disabled() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, || {
+            assert!(get_object_disk_write_absolute_cap().is_zero(), "absolute cap is disabled by default");
+        });
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, Some("120"), || {
+            assert_eq!(get_object_disk_write_absolute_cap(), Duration::from_secs(120));
+        });
+    }
+
    #[test]
    fn object_disk_read_timeout_uses_default_when_unset() {
        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_READ_TIMEOUT, || {
@@ -13,6 +13,7 @@
 // limitations under the License.

 use crate::config::storageclass::DEFAULT_INLINE_BLOCK;
+use crate::crash_inject::{self, CrashPoint};
 use crate::data_usage::local_snapshot::ensure_data_usage_layout;
 use crate::disk::disk_store::{get_drive_walkdir_stall_timeout, get_object_disk_read_timeout};
 use crate::disk::{
@@ -25,10 +26,7 @@ use crate::disk::{
    error::{DiskError, Error, FileAccessDeniedWithContext, Result},
    error_conv::{to_access_error, to_file_error, to_unformatted_disk_error, to_volume_error},
    format::FormatV3,
-    fs::{
-        O_APPEND, O_CREATE, O_RDONLY, O_TRUNC, O_WRONLY, access, access_std, lstat, lstat_std, remove, remove_all_std,
-        remove_std, rename,
-    },
+    fs::{O_APPEND, O_CREATE, O_RDONLY, O_TRUNC, O_WRONLY, access, lstat, lstat_std, remove, remove_all_std, remove_std, rename},
    os,
    os::{check_path_length, is_empty_dir, is_root_disk, rename_all, rename_all_ignore_missing_source},
 };
@@ -100,7 +98,7 @@ fn inline_metadata_rollback_dir(version_id: Uuid, meta: &FileMeta) -> Uuid {
 fn remove_file_if_exists(path: &Path) -> std::io::Result<()> {
    match std::fs::remove_file(path) {
        Ok(()) => Ok(()),
-        Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
+        Err(err) if err.kind() == ErrorKind::NotFound => Ok(()),
        Err(err) => Err(err),
    }
 }
@@ -108,7 +106,7 @@ fn remove_file_if_exists(path: &Path) -> std::io::Result<()> {
 fn remove_dir_all_if_exists(path: &Path) -> std::io::Result<()> {
    match std::fs::remove_dir_all(path) {
        Ok(()) => Ok(()),
-        Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()),
+        Err(err) if err.kind() == ErrorKind::NotFound => Ok(()),
        Err(err) => Err(err),
    }
 }
@@ -120,7 +118,7 @@ fn rollback_committed_rename_std(
 ) -> std::io::Result<()> {
    if let Some(old_data_dir) = rollback_data_dir {
        let Some(dst_parent) = dst_file_path.parent() else {
-            return Err(std::io::Error::new(std::io::ErrorKind::InvalidInput, "missing object metadata parent"));
+            return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing object metadata parent"));
        };
        let backup_path = dst_parent.join(old_data_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP);
        std::fs::rename(backup_path, dst_file_path)?;
@@ -590,7 +588,7 @@ impl DurabilityMode {
            "strict" => Some(Self::Strict),
            "relaxed" => Some(Self::Relaxed),
            "none" => Some(Self::None),
-            _ => Option::None,
+            _ => None,
        }
    }

@@ -798,7 +796,7 @@ pub(crate) mod bucket_durability {
 /// configured durability mode: their contents commit into user buckets and
 /// are exactly the writes the relaxed tiers exist for.
 fn is_scratch_volume(volume: &str) -> bool {
-    for scratch in [super::RUSTFS_META_TMP_BUCKET, super::RUSTFS_META_MULTIPART_BUCKET] {
+    for scratch in [RUSTFS_META_TMP_BUCKET, super::RUSTFS_META_MULTIPART_BUCKET] {
        if volume == scratch || volume.strip_prefix(scratch).is_some_and(|rest| rest.starts_with('/')) {
            return true;
        }
@@ -815,7 +813,7 @@ fn is_system_critical_volume(volume: &str) -> bool {
    if is_scratch_volume(volume) {
        return false;
    }
-    for meta in [super::RUSTFS_META_BUCKET, super::MIGRATING_META_BUCKET] {
+    for meta in [RUSTFS_META_BUCKET, super::MIGRATING_META_BUCKET] {
        if volume == meta || volume.strip_prefix(meta).is_some_and(|rest| rest.starts_with('/')) {
            return true;
        }
@@ -934,12 +932,12 @@ struct AlignedBuf {
 impl AlignedBuf {
    fn new(len: usize, align: usize) -> std::io::Result<Self> {
        debug_assert!(len > 0, "AlignedBuf must not be zero-sized");
-        let layout = std::alloc::Layout::from_size_align(len, align)
-            .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidInput, e))?;
+        let layout =
+            std::alloc::Layout::from_size_align(len, align).map_err(|e| std::io::Error::new(ErrorKind::InvalidInput, e))?;
        // SAFETY: `layout` has non-zero size (callers guarantee len > 0) and a
        // valid power-of-two alignment enforced by Layout::from_size_align.
        let ptr = unsafe { std::alloc::alloc_zeroed(layout) };
-        let ptr = std::ptr::NonNull::new(ptr).ok_or(std::io::ErrorKind::OutOfMemory)?;
+        let ptr = std::ptr::NonNull::new(ptr).ok_or(ErrorKind::OutOfMemory)?;
        Ok(Self { ptr, len, layout })
    }

@@ -993,10 +991,9 @@ fn pread_direct_aligned(file_path: &Path, offset: u64, length: usize, state: &Di
    let align_u64 = align as u64;

    let aligned_offset = offset - (offset % align_u64);
-    let logical_start =
-        usize::try_from(offset - aligned_offset).map_err(|_| std::io::Error::from(std::io::ErrorKind::InvalidInput))?;
-    let logical_end = logical_start.checked_add(length).ok_or(std::io::ErrorKind::InvalidInput)?;
-    let aligned_len = logical_end.checked_add(align - 1).ok_or(std::io::ErrorKind::InvalidInput)? / align * align;
+    let logical_start = usize::try_from(offset - aligned_offset).map_err(|_| std::io::Error::from(ErrorKind::InvalidInput))?;
+    let logical_end = logical_start.checked_add(length).ok_or(ErrorKind::InvalidInput)?;
+    let aligned_len = logical_end.checked_add(align - 1).ok_or(ErrorKind::InvalidInput)? / align * align;

    let mut buf = AlignedBuf::new(aligned_len, align)?;

@@ -1011,7 +1008,7 @@ fn pread_direct_aligned(file_path: &Path, offset: u64, length: usize, state: &Di
        filled += n;
    }
    if filled < logical_end {
-        return Err(std::io::Error::new(std::io::ErrorKind::UnexpectedEof, "short O_DIRECT read"));
+        return Err(std::io::Error::new(ErrorKind::UnexpectedEof, "short O_DIRECT read"));
    }

    Ok(Bytes::copy_from_slice(&buf.as_slice()[logical_start..logical_end]))
@@ -1093,10 +1090,7 @@ fn pwrite_all(file: &std::fs::File, mut buf: &[u8], mut offset: u64) -> std::io:
    while !buf.is_empty() {
        let n = file.write_at(buf, offset)?;
        if n == 0 {
-            return Err(std::io::Error::new(
-                std::io::ErrorKind::WriteZero,
-                "O_DIRECT positioned write wrote 0 bytes",
-            ));
+            return Err(std::io::Error::new(ErrorKind::WriteZero, "O_DIRECT positioned write wrote 0 bytes"));
        }
        buf = &buf[n..];
        offset += n as u64;
@@ -1478,79 +1472,6 @@ fn should_fail_before_old_metadata_backup(_dst_path: &str) -> bool {
    false
 }

-/// Commit-sequence points where the rename_data crash-consistency harness
-/// (rustfs/backlog#935, test plan in rustfs/backlog#896) can simulate an
-/// abrupt power loss.
-///
-/// Unlike [`should_fail_before_old_metadata_backup`], which exercises the
-/// graceful in-process rollback (delete the staged data dir, return an
-/// error), a crash point models a hard power loss: the commit sequence stops
-/// dead at the armed step with **no** cleanup, leaving the on-disk state
-/// exactly as the preceding steps left it. The harness then reopens the disk
-/// and asserts the raw state is coherent — the object reads back as either the
-/// old version or the new version, never a mixed or corrupt one — without any
-/// rollback code having run.
-///
-/// The variants are constructed at the real commit-path call sites in every
-/// build, but the arming static and [`should_crash_rename_data_at`] are
-/// `#[cfg(test)]`; in production the guard is a const-`false` no-op, so the
-/// injection points compile away to nothing.
-// The `After<step>` naming is deliberate: every crash point names the
-// commit-sequence step it fires immediately after, so the shared prefix is the
-// point, not noise.
-#[allow(clippy::enum_variant_names)]
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
-pub(crate) enum RenameDataCrashPoint {
-    /// After the data dir has been renamed into its destination but before the
-    /// old-metadata rollback backup is written. xl.meta has not been committed
-    /// yet, so a crash here must leave the object readable as the old version.
-    AfterDataRename,
-    /// After the rollback backup is persisted, immediately before the xl.meta
-    /// commit rename that makes the new version visible. Still pre-commit, so a
-    /// crash here must also leave the object readable as the old version.
-    AfterBackupBeforeMetaCommit,
-    /// After the xl.meta commit rename has made the new version visible, in the
-    /// same window the graceful [`should_fail_after_metadata_commit`] failpoint
-    /// covers — but as a hard power loss with **no** rollback. The commit rename
-    /// already landed on disk, so a crash here must leave the object readable as
-    /// the *new* version. This is the post-commit counterpart to the two
-    /// pre-commit points above, closing the "old or new, never mixed" invariant
-    /// (rustfs/backlog#878) from the new-version side.
-    AfterMetaCommit,
-}
-
-#[cfg(test)]
-static RENAME_DATA_CRASH_POINT: std::sync::Mutex<Option<(RenameDataCrashPoint, String)>> = std::sync::Mutex::new(None);
-
-/// Arm a one-shot crash injection: the next `rename_data` committing into
-/// `dst_path` stops at `point`. Consumed on the first match so it never leaks
-/// into an unrelated commit.
-#[cfg(test)]
-fn arm_rename_data_crash(point: RenameDataCrashPoint, dst_path: &str) {
-    *RENAME_DATA_CRASH_POINT
-        .lock()
-        .expect("test crash point lock should not be poisoned") = Some((point, dst_path.to_string()));
-}
-
-#[cfg(test)]
-fn should_crash_rename_data_at(point: RenameDataCrashPoint, dst_path: &str) -> bool {
-    let mut armed = RENAME_DATA_CRASH_POINT
-        .lock()
-        .expect("test crash point lock should not be poisoned");
-    if armed.as_ref().is_some_and(|(p, path)| *p == point && path == dst_path) {
-        armed.take();
-        true
-    } else {
-        false
-    }
-}
-
-#[cfg(not(test))]
-#[inline(always)]
-fn should_crash_rename_data_at(_point: RenameDataCrashPoint, _dst_path: &str) -> bool {
-    false
-}
-
 #[cfg(not(test))]
 fn should_fail_after_metadata_commit(_dst_path: &str) -> bool {
    false
@@ -2200,7 +2121,8 @@ impl LocalIoBackend for StdBackend {
                let access_check_start = metrics_enabled.then(StdInstant::now);
                let volume_dir = local_disk_bucket_path(&root, &volume_owned)?;
                if !skip_access_checks(&volume_owned) {
-                    access_std(&volume_dir).map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
+                    crate::disk::fs::access_std(&volume_dir)
+                        .map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
                }
                let access_check_duration = access_check_start.map_or(StdDuration::ZERO, |started_at| started_at.elapsed());

@@ -3020,7 +2942,8 @@ fn is_io_uring_unsupported(err: &std::io::Error) -> bool {
 fn resolve_uring_object_path(root: &Path, volume: &str, path: &str) -> Result<PathBuf> {
    let volume_dir = local_disk_bucket_path(root, volume)?;
    if !skip_access_checks(volume) {
-        access_std(&volume_dir).map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
+        crate::disk::fs::access_std(&volume_dir)
+            .map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
    }
    let file_path = local_disk_object_path(root, volume, path)?;
    check_path_length(file_path.to_string_lossy().as_ref())?;
@@ -3869,10 +3792,9 @@ impl LocalDisk {
                        // rebuilding (see rustfs-heal); surface it so scanner
                        // coordination, lock selection and admin/metrics see
                        // the rebuild. Refreshed with this cache (~1s).
-                        let healing =
-                            tokio::fs::try_exists(root.join(super::RUSTFS_META_BUCKET).join(super::HEALING_MARKER_PATH))
-                                .await
-                                .unwrap_or(false);
+                        let healing = tokio::fs::try_exists(root.join(RUSTFS_META_BUCKET).join(super::HEALING_MARKER_PATH))
+                            .await
+                            .unwrap_or(false);
                        let disk_info = DiskInfo {
                            total: info.total,
                            free: info.free,
@@ -4755,9 +4677,7 @@ impl LocalDisk {
                        .await);
                    }
                    let rollback_data_path = rollback_path.join(dir.to_string());
-                    if let Err(err) = rename_all(&dir_path, &rollback_data_path, &rollback_path).await
-                        && !(err == DiskError::FileNotFound || err == DiskError::VolumeNotFound)
-                    {
+                    if let Err(err) = rename_all_ignore_missing_source(&dir_path, &rollback_data_path, &rollback_path).await {
                        return Err(restore_delete_rollback_after_error(
                            object_dir,
                            &xlpath,
@@ -4854,8 +4774,8 @@ impl LocalDisk {
        let file_path = self.get_object_path(volume, path)?;
        check_path_length(file_path.to_string_lossy().as_ref())?;

-        let tmp_volume_dir = self.get_bucket_path(super::RUSTFS_META_TMP_BUCKET)?;
-        let tmp_file_path = self.get_object_path(super::RUSTFS_META_TMP_BUCKET, Uuid::new_v4().to_string().as_str())?;
+        let tmp_volume_dir = self.get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
+        let tmp_file_path = self.get_object_path(RUSTFS_META_TMP_BUCKET, Uuid::new_v4().to_string().as_str())?;

        let durability = effective_durability(volume);

@@ -4872,6 +4792,16 @@ impl LocalDisk {
        self.write_all_internal(&tmp_file_path, InternalBuf::Ref(buf), tmp_sync, &tmp_volume_dir)
            .await?;

+        // Crash-consistency injection: hard power loss after the replacement
+        // xl.meta is staged in the tmp bucket but before the atomic rename that
+        // publishes it. The destination xl.meta is untouched, so a crash here
+        // must leave the object's metadata byte-for-byte the old version
+        // (rustfs/backlog#864); the staged tmp file is a harmless orphan swept by
+        // tmp-bucket GC. Compiles to a no-op outside `#[cfg(test)]`.
+        if crash_inject::should_crash_at(CrashPoint::MetaWriteAfterTmpBeforeRename, path) {
+            return Err(DiskError::Unexpected);
+        }
+
        rename_all(tmp_file_path, &file_path, volume_dir).await?;

        if sync
@@ -5705,7 +5635,7 @@ async fn read_file_metadata(p: impl AsRef<Path>) -> Result<Metadata> {
 fn skip_access_checks(p: impl AsRef<str>) -> bool {
    let vols = [
        RUSTFS_META_TMP_DELETED_BUCKET,
-        super::RUSTFS_META_TMP_BUCKET,
+        RUSTFS_META_TMP_BUCKET,
        super::RUSTFS_META_MULTIPART_BUCKET,
        RUSTFS_META_BUCKET,
    ];
@@ -6016,12 +5946,11 @@ impl DiskAPI for LocalDisk {
        );
        for (i, part) in fi.parts.iter().enumerate() {
            let checksum_info = erasure.get_checksum_info(part.number);
-            let checksum_algo =
-                if fi.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
-                    rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
-                } else {
-                    checksum_info.algorithm
-                };
+            let checksum_algo = if fi.uses_legacy_checksum && checksum_info.algorithm == HashAlgorithm::HighwayHash256S {
+                HashAlgorithm::HighwayHash256SLegacy
+            } else {
+                checksum_info.algorithm
+            };
            let part_path = self.get_object_path(
                volume,
                path_join_buf(&[
@@ -6855,7 +6784,7 @@ impl DiskAPI for LocalDisk {
            // is in place but before xl.meta commits. No cleanup — the harness
            // reopens the disk and asserts the object still reads as the old
            // version (the staged data dir is a harmless orphan for GC).
-            if should_crash_rename_data_at(RenameDataCrashPoint::AfterDataRename, dst_path) {
+            if crash_inject::should_crash_at(CrashPoint::RenameAfterDataRename, dst_path) {
                return Err(DiskError::Unexpected);
            }

@@ -6913,7 +6842,7 @@ impl DiskAPI for LocalDisk {
            // backup is durable but before the xl.meta commit rename. No
            // cleanup — the harness asserts the object still reads as the old
            // version, since the destination xl.meta is untouched here.
-            if should_crash_rename_data_at(RenameDataCrashPoint::AfterBackupBeforeMetaCommit, dst_path) {
+            if crash_inject::should_crash_at(CrashPoint::RenameAfterBackupBeforeMetaCommit, dst_path) {
                return Err(DiskError::Unexpected);
            }

@@ -6946,7 +6875,7 @@ impl DiskAPI for LocalDisk {
            // graceful failpoint above, no rollback runs — the commit rename is
            // already on disk, so the harness asserts the object reads back as
            // the new version.
-            if should_crash_rename_data_at(RenameDataCrashPoint::AfterMetaCommit, dst_path) {
+            if crash_inject::should_crash_at(CrashPoint::RenameAfterMetaCommit, dst_path) {
                return Err(DiskError::Unexpected);
            }

@@ -7049,7 +6978,7 @@ impl DiskAPI for LocalDisk {
                // Read existing xl.meta
                let has_dst_buf = match std::fs::read(&dst) {
                    Ok(buf) => Some(Bytes::from(buf)),
-                    Err(e) if e.kind() == std::io::ErrorKind::NotFound => None,
+                    Err(e) if e.kind() == ErrorKind::NotFound => None,
                    Err(e) => return Err(to_file_error(e)),
                };

@@ -7141,8 +7070,8 @@ impl DiskAPI for LocalDisk {

                match std::fs::rename(&src, &dst) {
                    Ok(()) => Ok(()),
-                    Err(err) if err.kind() == std::io::ErrorKind::NotFound && !src.exists() => Ok(()),
-                    Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
+                    Err(err) if err.kind() == ErrorKind::NotFound && !src.exists() => Ok(()),
+                    Err(err) if err.kind() == ErrorKind::NotFound => {
                        if let Some(parent) = dst.parent() {
                            std::fs::create_dir_all(parent)?;
                        }
@@ -7190,7 +7119,7 @@ impl DiskAPI for LocalDisk {
                    }
                }

-                Ok::<(Option<uuid::Uuid>, Option<Vec<u8>>, Option<OldCurrentSize>), std::io::Error>((
+                Ok::<(Option<Uuid>, Option<Vec<u8>>, Option<OldCurrentSize>), std::io::Error>((
                    rollback_data_dir,
                    version_signature,
                    old_current_size,
@@ -7681,10 +7610,7 @@ impl DiskAPI for LocalDisk {
                    .await);
                }
                let rollback_data_path = rollback_path.join(uuid.to_string());
-                if let Err(err) = rename_all(&old_path, &rollback_data_path, &rollback_path).await
-                    && err != DiskError::FileNotFound
-                    && err != DiskError::VolumeNotFound
-                {
+                if let Err(err) = rename_all_ignore_missing_source(&old_path, &rollback_data_path, &rollback_path).await {
                    return Err(restore_delete_rollback_after_error(
                        file_path.as_path(),
                        &xl_path,
@@ -7962,10 +7888,56 @@ async fn get_disk_info(drive_path: PathBuf) -> Result<(rustfs_utils::os::DiskInf
 mod test {
    use super::*;
    use rustfs_filemeta::ErasureInfo;
-    use std::io;
+    use std::io::{self, Write};
    use std::pin::Pin;
+    use std::sync::{Arc, Mutex};
    use std::task::{Context, Poll};
    use tokio::io::{AsyncReadExt, AsyncWrite, AsyncWriteExt, ReadBuf};
+    use tracing_subscriber::fmt::MakeWriter;
+
+    #[derive(Clone, Default)]
+    struct CapturedLogs {
+        buffer: Arc<Mutex<Vec<u8>>>,
+    }
+
+    struct CapturedLogWriter {
+        buffer: Arc<Mutex<Vec<u8>>>,
+    }
+
+    impl CapturedLogs {
+        fn contents(&self) -> String {
+            let buffer = self
+                .buffer
+                .lock()
+                .expect("captured logs mutex should not be poisoned")
+                .clone();
+            String::from_utf8(buffer).expect("captured logs should be valid UTF-8")
+        }
+    }
+
+    impl Write for CapturedLogWriter {
+        fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
+            self.buffer
+                .lock()
+                .expect("captured logs mutex should not be poisoned")
+                .extend_from_slice(buf);
+            Ok(buf.len())
+        }
+
+        fn flush(&mut self) -> io::Result<()> {
+            Ok(())
+        }
+    }
+
+    impl<'a> MakeWriter<'a> for CapturedLogs {
+        type Writer = CapturedLogWriter;
+
+        fn make_writer(&'a self) -> Self::Writer {
+            CapturedLogWriter {
+                buffer: Arc::clone(&self.buffer),
+            }
+        }
+    }

    fn test_file_info(name: &str, version_id: Uuid, data_dir: Option<Uuid>, data: Option<Bytes>) -> FileInfo {
        let size = data
@@ -8131,7 +8103,7 @@ mod test {

    /// Crash-consistency harness for the rename_data commit sequence
    /// (rustfs/backlog#935 HP-14, test plan rustfs/backlog#896; hard rule from
-    /// rustfs/backlog#878: "partial commit 后对象只能是旧版本或新版本,不能混合").
+    /// rustfs/backlog#878: "After partial commit, the object can only be an old or new version; it cannot be mixed").
    ///
    /// For every pre-commit crash point × durability tier, it seeds a committed
    /// object, stages a replacement, injects a hard power loss (no in-process
@@ -8143,13 +8115,14 @@ mod test {
    /// old-or-new invariant, only with a wider (documented) power-loss window.
    mod crash_consistency {
        use super::*;
+        use crate::crash_inject::{self, CrashPoint};
        use tempfile::tempdir;

        const VERSION_ID: &str = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa";
        const OLD_DATA_DIR: &str = "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb";
        const NEW_DATA_DIR: &str = "cccccccc-cccc-cccc-cccc-cccccccccccc";

-        async fn run_scenario(mode: DurabilityMode, crash: Option<RenameDataCrashPoint>, with_old_version: bool) {
+        async fn run_scenario(mode: DurabilityMode, crash: Option<CrashPoint>, with_old_version: bool) {
            // Serializes with every other durability-sensitive test and pins the
            // resolved tier for the whole scenario (held until dropped).
            let _mode = durability_mode_override::set(mode);
@@ -8200,7 +8173,7 @@ mod test {
                .expect("new tmp data should be written");

            if let Some(point) = crash {
-                arm_rename_data_crash(point, object);
+                crash_inject::arm(point, object);
            }
            let new_fi = test_file_info(object, version_id, Some(new_data_dir), None);
            let result = disk
@@ -8215,7 +8188,7 @@ mod test {
                .await;

            match crash {
-                Some(RenameDataCrashPoint::AfterMetaCommit) => {
+                Some(CrashPoint::RenameAfterMetaCommit) => {
                    // Hard power loss right after the xl.meta commit rename: the
                    // commit landed and no rollback ran, so the object must read
                    // back as the new version — whether or not an old version
@@ -8283,9 +8256,9 @@ mod test {
            }
        }

-        const CRASH_POINTS: [RenameDataCrashPoint; 2] = [
-            RenameDataCrashPoint::AfterDataRename,
-            RenameDataCrashPoint::AfterBackupBeforeMetaCommit,
+        const CRASH_POINTS: [CrashPoint; 2] = [
+            CrashPoint::RenameAfterDataRename,
+            CrashPoint::RenameAfterBackupBeforeMetaCommit,
        ];

        #[tokio::test]
@@ -8331,29 +8304,145 @@ mod test {
        // the same invariant.
        #[tokio::test]
        async fn overwrite_post_commit_crash_keeps_new_version_strict() {
-            run_scenario(DurabilityMode::Strict, Some(RenameDataCrashPoint::AfterMetaCommit), true).await;
+            run_scenario(DurabilityMode::Strict, Some(CrashPoint::RenameAfterMetaCommit), true).await;
        }

        #[tokio::test]
        async fn overwrite_post_commit_crash_keeps_new_version_relaxed() {
-            run_scenario(DurabilityMode::Relaxed, Some(RenameDataCrashPoint::AfterMetaCommit), true).await;
+            run_scenario(DurabilityMode::Relaxed, Some(CrashPoint::RenameAfterMetaCommit), true).await;
        }

        #[tokio::test]
        async fn fresh_post_commit_crash_keeps_new_version_strict() {
-            run_scenario(DurabilityMode::Strict, Some(RenameDataCrashPoint::AfterMetaCommit), false).await;
+            run_scenario(DurabilityMode::Strict, Some(CrashPoint::RenameAfterMetaCommit), false).await;
        }

        #[tokio::test]
        async fn fresh_post_commit_crash_keeps_new_version_relaxed() {
-            run_scenario(DurabilityMode::Relaxed, Some(RenameDataCrashPoint::AfterMetaCommit), false).await;
+            run_scenario(DurabilityMode::Relaxed, Some(CrashPoint::RenameAfterMetaCommit), false).await;
+        }
+    }
+
+    /// Crash-consistency for the in-place xl.meta update path — the atomic
+    /// temp+rename inside [`LocalDisk::write_all_meta`] shared by `update_metadata`
+    /// and `write_metadata` (delete markers, tag/metadata rewrites, decommission).
+    ///
+    /// rustfs/backlog#864: a fault that interrupts an in-place metadata rewrite
+    /// must not mutate the committed object. [`CrashPoint::MetaWriteAfterTmpBeforeRename`]
+    /// models a hard power loss after the replacement xl.meta is staged in the tmp
+    /// bucket but before the publishing rename. Both durability tiers are held to
+    /// the same rule: the destination xl.meta survives byte-for-byte, the object
+    /// still reads as the old version, the staged tmp file is a reclaimable orphan
+    /// confined to the tmp bucket, and a later un-injected rewrite publishes
+    /// cleanly (retryable). This path previously had only parser-level unit tests.
+    mod meta_write_crash_consistency {
+        use super::*;
+        use crate::crash_inject::{self, CrashPoint};
+        use tempfile::tempdir;
+
+        async fn run(mode: DurabilityMode) {
+            // Serialize with every other durability-sensitive test and pin the
+            // resolved tier for the whole scenario.
+            let _mode = durability_mode_override::set(mode);
+
+            let dir = tempdir().expect("temp dir should be created");
+            let endpoint =
+                Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
+            let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
+
+            let bucket = "bucket";
+            let object = "meta-write-crash-object";
+            ensure_test_volume(&disk, bucket).await;
+            ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
+
+            // Seed a committed object (version 1) by writing its xl.meta directly:
+            // read_data=false reads never touch the data dir, so no shard staging
+            // is needed to exercise the metadata commit window.
+            let object_dir = dir.path().join(bucket).join(object);
+            fs::create_dir_all(&object_dir).await.expect("object dir should be created");
+            let meta_path = object_dir.join(STORAGE_FORMAT_FILE);
+            let v1 = Uuid::new_v4();
+            let old_meta = test_meta(test_file_info(object, v1, Some(Uuid::new_v4()), None));
+            fs::write(&meta_path, &old_meta)
+                .await
+                .expect("committed xl.meta should be written");
+
+            // Arm the crash, then attempt an in-place rewrite that adds version 2.
+            let meta_key = format!("{object}/{STORAGE_FORMAT_FILE}");
+            crash_inject::arm(CrashPoint::MetaWriteAfterTmpBeforeRename, &meta_key);
+            let v2 = Uuid::new_v4();
+            let result = disk
+                .write_metadata("", bucket, object, test_file_info(object, v2, Some(Uuid::new_v4()), None))
+                .await;
+            assert!(
+                matches!(result, Err(DiskError::Unexpected)),
+                "{mode:?}: the armed crash point must be the failure that surfaced, got {result:?}"
+            );
+
+            // Reopen the disk to model a process restart after the crash.
+            drop(disk);
+            let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should reopen");
+
+            // rustfs/backlog#864: the committed xl.meta is byte-for-byte the old
+            // version — the interrupted rewrite never published.
+            let after = fs::read(&meta_path).await.expect("xl.meta must survive the crash");
+            assert_eq!(&after, &old_meta, "{mode:?}: xl.meta must remain the old version byte-for-byte");
+
+            // The old version still reads; the un-published new version is absent.
+            disk.read_version("", bucket, object, &v1.to_string(), &ReadOptions::default())
+                .await
+                .expect("the old version must remain readable after the crash");
+            let new_read = disk
+                .read_version("", bucket, object, &v2.to_string(), &ReadOptions::default())
+                .await;
+            assert!(
+                matches!(new_read, Err(DiskError::FileVersionNotFound)),
+                "{mode:?}: the interrupted update must not publish the new version, got {new_read:?}"
+            );
+
+            // The crash leaves no staging debris in the object directory; the
+            // staged replacement is a reclaimable orphan under the tmp bucket.
+            let mut object_entries = fs::read_dir(&object_dir).await.expect("object dir should list");
+            let mut object_files = Vec::new();
+            while let Some(entry) = object_entries
+                .next_entry()
+                .await
+                .expect("object dir entry should be readable")
+            {
+                object_files.push(entry.file_name().to_string_lossy().to_string());
+            }
+            assert_eq!(
+                object_files,
+                vec![STORAGE_FORMAT_FILE.to_string()],
+                "{mode:?}: the object directory must hold only its committed xl.meta"
+            );
+
+            // A retried rewrite (un-injected) publishes cleanly: the path is safely
+            // retryable after the crash.
+            crash_inject::disarm(CrashPoint::MetaWriteAfterTmpBeforeRename, &meta_key);
+            disk.write_metadata("", bucket, object, test_file_info(object, v2, Some(Uuid::new_v4()), None))
+                .await
+                .expect("a retried metadata rewrite must succeed after the crash");
+            disk.read_version("", bucket, object, &v2.to_string(), &ReadOptions::default())
+                .await
+                .expect("the retried version must be readable");
+        }
+
+        #[tokio::test]
+        async fn meta_write_crash_before_rename_keeps_old_version_strict() {
+            run(DurabilityMode::Strict).await;
+        }
+
+        #[tokio::test]
+        async fn meta_write_crash_before_rename_keeps_old_version_relaxed() {
+            run(DurabilityMode::Relaxed).await;
        }
    }

    /// Backdate a path's mtime so zero-expiry cleanup tests classify it as
    /// stale deterministically, instead of sleeping and hoping the filesystem
    /// timestamp granularity (or a backward wall-clock step) cooperates.
-    fn backdate_mtime(path: &std::path::Path, age: Duration) {
+    fn backdate_mtime(path: &Path, age: Duration) {
        use std::fs::{File, FileTimes};
        let mtime = std::time::SystemTime::now() - age;
        File::open(path)
@@ -8762,17 +8851,13 @@ mod test {
    async fn blocking_scan_writer_keeps_flush_and_shutdown_pending() {
        let mut flush_writer = BlockingScanWriter { entered_tx: None };
        assert!(
-            tokio::time::timeout(Duration::from_millis(10), flush_writer.flush())
-                .await
-                .is_err(),
+            timeout(Duration::from_millis(10), flush_writer.flush()).await.is_err(),
            "blocking scan writer flush should stay pending"
        );

        let mut shutdown_writer = BlockingScanWriter { entered_tx: None };
        assert!(
-            tokio::time::timeout(Duration::from_millis(10), shutdown_writer.shutdown())
-                .await
-                .is_err(),
+            timeout(Duration::from_millis(10), shutdown_writer.shutdown()).await.is_err(),
            "blocking scan writer shutdown should stay pending"
        );
    }
@@ -8781,15 +8866,11 @@ mod test {
    /// consumer (quorum merge, a lagging peer drive).
    struct SlowWriter {
        delay: Duration,
-        sleep: Option<std::pin::Pin<Box<Sleep>>>,
+        sleep: Option<Pin<Box<Sleep>>>,
    }

    impl AsyncWrite for SlowWriter {
-        fn poll_write(
-            mut self: std::pin::Pin<&mut Self>,
-            cx: &mut std::task::Context<'_>,
-            buf: &[u8],
-        ) -> std::task::Poll<std::io::Result<usize>> {
+        fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8]) -> Poll<io::Result<usize>> {
            if self.sleep.is_none() {
                let delay = self.delay;
                self.sleep = Some(Box::pin(tokio::time::sleep(delay)));
@@ -8797,23 +8878,20 @@ mod test {

            let sleep = self.sleep.as_mut().expect("sleep was just installed");
            match sleep.as_mut().poll(cx) {
-                std::task::Poll::Ready(()) => {
+                Poll::Ready(()) => {
                    self.sleep = None;
-                    std::task::Poll::Ready(Ok(buf.len()))
+                    Poll::Ready(Ok(buf.len()))
                }
-                std::task::Poll::Pending => std::task::Poll::Pending,
+                Poll::Pending => Poll::Pending,
            }
        }

-        fn poll_flush(self: std::pin::Pin<&mut Self>, _cx: &mut std::task::Context<'_>) -> std::task::Poll<std::io::Result<()>> {
-            std::task::Poll::Ready(Ok(()))
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
        }

-        fn poll_shutdown(
-            self: std::pin::Pin<&mut Self>,
-            _cx: &mut std::task::Context<'_>,
-        ) -> std::task::Poll<std::io::Result<()>> {
-            std::task::Poll::Ready(Ok(()))
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+            Poll::Ready(Ok(()))
        }
    }

@@ -9509,14 +9587,14 @@ mod test {
        bucket_durability::set(super::super::RUSTFS_META_MULTIPART_BUCKET, Some(DurabilityMode::Relaxed));
        bucket_durability::set("", Some(DurabilityMode::Relaxed));

-        assert_eq!(bucket_durability::lookup(RUSTFS_META_BUCKET), Option::None);
-        assert_eq!(bucket_durability::lookup(RUSTFS_META_TMP_BUCKET), Option::None);
+        assert_eq!(bucket_durability::lookup(RUSTFS_META_BUCKET), None);
+        assert_eq!(bucket_durability::lookup(RUSTFS_META_TMP_BUCKET), None);
        assert_eq!(effective_durability(RUSTFS_META_BUCKET), DurabilityMode::Strict);

        // The legacy full-off tier is process-wide only: registering it per
        // bucket is dropped, not stored.
        bucket_durability::set("hp5b-legacy-refused", Some(DurabilityMode::LegacyOff));
-        assert_eq!(bucket_durability::lookup("hp5b-legacy-refused"), Option::None);
+        assert_eq!(bucket_durability::lookup("hp5b-legacy-refused"), None);
    }

    #[test]
@@ -10376,6 +10454,149 @@ mod test {
        );
    }

+    #[tokio::test(flavor = "current_thread")]
+    async fn test_delete_version_missing_inline_data_dir_does_not_warn() {
+        use tempfile::tempdir;
+
+        let dir = tempdir().expect("temp dir should be created");
+        let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
+        let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
+
+        let bucket = "bucket";
+        let object = "dir/inline-object";
+        let version_id = Uuid::parse_str("31313131-1111-2222-3333-444444444444").expect("version id should parse");
+        let data_dir = Uuid::parse_str("32323232-1111-2222-3333-444444444444").expect("data dir should parse");
+        let rollback_dir = Uuid::parse_str("33333333-1111-2222-3333-444444444444").expect("rollback dir should parse");
+
+        ensure_test_volume(&disk, bucket).await;
+
+        let object_dir = dir.path().join(bucket).join(object);
+        fs::create_dir_all(&object_dir).await.expect("object dir should be created");
+        let fi = test_file_info(object, version_id, Some(data_dir), Some(Bytes::from_static(b"inline")));
+        fs::write(object_dir.join(STORAGE_FORMAT_FILE), test_meta(fi.clone()))
+            .await
+            .expect("inline metadata should be written");
+        assert!(!object_dir.join(data_dir.to_string()).exists());
+
+        let logs = CapturedLogs::default();
+        let subscriber = tracing_subscriber::fmt()
+            .with_writer(logs.clone())
+            .with_ansi(false)
+            .without_time()
+            .finish();
+        let _guard = tracing::subscriber::set_default(subscriber);
+
+        disk.delete_version(
+            bucket,
+            object,
+            fi,
+            false,
+            DeleteOptions {
+                old_data_dir: Some(rollback_dir),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("missing inline data dir should not fail deletion");
+
+        assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists());
+        assert!(
+            object_dir
+                .join(rollback_dir.to_string())
+                .join(STORAGE_FORMAT_FILE_BACKUP)
+                .exists()
+        );
+        assert!(!logs.contents().contains("reliable_rename failed"));
+    }
+
+    #[tokio::test(flavor = "current_thread")]
+    async fn test_delete_versions_missing_inline_data_dir_does_not_warn() {
+        use tempfile::tempdir;
+
+        let dir = tempdir().expect("temp dir should be created");
+        let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
+        let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
+
+        let bucket = "bucket";
+        let object = "dir/inline-object-batch";
+        let version_id = Uuid::parse_str("34343434-1111-2222-3333-444444444444").expect("version id should parse");
+        let data_dir = Uuid::parse_str("35353535-1111-2222-3333-444444444444").expect("data dir should parse");
+        let rollback_dir = Uuid::parse_str("36363636-1111-2222-3333-444444444444").expect("rollback dir should parse");
+
+        ensure_test_volume(&disk, bucket).await;
+
+        let object_dir = dir.path().join(bucket).join(object);
+        fs::create_dir_all(&object_dir).await.expect("object dir should be created");
+        let fi = test_file_info(object, version_id, Some(data_dir), Some(Bytes::from_static(b"inline")));
+        fs::write(object_dir.join(STORAGE_FORMAT_FILE), test_meta(fi.clone()))
+            .await
+            .expect("inline metadata should be written");
+        assert!(!object_dir.join(data_dir.to_string()).exists());
+
+        let logs = CapturedLogs::default();
+        let subscriber = tracing_subscriber::fmt()
+            .with_writer(logs.clone())
+            .with_ansi(false)
+            .without_time()
+            .finish();
+        let _guard = tracing::subscriber::set_default(subscriber);
+
+        disk.delete_versions_internal(
+            bucket,
+            object,
+            &[fi],
+            &DeleteOptions {
+                old_data_dir: Some(rollback_dir),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("missing inline data dir should not fail batch deletion");
+
+        assert!(!object_dir.join(STORAGE_FORMAT_FILE).exists());
+        assert!(
+            object_dir
+                .join(rollback_dir.to_string())
+                .join(STORAGE_FORMAT_FILE_BACKUP)
+                .exists()
+        );
+        assert!(!logs.contents().contains("reliable_rename failed"));
+    }
+
+    #[tokio::test(flavor = "current_thread")]
+    async fn test_ignore_missing_source_helper_still_warns_on_real_error() {
+        use tempfile::tempdir;
+
+        let dir = tempdir().expect("temp dir should be created");
+        let src = dir.path().join("src");
+        let dst = dir.path().join("dst");
+        fs::create_dir_all(&src).await.expect("source dir should be created");
+        fs::write(src.join("part.1"), b"live-data")
+            .await
+            .expect("source data should be written");
+        fs::create_dir_all(&dst).await.expect("destination dir should be created");
+        fs::write(dst.join("sentinel"), b"conflict")
+            .await
+            .expect("destination conflict should be written");
+
+        let logs = CapturedLogs::default();
+        let subscriber = tracing_subscriber::fmt()
+            .with_writer(logs.clone())
+            .with_ansi(false)
+            .without_time()
+            .finish();
+        let _guard = tracing::subscriber::set_default(subscriber);
+
+        let err = rename_all_ignore_missing_source(&src, &dst, dir.path())
+            .await
+            .expect_err("a non-empty rollback destination must reject rename");
+
+        assert_ne!(err, DiskError::FileNotFound);
+        assert!(src.exists());
+        assert!(dst.join("sentinel").exists());
+        assert!(logs.contents().contains("reliable_rename failed"));
+    }
+
    #[tokio::test]
    async fn test_delete_version_rollback_restores_staged_data_dir() {
        use tempfile::tempdir;
@@ -10832,7 +11053,7 @@ mod test {

        let vols = [
            RUSTFS_META_TMP_DELETED_BUCKET,
-            super::super::RUSTFS_META_TMP_BUCKET,
+            RUSTFS_META_TMP_BUCKET,
            super::super::RUSTFS_META_MULTIPART_BUCKET,
            RUSTFS_META_BUCKET,
        ];
@@ -10871,7 +11092,7 @@ mod test {
        let mut reader = StallTimeoutReader::new(PendingTestReader, Duration::ZERO);
        let mut buf = [0; 1];

-        let result = tokio::time::timeout(Duration::from_millis(10), reader.read(&mut buf)).await;
+        let result = timeout(Duration::from_millis(10), reader.read(&mut buf)).await;

        assert!(result.is_err(), "zero timeout must leave stalled reads pending instead of failing");
    }
@@ -11001,10 +11222,10 @@ mod test {

    #[tokio::test(start_paused = true)]
    async fn cleanup_loop_interval_does_not_tick_immediately() {
-        let start_at = tokio::time::Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
+        let start_at = Instant::now() + DELETED_OBJECTS_CLEANUP_INTERVAL;
        let mut interval = interval_at(start_at, DELETED_OBJECTS_CLEANUP_INTERVAL);

-        assert!(tokio::time::timeout(Duration::from_secs(1), interval.tick()).await.is_err());
+        assert!(timeout(Duration::from_secs(1), interval.tick()).await.is_err());

        tokio::time::advance(DELETED_OBJECTS_CLEANUP_INTERVAL).await;
        interval.tick().await;
@@ -11052,26 +11273,16 @@ mod test {
        struct BrokenPipeWriter;

        impl AsyncWrite for BrokenPipeWriter {
-            fn poll_write(
-                self: std::pin::Pin<&mut Self>,
-                _cx: &mut std::task::Context<'_>,
-                _buf: &[u8],
-            ) -> std::task::Poll<std::io::Result<usize>> {
-                std::task::Poll::Ready(Err(std::io::Error::new(std::io::ErrorKind::BrokenPipe, "closed")))
+            fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<io::Result<usize>> {
+                Poll::Ready(Err(io::Error::new(ErrorKind::BrokenPipe, "closed")))
            }

-            fn poll_flush(
-                self: std::pin::Pin<&mut Self>,
-                _cx: &mut std::task::Context<'_>,
-            ) -> std::task::Poll<std::io::Result<()>> {
-                std::task::Poll::Ready(Ok(()))
+            fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+                Poll::Ready(Ok(()))
            }

-            fn poll_shutdown(
-                self: std::pin::Pin<&mut Self>,
-                _cx: &mut std::task::Context<'_>,
-            ) -> std::task::Poll<std::io::Result<()>> {
-                std::task::Poll::Ready(Ok(()))
+            fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
+                Poll::Ready(Ok(()))
            }
        }

@@ -12203,7 +12414,7 @@ mod test {
        fs::set_permissions(&meta_path, Permissions::from_mode(0o000))
            .await
            .expect("operation should succeed");
-        if fs::File::open(&meta_path).await.is_ok() {
+        if File::open(&meta_path).await.is_ok() {
            fs::set_permissions(&meta_path, original_permissions)
                .await
                .expect("operation should succeed");
@@ -12738,7 +12949,7 @@ mod test {
        // It only checks length and platform-specific special characters
        // System volume names are valid according to the current implementation
        assert!(LocalDisk::is_valid_volname(RUSTFS_META_BUCKET));
-        assert!(LocalDisk::is_valid_volname(super::super::RUSTFS_META_TMP_BUCKET));
+        assert!(LocalDisk::is_valid_volname(RUSTFS_META_TMP_BUCKET));

        // Testing platform-specific behavior for special characters
        #[cfg(windows)]
@@ -13082,24 +13293,20 @@ mod test {

        #[cfg(unix)]
        {
-            for (kind, expected) in [("minor", 1), ("major", 2)] {
+            for (kind, expected) in [("minor", 1), ("major", 2)].iter().copied() {
+                let labels = [("path", "local_test"), ("stage", "mmap_map"), ("kind", kind)];
                assert_eq!(
-                    recorder.counter_value(
-                        METRIC_GET_OBJECT_MMAP_PAGE_FAULTS_TOTAL,
-                        &[("path", "local_test"), ("stage", "mmap_map"), ("kind", kind)]
-                    ),
+                    recorder.counter_value(METRIC_GET_OBJECT_MMAP_PAGE_FAULTS_TOTAL, &labels),
                    expected,
-                    "zero deltas must not emit and positive {kind} deltas must accumulate exactly"
+                    "zero deltas must not emit and positive mmap page fault deltas must accumulate exactly"
                );
            }
-            for (kind, expected) in [("minor", 3), ("major", 4)] {
+            for (kind, expected) in [("minor", 3), ("major", 4)].iter().copied() {
+                let labels = [("path", "local_test"), ("stage", "direct_read_copy"), ("kind", kind)];
                assert_eq!(
-                    recorder.counter_value(
-                        METRIC_GET_OBJECT_DIRECT_READ_PAGE_FAULTS_TOTAL,
-                        &[("path", "local_test"), ("stage", "direct_read_copy"), ("kind", kind)]
-                    ),
+                    recorder.counter_value(METRIC_GET_OBJECT_DIRECT_READ_PAGE_FAULTS_TOTAL, &labels),
                    expected,
-                    "zero deltas must not emit and positive {kind} deltas must accumulate exactly"
+                    "zero deltas must not emit and positive direct-read page fault deltas must accumulate exactly"
                );
            }
        }
@@ -13314,15 +13521,15 @@ mod test {

    #[test]
    fn test_is_bitrot_size_mismatch_error_only_matches_target_message() {
-        assert!(is_bitrot_size_mismatch_error(&std::io::Error::other("bitrot shard file size mismatch")));
-        assert!(!is_bitrot_size_mismatch_error(&std::io::Error::other("bitrot hash mismatch")));
+        assert!(is_bitrot_size_mismatch_error(&io::Error::other("bitrot shard file size mismatch")));
+        assert!(!is_bitrot_size_mismatch_error(&io::Error::other("bitrot hash mismatch")));
    }

    #[test]
    fn test_is_bitrot_verification_error_matches_hash_and_size_mismatch() {
-        assert!(is_bitrot_verification_error(&std::io::Error::other("bitrot shard file size mismatch")));
-        assert!(is_bitrot_verification_error(&std::io::Error::other("bitrot hash mismatch")));
-        assert!(!is_bitrot_verification_error(&std::io::Error::other("unrelated io failure")));
+        assert!(is_bitrot_verification_error(&io::Error::other("bitrot shard file size mismatch")));
+        assert!(is_bitrot_verification_error(&io::Error::other("bitrot hash mismatch")));
+        assert!(!is_bitrot_verification_error(&io::Error::other("unrelated io failure")));
    }

    #[tokio::test]
@@ -13361,7 +13568,7 @@ mod test {
            ..Default::default()
        }];

-        let mut writer = BitrotWriter::new(std::io::Cursor::new(Vec::new()), file_info.erasure.shard_size(), checksum_algo);
+        let mut writer = BitrotWriter::new(io::Cursor::new(Vec::new()), file_info.erasure.shard_size(), checksum_algo);
        writer
            .write(&payload)
            .await
@@ -13428,7 +13635,7 @@ mod test {
            } else {
                HashAlgorithm::HighwayHash256S
            };
-            let mut writer = BitrotWriter::new(std::io::Cursor::new(Vec::new()), codec_erasure.shard_size(), checksum_algo);
+            let mut writer = BitrotWriter::new(io::Cursor::new(Vec::new()), codec_erasure.shard_size(), checksum_algo);
            writer.write(&payload[..8]).await.expect("first shard block should encode");
            writer.write(&payload[8..]).await.expect("final shard block should encode");
            writer.shutdown().await.expect("bitrot writer should flush test payload");
@@ -14285,7 +14492,7 @@ mod test {
        // SAFETY: `map.as_ptr()` is page-aligned and `len` bytes long; `vec` has
        // one byte per page of that range, which is what mincore writes.
        let rc = unsafe { libc::mincore(map.as_ptr() as *mut libc::c_void, len, vec.as_mut_ptr()) };
-        assert_eq!(rc, 0, "mincore failed: {}", std::io::Error::last_os_error());
+        assert_eq!(rc, 0, "mincore failed: {}", io::Error::last_os_error());
        vec.iter().filter(|b| *b & 1 == 1).count()
    }

@@ -26,7 +26,7 @@ use std::{
 };
 use tokio::fs;
 use tokio::sync::{OwnedSemaphorePermit, Semaphore, SemaphorePermit};
-use tracing::{debug, warn};
+use tracing::warn;

 /// Check path length according to OS limits.
 pub fn check_path_length(path_name: &str) -> Result<()> {
@@ -527,7 +527,7 @@ async fn reliable_rename_inner(
    src_file_path: impl AsRef<Path>,
    dst_file_path: impl AsRef<Path>,
    base_dir: impl AsRef<Path>,
-    warn_on_failure: bool,
+    warn_on_missing_source: bool,
 ) -> io::Result<()> {
    if let Some(parent) = dst_file_path.as_ref().parent()
        && !file_exists(parent)
@@ -542,32 +542,8 @@ async fn reliable_rename_inner(
                i += 1;
                continue;
            }
-            if warn_on_failure {
-                // NotFound is an expected outcome for speculative renames (staging
-                // a dataDir that inline objects never had, trashing an already-
-                // removed tmp path, restoring an absent rollback backup): those
-                // callers treat FileNotFound as acceptable, and on per-request
-                // delete paths a WARN becomes a log storm (one line per disk per
-                // DELETE). Keep it at debug so quorum-masked cases (e.g. a tmp
-                // file lost before a commit rename) stay diagnosable; genuine
-                // failures keep the WARN. The error is returned either way.
-                if e.kind() == io::ErrorKind::NotFound {
-                    debug!(
-                        "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
-                        src_file_path.as_ref(),
-                        dst_file_path.as_ref(),
-                        base_dir.as_ref(),
-                        e
-                    );
-                } else {
-                    warn!(
-                        "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
-                        src_file_path.as_ref(),
-                        dst_file_path.as_ref(),
-                        base_dir.as_ref(),
-                        e
-                    );
-                }
+            if warn_on_missing_source || e.kind() != io::ErrorKind::NotFound {
+                warn_reliable_rename_failure(src_file_path.as_ref(), dst_file_path.as_ref(), base_dir.as_ref(), &e);
            }
            return Err(e);
        }
@@ -578,6 +554,13 @@ async fn reliable_rename_inner(
    Ok(())
 }

+fn warn_reliable_rename_failure(src_file_path: &Path, dst_file_path: &Path, base_dir: &Path, err: &io::Error) {
+    warn!(
+        "reliable_rename failed. src_file_path: {:?}, dst_file_path: {:?}, base_dir: {:?}, err: {:?}",
+        src_file_path, dst_file_path, base_dir, err
+    );
+}
+
 /// Whether a failed `rename` in [`reliable_rename_inner`] should be retried.
 ///
 /// Only the first failure is retried, and `NotFound` is never retried: the
@@ -755,24 +738,22 @@ mod tests {
    }

    #[tokio::test]
-    async fn rename_all_ignore_missing_source_returns_ok() {
+    async fn rename_all_ignore_missing_source_returns_ok_without_warn() {
        let temp_dir = tempdir().expect("create temp dir");
        let src = temp_dir.path().join("missing");
        let dst = temp_dir.path().join("dst");

+        let (logs, _guard) = warn_capture();
        rename_all_ignore_missing_source(&src, &dst, temp_dir.path())
            .await
            .expect("missing cleanup source must be ignored");

        assert!(!dst.exists());
+        assert!(!logs.contents().contains("reliable_rename failed"));
    }

    #[tokio::test]
-    async fn rename_all_missing_source_skips_rename_warn() {
-        // Callers like delete_version's rollback staging accept FileNotFound
-        // (inline objects have no dataDir), so a missing source must not emit
-        // the per-disk "reliable_rename failed" WARN — under delete storms it
-        // was pure log noise. The error itself is still returned.
+    async fn rename_all_missing_source_still_warns() {
        let temp_dir = tempdir().expect("create temp dir");
        let src = temp_dir.path().join("missing");
        let dst = temp_dir.path().join("dst");
@@ -785,8 +766,8 @@ mod tests {
        assert!(matches!(err, DiskError::FileNotFound));
        let captured = logs.contents();
        assert!(
-            !captured.contains("reliable_rename failed"),
-            "NotFound must not emit the reliable_rename WARN, got: {captured}"
+            captured.contains("reliable_rename failed"),
+            "ordinary missing-source failures must keep the WARN, got: {captured}"
        );
    }

@@ -40,6 +40,10 @@ use tracing::{error, warn};

 type ShardReadFuture<'a> = Pin<Box<dyn Future<Output = (usize, ShardReadCost, Result<Vec<u8>, Error>, bool)> + Send + 'a>>;

+/// One stripe's worth of shard buffers plus the per-shard read errors, as
+/// returned by `ParallelReader::read` / `read_stripe_timed`.
+type StripeReadOutput = (Vec<Option<Vec<u8>>>, Vec<Option<Error>>);
+
 const ENV_RUSTFS_SHARD_LOCALITY_SCHEDULING: &str = "RUSTFS_SHARD_LOCALITY_SCHEDULING";
 const ENV_RUSTFS_GET_SHARD_LOCALITY_PREFERENCE_ENABLE: &str = "RUSTFS_GET_SHARD_LOCALITY_PREFERENCE_ENABLE";
 const SHARD_LOCALITY_SCHEDULING_OFF: &str = "off";
@@ -1091,6 +1095,7 @@ where
        let mut completed = 0usize;
        let mut failed = 0usize;
        let mut first_shard_recorded = false;
+        let mut hedged = false;
        // Scope the reader borrow (held by `reader_iter`/`sets`) so it is released
        // before the retirement pass mutates `self.readers` below.
        {
@@ -1115,7 +1120,53 @@ where
                ));
            }

-            while let Some((i, _read_cost, result, _should_retire)) = sets.next().await {
+            // Hedge the straggler. Once every healthy shard has had `hedge_delay`
+            // to return, stop waiting out the full per-shard `read_timeout` for a
+            // slow shard: cancel the still-pending reads and let the stripe-aligned
+            // parity substitution below cover their slots — but only when enough
+            // unengaged parity remains to reconstruct them, otherwise keep waiting.
+            // A slow shard here otherwise stalls the whole stripe (and the GET's
+            // first byte) until the 10s read timeout fires: the large-object GET
+            // first-byte tail seen in backlog#1156.
+            let hedge_delay = shard_read_hedge_delay(read_timeout);
+            let hedge_sleep = hedge_delay.map(tokio::time::sleep);
+            tokio::pin!(hedge_sleep);
+            loop {
+                let item = match hedge_sleep.as_mut().as_pin_mut() {
+                    Some(sleep) if !hedged => {
+                        tokio::select! {
+                            biased;
+                            item = sets.next() => item,
+                            _ = sleep => {
+                                hedged = true;
+                                // Stop waiting for the straggler(s) only once the stripe
+                                // can be finished without them: either every data shard
+                                // is already in (no reconstruction needed), or there are
+                                // at least `data_shards + 1` shards — decode quorum plus
+                                // a reconstruction-verification source, so a missing data
+                                // shard is reconstructed *and* verified, never settled at
+                                // exactly `data_shards` (which would silently skip
+                                // verification, erasure.rs). In the read-all default every
+                                // slot is engaged, so `sets` already holds data + parity
+                                // and this stops one slow shard from stalling the stripe
+                                // (and the GET's first byte) for the full read timeout
+                                // (backlog#1156). Otherwise keep waiting — the straggler
+                                // may be the verification source, or the only route to
+                                // quorum on a degraded stripe.
+                                let succeeded = shards.iter().filter(|shard| shard.is_some()).count();
+                                let data_missing = shards.iter().take(data_shards).any(|shard| shard.is_none());
+                                if !data_missing || succeeded > data_shards {
+                                    break;
+                                }
+                                continue;
+                            }
+                        }
+                    }
+                    _ => sets.next().await,
+                };
+                let Some((i, _read_cost, result, _should_retire)) = item else {
+                    break;
+                };
                completed += 1;
                if !first_shard_recorded {
                    if let Some(path) = metrics_path {
@@ -1134,14 +1185,46 @@ where
                        failed += 1;
                    }
                }
+                // Once the hedge has fired, stop as soon as the stripe can finish
+                // (all data present, or the decode+verification quorum) rather than
+                // draining every straggler — a shard that resolves just after the
+                // hedge must not pull the first byte back out to the read timeout.
+                // Gated on `hedged` so a healthy stripe still reads every shard
+                // (the read-all default invariant, backlog#923 gate off).
+                if hedged {
+                    let succeeded = shards.iter().filter(|shard| shard.is_some()).count();
+                    let data_missing = shards.iter().take(data_shards).any(|shard| shard.is_none());
+                    if !data_missing || succeeded > data_shards {
+                        break;
+                    }
+                }
            }
        }

-        // A data shard may have died during this stripe's reads. The unengaged
+        // Retire any shard the hedge left in flight: dropping `sets` above
+        // cancelled its read, so the reader is now at an indeterminate stream
+        // position and can never be block-aligned again — the same reason a
+        // mid-block read error retires a reader. Its slot stays missing and is
+        // covered by the stripe-aligned parity substitution below.
+        if hedged {
+            for i in 0..num_readers {
+                if participating[i] && shards[i].is_none() && errs[i].is_none() {
+                    errs[i] = Some(Error::from(io::Error::new(ErrorKind::TimedOut, "shard read hedged after a slow shard")));
+                    retire_readers.push(i);
+                }
+            }
+        }
+
+        // A data shard may have died or been hedged this stripe. The unengaged
        // parity readers are still unopened, so they can be aligned to *this*
        // stripe and read now: engage them one at a time until the stripe has
        // `data_shards + 1` successful shards (decode quorum plus one
-        // reconstruction-verification source) or parity runs out.
+        // reconstruction-verification source) or parity runs out. In the read-all
+        // default (`RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE` off) every parity
+        // slot is already engaged, so `!self.engaged[idx]` finds nothing and this
+        // loop is a no-op — the hedged data shard is reconstructed from the parity
+        // already read above. It only substitutes deferred parity in the
+        // data-shards-only path (backlog#923).
        loop {
            let success_now = shards.iter().filter(|shard| shard.is_some()).count();
            let data_shard_missing = shards.iter().take(data_shards).any(|shard| shard.is_none());
@@ -1691,29 +1774,78 @@ impl Erasure {

                    if idx + 1 < blocks.len() {
                        // Overlap: read stripe idx+1 while reconstructing/emitting idx.
-                        let read_fut = read_stripe_timed(&mut reader, stage_metrics_enabled);
-                        let emit_fut = self.emit_decoded_stripe(
-                            writer,
-                            &mut shards,
-                            &errs,
-                            block_offset,
-                            block_length,
-                            &mut written,
-                            &mut ret_err,
-                            stage_metrics_enabled,
-                        );
-                        let (next, flow) = tokio::join!(read_fut, emit_fut);
+                        //
+                        // Cancel-safety (https://github.com/rustfs/backlog/issues/1310):
+                        // the prefetch read and the emit run concurrently, but if emit
+                        // terminates the loop — client disconnect or any emit error —
+                        // we must NOT keep waiting for the speculative next-stripe read.
+                        // A plain `tokio::join!` did exactly that: it drove both futures
+                        // to completion, so a `Stop` still blocked on the prefetch read,
+                        // which could stall for a whole shard-read deadline on a slow or
+                        // wedged remote shard before the GET could fail. Instead, drive
+                        // both with a biased `select!` and, the moment emit reports
+                        // `Stop`, drop the in-flight read future.
+                        //
+                        // Dropping the future is a real cancellation, not a detach: the
+                        // entire read pipeline is structured async (a `FuturesUnordered`
+                        // of `read_shard` futures inside `ParallelReader::read`, with no
+                        // `tokio::spawn`), so dropping `read_fut` drops every in-flight
+                        // shard read and propagates cancellation down to the
+                        // RemoteDisk / HTTP reader — no background read is left behind.
+                        //
+                        // The default path (`legacy_stripe_prefetch_enabled() == false`)
+                        // never reaches this branch, so this changes nothing for the
+                        // strictly-serial read → reconstruct → emit default.
+                        //
+                        // The `select!` is scoped so that both pinned futures are
+                        // dropped at the end of this inner block — before `reader` and
+                        // `shards` are borrowed again below. In the `Stop` case that
+                        // drop is what cancels the still-in-flight prefetch read.
+                        let (flow, next): (Option<StripeFlow>, Option<StripeReadOutput>) = {
+                            let read_fut = read_stripe_timed(&mut reader, stage_metrics_enabled);
+                            let emit_fut = self.emit_decoded_stripe(
+                                writer,
+                                &mut shards,
+                                &errs,
+                                block_offset,
+                                block_length,
+                                &mut written,
+                                &mut ret_err,
+                                stage_metrics_enabled,
+                            );
+                            tokio::pin!(read_fut);
+                            tokio::pin!(emit_fut);
+
+                            let mut next: Option<StripeReadOutput> = None;
+                            let mut flow: Option<StripeFlow> = None;
+                            // Poll until emit finishes. Once it has, only keep waiting for
+                            // the prefetch read when emit said `Continue` (that stripe is
+                            // still needed). On `Stop` the loop condition is false
+                            // immediately and we break; the pinned read future is then
+                            // dropped at the end of this block, cancelling the in-flight
+                            // read. `biased` polls emit first so a ready `Stop` cancels the
+                            // read at the earliest opportunity.
+                            while flow.is_none() || (matches!(flow, Some(StripeFlow::Continue)) && next.is_none()) {
+                                tokio::select! {
+                                    biased;
+                                    f = &mut emit_fut, if flow.is_none() => flow = Some(f),
+                                    n = &mut read_fut, if next.is_none() => next = Some(n),
+                                }
+                            }
+                            (flow, next)
+                        };
+
                        match flow {
-                            StripeFlow::Continue => {
+                            // `next` is guaranteed `Some` here: the loop only exits with
+                            // `Continue` once the read has also completed.
+                            Some(StripeFlow::Continue) => {
                                reader.recycle_shards(&mut shards);
-                                current = Some(next);
-                            }
-                            StripeFlow::Stop => {
-                                // `next` is speculative; drop it without surfacing
-                                // its errors against the stripe that stopped.
-                                drop(next);
-                                break;
+                                current = next;
                            }
+                            // Emit stopped the loop. The speculative read is cancelled by
+                            // dropping `read_fut` (it goes out of scope on `break`); its
+                            // errors never surface against the stripe that stopped.
+                            _ => break,
                        }
                    } else {
                        // Final stripe: nothing left to prefetch.
@@ -1895,6 +2027,16 @@ mod tests {
            emitted: bool,
        },
        TimedOut,
+        /// Serves `cursor` (typically the first stripe's bytes) normally, then
+        /// once it is exhausted parks on a long `sleep` instead of returning EOF —
+        /// modelling a shard whose *next*-stripe read never completes (a wedged or
+        /// very slow remote shard). Used by the prefetch cancel-safety test: the
+        /// speculative next-stripe read must be cancelled, not waited out.
+        PrefixThenSlow {
+            cursor: Cursor<Vec<u8>>,
+            stall: Duration,
+            sleep: Option<Pin<Box<Sleep>>>,
+        },
    }

    impl crate::erasure::coding::ShardSource for TestShardReader {}
@@ -1928,6 +2070,24 @@ mod tests {
                    Poll::Ready(Ok(()))
                }
                TestShardReader::TimedOut => Poll::Ready(Err(io::Error::new(ErrorKind::TimedOut, "test shard read timed out"))),
+                TestShardReader::PrefixThenSlow { cursor, stall, sleep } => {
+                    let before = buf.filled().len();
+                    match Pin::new(cursor).poll_read(cx, buf) {
+                        // Cursor still has bytes for the current stripe: serve them.
+                        Poll::Ready(Ok(())) if buf.filled().len() != before => Poll::Ready(Ok(())),
+                        // Cursor exhausted (would-be EOF): stall on a long sleep rather
+                        // than signalling EOF, so the next-stripe read hangs. This parks
+                        // the task cleanly (no busy `wake_by_ref` spin), letting the
+                        // `#[tokio::test(start_paused = true)]` clock auto-advance.
+                        Poll::Ready(Ok(())) => {
+                            let stall = *stall;
+                            let sleeper = sleep.get_or_insert_with(|| Box::pin(tokio::time::sleep(stall)));
+                            let _ = sleeper.as_mut().poll(cx);
+                            Poll::Pending
+                        }
+                        other => other,
+                    }
+                }
            }
        }
    }
@@ -2709,6 +2869,77 @@ mod tests {
        });
    }

+    /// Cancel-safety (https://github.com/rustfs/backlog/issues/1310): when the
+    /// current stripe's emit fails (client disconnect / broken pipe), the
+    /// speculatively prefetched next-stripe read must be *cancelled*, not waited
+    /// out. Each shard serves the first stripe and then stalls for `STALL` on the
+    /// next-stripe read (a wedged remote shard); the per-shard read timeout is
+    /// pinned far above the assertion window. Under the paused clock the correct
+    /// path returns at virtual t≈0 (read future dropped); a `tokio::join!`
+    /// regression would block until the shard read timeout, so this test fails
+    /// closed if cancel-safety is reverted.
+    #[tokio::test(start_paused = true)]
+    #[serial_test::serial]
+    async fn test_legacy_prefetch_cancels_next_read_on_emit_failure() {
+        const DATA_SHARDS: usize = 4;
+        const PARITY_SHARDS: usize = 2;
+        const BLOCK_SIZE: usize = 64;
+        // Two full stripes: stripe 0 emits (and fails), stripe 1 is the prefetch
+        // that must be cancelled.
+        let total_data: Vec<u8> = (0..(2 * BLOCK_SIZE) as u32).map(|i| i as u8).collect();
+        let total_len = total_data.len();
+        let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+        let shard_size = erasure.shard_size();
+        let hash_algo = HashAlgorithm::HighwayHash256;
+        let hash_size = hash_algo.size();
+        let shard_bufs = encode_prefetch_object(&erasure, &total_data, &hash_algo).await;
+        let first_block_len = hash_size + shard_size;
+
+        // Shard read timeout and next-stripe stall both far exceed the assertion
+        // window, so a regression cannot "pass" by timing out inside the window.
+        const READ_TIMEOUT_SECS: &str = "600";
+        const STALL: Duration = Duration::from_secs(3600);
+        const ASSERT_WINDOW: Duration = Duration::from_secs(5);
+
+        // Overlap on (bitrot-decode overlap switch) so the prefetch branch runs.
+        let vars = [
+            (ENV_RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE, Some("true")),
+            (rustfs_config::ENV_OBJECT_DISK_READ_TIMEOUT, Some(READ_TIMEOUT_SECS)),
+        ];
+        temp_env::async_with_vars(vars, async {
+            let readers: Vec<Option<BitrotReader<TestShardReader>>> = shard_bufs
+                .iter()
+                .map(|buf| {
+                    // Serve only the first stripe's bytes, then stall the next read.
+                    let prefix = buf[..first_block_len.min(buf.len())].to_vec();
+                    let reader = TestShardReader::PrefixThenSlow {
+                        cursor: Cursor::new(prefix),
+                        stall: STALL,
+                        sleep: None,
+                    };
+                    Some(BitrotReader::new(reader, shard_size, hash_algo.clone(), false))
+                })
+                .collect();
+
+            let mut writer = FailingEmitWriter;
+            let start = TokioInstant::now();
+            let (written, err) = erasure.decode(&mut writer, readers, 0, total_len, total_len).await;
+            let elapsed = start.elapsed();
+
+            // Emit failed on stripe 0, so the GET fails with no bytes emitted.
+            assert!(err.is_some(), "emit failure must surface as an error");
+            assert_eq!(written, 0, "the failing writer accepts no bytes");
+            // The decisive assertion: the prefetch read was cancelled rather than
+            // awaited. Without cancel-safety this would take READ_TIMEOUT_SECS.
+            assert!(
+                elapsed < ASSERT_WINDOW,
+                "prefetch next-stripe read was not cancelled on emit failure: took {elapsed:?} \
+                 (>= {ASSERT_WINDOW:?}); a tokio::join! regression waits out the shard read timeout"
+            );
+        })
+        .await;
+    }
+
    /// Decode a healthy 2+2 multi-stripe object through the lockstep GET path
    /// and return the raw bytes pulled from each shard stream.
    async fn healthy_lockstep_shard_bytes() -> (usize, Vec<usize>) {
@@ -3632,6 +3863,152 @@ mod tests {
        assert_eq!(DATA_SHARDS, bufs.iter().filter(|buf| buf.is_some()).count());
    }

+    /// Lockstep GET regression for the large-object first-byte tail (backlog#1156).
+    /// A single slow data shard must not stall the stripe for the full per-shard
+    /// `read_timeout`: once the hedge delay elapses and the stripe already holds a
+    /// decode-plus-verification quorum, the lockstep read retires the straggler and
+    /// returns in ~hedge_delay even though `read_timeout` is 60s. Before the hedge
+    /// the lockstep path waited for every launched read, so this completed in
+    /// ~read_timeout (the 10s stall observed on large-object GETs) instead of ~100ms.
+    #[tokio::test]
+    async fn test_lockstep_hedges_slow_shard_instead_of_waiting_read_timeout() {
+        const NUM_SHARDS: usize = 1;
+        const BLOCK_SIZE: usize = 64;
+        const DATA_SHARDS: usize = 2;
+        // Two parity so data_shards + 1 shards are available without the slow one.
+        const PARITY_SHARDS: usize = 2;
+        const SHARD_SIZE: usize = BLOCK_SIZE / DATA_SHARDS;
+
+        let hash_algo = HashAlgorithm::None;
+        let readers = vec![
+            // Data shard 0 never resolves; without the hedge the stripe waits out
+            // the full 60s read timeout for it.
+            Some(BitrotReader::new(TestShardReader::Pending, SHARD_SIZE, hash_algo.clone(), false)),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![1_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![2_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![3_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo,
+                false,
+            )),
+        ];
+
+        let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+        // Reconstruction verification selects the lockstep read path.
+        let mut parallel_reader = ParallelReader::new_with_metrics_path_read_costs_timeout_and_reconstruction_verification(
+            readers,
+            erasure,
+            0,
+            NUM_SHARDS * BLOCK_SIZE,
+            None,
+            vec![ShardReadCost::Unknown; 4],
+            Duration::from_secs(60),
+            true,
+        );
+        assert!(parallel_reader.verify_reconstruction, "test must exercise the lockstep read path");
+
+        let (bufs, errs) = tokio::time::timeout(Duration::from_millis(500), parallel_reader.read())
+            .await
+            .expect("lockstep read must hedge the slow shard instead of waiting out the 60s read timeout");
+
+        // The slow data shard was hedged: retired with a TimedOut error; the stripe
+        // reached data_shards + 1 shards from the healthy data + parity.
+        assert!(matches!(&errs[0], Some(DiskError::Io(err)) if err.kind() == ErrorKind::TimedOut));
+        assert!(parallel_reader.readers[0].is_none());
+        assert!(bufs[0].is_none());
+        assert!(bufs[1].is_some());
+        assert!(bufs[2].is_some());
+        assert!(bufs[3].is_some());
+        assert_eq!(DATA_SHARDS + 1, bufs.iter().filter(|buf| buf.is_some()).count());
+    }
+
+    /// Lockstep verification-quorum regression (backlog#1156). When a data shard is
+    /// missing, the hedge must settle only at `data_shards + 1` (decode quorum plus
+    /// a reconstruction-verification source), never at exactly `data_shards` — that
+    /// would silently skip reconstruction verification (erasure.rs verifies only
+    /// when `available_shards > data_shards`) and could stream a corrupt
+    /// reconstruction. Here `data1` and `parity2` are instant, `data0` is dead-slow,
+    /// and the verification-source `parity3` returns *after* the hedge delay. The
+    /// read must wait for `parity3` (reaching `data_shards + 1`) rather than settle
+    /// at `data_shards` and abandon it.
+    #[tokio::test]
+    async fn test_lockstep_hedge_waits_for_verification_quorum() {
+        const NUM_SHARDS: usize = 1;
+        const BLOCK_SIZE: usize = 64;
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const SHARD_SIZE: usize = BLOCK_SIZE / DATA_SHARDS;
+
+        let hash_algo = HashAlgorithm::None;
+        // parity3 resolves 200ms in — after the 100ms hedge, but well under the
+        // 500ms test bound and the 60s read timeout.
+        let parity3_ready_at = TokioInstant::now() + Duration::from_millis(200);
+        let readers = vec![
+            Some(BitrotReader::new(TestShardReader::Pending, SHARD_SIZE, hash_algo.clone(), false)),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![1_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![2_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::ReadyAt {
+                    cursor: Cursor::new(vec![3_u8; SHARD_SIZE * NUM_SHARDS]),
+                    ready_at: parity3_ready_at,
+                    sleep: None,
+                },
+                SHARD_SIZE,
+                hash_algo,
+                false,
+            )),
+        ];
+
+        let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+        let mut parallel_reader = ParallelReader::new_with_metrics_path_read_costs_timeout_and_reconstruction_verification(
+            readers,
+            erasure,
+            0,
+            NUM_SHARDS * BLOCK_SIZE,
+            None,
+            vec![ShardReadCost::Unknown; 4],
+            Duration::from_secs(60),
+            true,
+        );
+        assert!(parallel_reader.verify_reconstruction, "test must exercise the lockstep read path");
+
+        let (bufs, errs) = tokio::time::timeout(Duration::from_millis(500), parallel_reader.read())
+            .await
+            .expect("hedge must reach data_shards + 1, not stall to the 60s read timeout");
+
+        // data0 stays missing/retired; the verification-source parity3 was NOT
+        // abandoned at the hedge — the read waited for it to reach data_shards + 1.
+        assert!(matches!(&errs[0], Some(DiskError::Io(err)) if err.kind() == ErrorKind::TimedOut));
+        assert!(parallel_reader.readers[0].is_none());
+        assert!(bufs[0].is_none());
+        assert!(bufs[1].is_some());
+        assert!(bufs[2].is_some());
+        assert!(bufs[3].is_some(), "verification-source parity must be used, not settled away");
+        assert!(parallel_reader.readers[3].is_some(), "verification-source parity must not be retired");
+        assert_eq!(DATA_SHARDS + 1, bufs.iter().filter(|buf| buf.is_some()).count());
+    }
+
    #[tokio::test]
    async fn test_parallel_reader_drains_completed_shards_after_quorum() {
        const NUM_SHARDS: usize = 1;
@@ -171,19 +171,90 @@ fn quorum_dominant_error_metric_label(summary: &WriteQuorumFailureSummary) -> &'
    dominant_error_summary_label(summary)
 }

+/// Progress deadlines that bound how long shard writers may stall.
+///
+/// A single shard write (or writer shutdown) is a unit of forward progress: for
+/// a remote shard the underlying `HttpWriter` buffers the bytes and hands them
+/// to a background HTTP task, so a peer that accepts the connection but never
+/// drains the body eventually wedges the write once the bounded buffers fill.
+/// Without a deadline that stalled writer is awaited forever and pins an
+/// otherwise-healthy write quorum (rustfs/backlog#1319).
+///
+/// * `stall_timeout` is re-armed on every shard write, so it bounds a stall, not
+///   the total transfer time of a large object — a slow-but-honest writer that
+///   keeps completing shards is never killed.
+/// * `absolute_cap` is an optional administrator backstop: the shard writers for
+///   one object are engaged for at most this long in aggregate. It defends
+///   against a "slow-drip" peer that produces just enough progress to reset the
+///   per-shard timeout on every block while never converging. Disabled by
+///   default so a legitimate large upload over a slow link is not killed on
+///   total time alone.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct WriteProgressPolicy {
+    stall_timeout: Option<std::time::Duration>,
+    absolute_cap: Option<std::time::Duration>,
+}
+
+impl WriteProgressPolicy {
+    /// Build a policy, mapping a zero duration to "disabled" for both knobs.
+    pub(crate) fn new(stall_timeout: std::time::Duration, absolute_cap: std::time::Duration) -> Self {
+        Self {
+            stall_timeout: (!stall_timeout.is_zero()).then_some(stall_timeout),
+            absolute_cap: (!absolute_cap.is_zero()).then_some(absolute_cap),
+        }
+    }
+}
+
+impl Default for WriteProgressPolicy {
+    fn default() -> Self {
+        Self::new(
+            crate::disk::disk_store::get_object_disk_write_stall_timeout(),
+            crate::disk::disk_store::get_object_disk_write_absolute_cap(),
+        )
+    }
+}
+
 pub(crate) struct MultiWriter<'a> {
    writers: &'a mut [Option<BitrotWriterWrapper>],
    write_quorum: usize,
    errs: Vec<Option<Error>>,
+    policy: WriteProgressPolicy,
+    /// Absolute deadline for the whole object, armed lazily on the first
+    /// progress-bounded operation when `policy.absolute_cap` is set.
+    absolute_deadline: Option<tokio::time::Instant>,
 }

 impl<'a> MultiWriter<'a> {
    pub fn new(writers: &'a mut [Option<BitrotWriterWrapper>], write_quorum: usize) -> Self {
+        Self::with_policy(writers, write_quorum, WriteProgressPolicy::default())
+    }
+
+    pub fn with_policy(writers: &'a mut [Option<BitrotWriterWrapper>], write_quorum: usize, policy: WriteProgressPolicy) -> Self {
        let length = writers.len();
        MultiWriter {
            writers,
            write_quorum,
            errs: vec![None; length],
+            policy,
+            absolute_deadline: None,
+        }
+    }
+
+    /// Effective budget for one shard operation: the smaller of the per-shard
+    /// stall timeout and the time remaining until the object's absolute cap.
+    /// Returns `None` when neither deadline is configured (wait indefinitely).
+    fn next_progress_budget(&mut self) -> Option<std::time::Duration> {
+        if let Some(cap) = self.policy.absolute_cap {
+            let deadline = *self
+                .absolute_deadline
+                .get_or_insert_with(|| tokio::time::Instant::now() + cap);
+            let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
+            match self.policy.stall_timeout {
+                Some(stall) => Some(stall.min(remaining)),
+                None => Some(remaining),
+            }
+        } else {
+            self.policy.stall_timeout
        }
    }

@@ -214,13 +285,30 @@ impl<'a> MultiWriter<'a> {
    pub async fn write(&mut self, data: Vec<Bytes>) -> std::io::Result<()> {
        assert_eq!(data.len(), self.writers.len());

+        let budget = self.next_progress_budget();
        {
            let mut futures = FuturesUnordered::new();
            for ((writer_opt, err), shard) in self.writers.iter_mut().zip(self.errs.iter_mut()).zip(data.iter()) {
                if err.is_some() {
                    continue; // Skip if we already have an error for this writer
                }
-                futures.push(Self::write_shard(writer_opt, err, shard));
+                // A shard write that makes no forward progress within `budget` is
+                // failed and its disk dropped, so a stalled peer cannot pin an
+                // otherwise-healthy write quorum (rustfs/backlog#1319). `budget`
+                // is recomputed per block, so it bounds a stall — not the total
+                // transfer time — and a slow-but-honest writer is never killed.
+                futures.push(async move {
+                    match budget {
+                        Some(budget) => match tokio::time::timeout(budget, Self::write_shard(writer_opt, err, shard)).await {
+                            Ok(()) => {}
+                            Err(_elapsed) => {
+                                *err = Some(Error::Timeout);
+                                *writer_opt = None;
+                            }
+                        },
+                        None => Self::write_shard(writer_opt, err, shard).await,
+                    }
+                });
            }
            while let Some(()) = futures.next().await {}
        }
@@ -269,13 +357,30 @@ impl<'a> MultiWriter<'a> {

    pub async fn shutdown(&mut self) -> std::io::Result<()> {
        crate::hp_guard!("MultiWriter::shutdown");
+        let budget = self.next_progress_budget();
        {
            let mut futures = FuturesUnordered::new();
            for (writer_opt, err) in self.writers.iter_mut().zip(self.errs.iter_mut()) {
                if err.is_some() {
                    continue;
                }
-                futures.push(Self::shutdown_writer(writer_opt, err));
+                // Shutdown flushes the tail and waits for the remote to finish the
+                // HTTP request; a black-hole peer that never responds would hang
+                // here forever for a small object whose bytes were fully buffered
+                // (so `write` never blocked). Bound it with the same progress
+                // budget and drop the stalled writer before the quorum check.
+                futures.push(async move {
+                    match budget {
+                        Some(budget) => match tokio::time::timeout(budget, Self::shutdown_writer(writer_opt, err)).await {
+                            Ok(()) => {}
+                            Err(_elapsed) => {
+                                *err = Some(Error::Timeout);
+                                *writer_opt = None;
+                            }
+                        },
+                        None => Self::shutdown_writer(writer_opt, err).await,
+                    }
+                });
            }
            while let Some(()) = futures.next().await {}
        }
@@ -728,10 +833,12 @@ mod tests {
    use crate::erasure::coding::{BitrotWriterWrapper, CustomWriter};
    use rustfs_rio::HardLimitReader;
    use rustfs_utils::HashAlgorithm;
+    use std::future::Future;
    use std::io::Cursor;
    use std::pin::Pin;
    use std::sync::{Arc, Mutex};
    use std::task::{Context, Poll};
+    use std::time::Duration;
    use tokio::io::{AsyncWrite, AsyncWriteExt};

    #[derive(Clone, Default)]
@@ -821,6 +928,101 @@ mod tests {
        }
    }

+    /// A writer that models a "black-hole" peer: it accepts up to `stall_after`
+    /// bytes and then parks every subsequent `poll_write` forever (never
+    /// registering a waker), the way a remote `HttpWriter` wedges once its
+    /// bounded buffers fill and the peer never drains the body. With
+    /// `stall_after = 0` it stalls on the very first byte. `poll_shutdown`
+    /// completes only when `stall_shutdown` is false.
+    #[derive(Clone)]
+    struct StallingWriter {
+        accepted: Arc<std::sync::atomic::AtomicUsize>,
+        stall_after: usize,
+        stall_shutdown: bool,
+    }
+
+    impl StallingWriter {
+        fn stalls_on_write(stall_after: usize) -> Self {
+            Self {
+                accepted: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
+                stall_after,
+                stall_shutdown: false,
+            }
+        }
+
+        fn stalls_on_shutdown() -> Self {
+            Self {
+                accepted: Arc::new(std::sync::atomic::AtomicUsize::new(0)),
+                stall_after: usize::MAX,
+                stall_shutdown: true,
+            }
+        }
+    }
+
+    impl AsyncWrite for StallingWriter {
+        fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &[u8]) -> Poll<std::io::Result<usize>> {
+            use std::sync::atomic::Ordering;
+            let already = self.accepted.load(Ordering::SeqCst);
+            if already >= self.stall_after {
+                // Black hole: no forward progress, no waker registered.
+                return Poll::Pending;
+            }
+            let take = buf.len().min(self.stall_after - already);
+            self.accepted.fetch_add(take, Ordering::SeqCst);
+            Poll::Ready(Ok(take))
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            if self.stall_shutdown {
+                Poll::Pending
+            } else {
+                Poll::Ready(Ok(()))
+            }
+        }
+    }
+
+    /// A writer that always makes forward progress but each `poll_write` first
+    /// waits `delay` (on the runtime timer). Under a paused virtual clock this
+    /// deterministically models a slow-but-honest peer: it must never be killed
+    /// by the per-shard stall deadline as long as `delay < stall_timeout`, yet an
+    /// absolute cap can still bound its aggregate engagement.
+    struct SlowWriter {
+        delay: std::time::Duration,
+        timer: Option<Pin<Box<tokio::time::Sleep>>>,
+    }
+
+    impl SlowWriter {
+        fn new(delay: std::time::Duration) -> Self {
+            Self { delay, timer: None }
+        }
+    }
+
+    impl AsyncWrite for SlowWriter {
+        fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8]) -> Poll<std::io::Result<usize>> {
+            let delay = self.delay;
+            let timer = self.timer.get_or_insert_with(|| Box::pin(tokio::time::sleep(delay)));
+            match timer.as_mut().poll(cx) {
+                Poll::Ready(()) => {
+                    self.timer = None;
+                    Poll::Ready(Ok(buf.len()))
+                }
+                Poll::Pending => Poll::Pending,
+            }
+        }
+
+        fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+
+        fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<std::io::Result<()>> {
+            Poll::Ready(Ok(()))
+        }
+    }
+
    fn bitrot_writer<W>(writer: W, shard_size: usize) -> BitrotWriterWrapper
    where
        W: AsyncWrite + Send + Sync + Unpin + 'static,
@@ -828,6 +1030,16 @@ mod tests {
        BitrotWriterWrapper::new(CustomWriter::new_tokio_writer(writer), shard_size, HashAlgorithm::HighwayHash256S)
    }

+    /// Like `bitrot_writer` but with no interleaved hash, so one shard write
+    /// issues a single `poll_write` to the underlying fake. Used by the paused
+    /// virtual-clock stall tests so each block maps to exactly one modeled delay.
+    fn bitrot_writer_plain<W>(writer: W, shard_size: usize) -> BitrotWriterWrapper
+    where
+        W: AsyncWrite + Send + Sync + Unpin + 'static,
+    {
+        BitrotWriterWrapper::new(CustomWriter::new_tokio_writer(writer), shard_size, HashAlgorithm::None)
+    }
+
    #[tokio::test]
    async fn helper_writers_cover_flush_and_shutdown_paths() {
        let mut failing_write = FailingWriteWriter;
@@ -922,6 +1134,194 @@ mod tests {
        assert!(shutdown_err.contains("erasure write quorum"));
    }

+    // The production wiring (`MultiWriter::new`) must arm a real deadline by
+    // default, and honor `0` as "disabled". This guards against the fix silently
+    // regressing to wait-forever behavior on the default PUT path.
+    #[test]
+    fn default_write_progress_policy_is_armed_and_configurable() {
+        temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, || {
+            temp_env::with_var_unset(rustfs_config::ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP, || {
+                let policy = WriteProgressPolicy::default();
+                assert_eq!(
+                    policy.stall_timeout,
+                    Some(Duration::from_secs(rustfs_config::DEFAULT_OBJECT_DISK_WRITE_STALL_TIMEOUT)),
+                    "the default PUT path must be protected by a stall deadline"
+                );
+                assert_eq!(policy.absolute_cap, None, "the absolute cap is disabled by default");
+            });
+        });
+
+        temp_env::with_var(rustfs_config::ENV_OBJECT_DISK_WRITE_STALL_TIMEOUT, Some("0"), || {
+            let policy = WriteProgressPolicy::default();
+            assert_eq!(policy.stall_timeout, None, "0 disables the stall deadline (wait indefinitely)");
+        });
+    }
+
+    fn four_shards() -> Vec<Bytes> {
+        vec![
+            Bytes::from_static(b"shard-0"),
+            Bytes::from_static(b"shard-1"),
+            Bytes::from_static(b"shard-2"),
+            Bytes::from_static(b"shard-3"),
+        ]
+    }
+
+    /// Four full-`shard_size` shards. A shard shorter than `shard_size` marks the
+    /// bitrot writer `finished`, so multi-block tests must send full shards to
+    /// keep writing across iterations.
+    fn four_full_shards(shard_size: usize) -> Vec<Bytes> {
+        (0..4).map(|i| Bytes::from(vec![0x40 + i as u8; shard_size])).collect()
+    }
+
+    // rustfs/backlog#1319: a single black-hole writer that never accepts a byte
+    // must be failed within the stall budget so the remaining 3/4 writers still
+    // meet a write quorum of 3 — the write must not wait forever.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_write_stall_fails_black_hole_but_meets_quorum() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..3).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_write(0), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[2].clone()), 64)),
+        ];
+
+        {
+            let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            mw.write(four_shards())
+                .await
+                .expect("one stalled writer must still satisfy a 3/4 write quorum without hanging");
+        }
+
+        assert!(writers[0].is_none(), "the black-hole writer must be failed and dropped before commit");
+        assert!(writers[1].is_some() && writers[2].is_some() && writers[3].is_some());
+    }
+
+    // Two black-hole writers drop the healthy count to 2/4, below the quorum of
+    // 3, so the write must fail cleanly (not hang).
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_write_stall_two_black_holes_fail_quorum() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..2).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_write(0), 64)),
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_write(0), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+        ];
+
+        let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+        let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+        let err = mw
+            .write(four_shards())
+            .await
+            .expect_err("two stalled writers must fail the write quorum instead of hanging");
+        assert!(err.to_string().contains("Failed to write data"));
+    }
+
+    // A small object whose bytes were fully buffered leaves `write` succeeding
+    // for a black-hole peer; the stall must then be caught during shutdown so it
+    // cannot pin the quorum there either.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_shutdown_stall_fails_black_hole_but_meets_quorum() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..3).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_shutdown(), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[2].clone()), 64)),
+        ];
+
+        {
+            let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            mw.write(four_shards()).await.expect("all writers accept the buffered shard");
+            mw.shutdown()
+                .await
+                .expect("a single shutdown stall must still satisfy a 3/4 quorum without hanging");
+        }
+
+        assert!(writers[0].is_none(), "the writer that stalled shutdown must be failed and dropped");
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_shutdown_stall_two_black_holes_fail_quorum() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..2).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_shutdown(), 64)),
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_shutdown(), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+        ];
+
+        let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+        let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+        mw.write(four_shards()).await.expect("all writers accept the buffered shard");
+        let err = mw
+            .shutdown()
+            .await
+            .expect_err("two shutdown stalls must fail the shutdown quorum instead of hanging");
+        assert!(err.to_string().contains("Failed to shutdown writers"));
+    }
+
+    // A slow-but-honest writer that keeps completing shards (delay < stall
+    // budget) must never be killed, even across many blocks, when no absolute cap
+    // is configured.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_slow_but_progressing_writer_is_not_killed() {
+        let mut writers = vec![
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(1)), 64)),
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(1)), 64)),
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(1)), 64)),
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(1)), 64)),
+        ];
+
+        {
+            let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            for _ in 0..8 {
+                mw.write(four_full_shards(64))
+                    .await
+                    .expect("a writer that keeps making progress within the stall budget must not be failed");
+            }
+        }
+
+        assert!(writers.iter().all(Option::is_some), "no slow-but-honest writer should be dropped");
+    }
+
+    // Slow-drip defense: the per-shard stall timer resets on every block, so a
+    // peer that dribbles just enough progress each block is not caught by the
+    // stall timeout alone. The optional absolute cap bounds its aggregate
+    // engagement and fails it within a finite budget, while the healthy writers
+    // still meet quorum.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_absolute_cap_bounds_slow_drip_writer() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..3).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_secs(8)), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[2].clone()), 64)),
+        ];
+
+        {
+            // stall (10s) never fires for an 8s-per-block writer, but the 15s
+            // absolute cap does once its aggregate engagement crosses the cap.
+            let policy = WriteProgressPolicy::new(Duration::from_secs(10), Duration::from_secs(15));
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            for _ in 0..4 {
+                mw.write(four_full_shards(64))
+                    .await
+                    .expect("healthy writers keep the 3/4 quorum while the drip writer is bounded");
+            }
+        }
+
+        assert!(
+            writers[0].is_none(),
+            "the slow-drip writer must be failed within the absolute cap, not held forever"
+        );
+    }
+
    #[tokio::test]
    async fn drain_queued_inflight_bytes_consumes_pending_blocks() {
        let (tx, mut rx) = mpsc::channel(2);
@@ -36,6 +36,7 @@ mod cache_value;
 mod cluster;
 mod config;
 mod core;
+mod crash_inject;
 mod data_movement;
 mod data_usage;
 mod diagnostics;
@@ -429,4 +429,19 @@ mod tests {
        assert!(!opts.user_metadata.contains_key(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()));
        assert!(!opts.user_metadata.contains_key(X_AMZ_REPLICATION_STATUS.as_str()));
    }
+
+    #[test]
+    fn build_transition_put_options_requests_no_checksum_and_content_md5() {
+        // Regression for rustfs/rustfs#4811: transition uploads must leave the
+        // additional-checksum modes unset and rely on Content-MD5. If `checksum`
+        // were (incorrectly) reported as set, the >128 MiB multipart put path
+        // would call `ChecksumNone.hasher()` and fail with "unsupported checksum
+        // type". Objects <=128 MiB take the single-part path and only worked by
+        // silently dropping the checksum, so pin both invariants here.
+        let opts = build_transition_put_options("COLD".to_string(), HashMap::new());
+
+        assert!(!opts.checksum.is_set(), "transition put must not request an additional checksum");
+        assert!(!opts.auto_checksum.is_set(), "transition put must not preset auto_checksum");
+        assert!(opts.send_content_md5, "transition put must send Content-MD5");
+    }
 }
@@ -1275,6 +1275,10 @@ pub(in crate::set_disk) fn fill_deferred_bitrot_readers(
            continue;
        }

+        if files[idx].data.is_none() && disks[idx].is_none() {
+            continue;
+        }
+
        let inline_data = files[idx].data.clone();
        let disk = disks[idx].clone();
        let data_dir = files[idx].data_dir.unwrap_or_default();
@@ -2046,7 +2050,7 @@ impl SetDisks {
        let bucket = Arc::new(bucket.to_string());
        let object = Arc::new(object.to_string());
        let version_id = Arc::new(version_id.to_string());
-        let futures = disks.iter().map(|disk| {
+        let futures = disks.iter().enumerate().map(|(disk_index, disk)| {
            let disk = disk.clone();
            let opts = opts.clone();
            let org_bucket = org_bucket.clone();
@@ -2056,6 +2060,7 @@ impl SetDisks {
            tokio::spawn(async move {
                let response_start = observe.then(Instant::now);
                let result = if let Some(disk) = disk {
+                    Self::record_read_version_call(&object, disk_index);
                    disk.read_version(&org_bucket, &bucket, &object, &version_id, &opts).await
                } else {
                    Err(DiskError::DiskNotFound)
@@ -2141,6 +2146,7 @@ impl SetDisks {
            join_set.spawn(async move {
                let response_start = Instant::now();
                let result = if let Some(disk) = disk {
+                    Self::record_read_version_call(&object, index);
                    disk.read_version(&org_bucket, &bucket, &object, &version_id, &opts).await
                } else {
                    Err(DiskError::DiskNotFound)
@@ -2518,11 +2524,57 @@ enum OrphanDirScan {
    Missing,
 }

-fn rename_data_versions_key(versions: &[u8]) -> Option<[u8; 8]> {
-    let prefix = versions.get(..8)?;
-    let mut key = [0; 8];
-    key.copy_from_slice(prefix);
-    Some(key)
+/// Outcome of a *post-quorum* `rename_data` commit, classifying whether the
+/// committed replicas converged so the caller can decide heal admission
+/// WITHOUT conflating "a version signature exists" with "this write needs
+/// heal" (rustfs/backlog#1321).
+///
+/// `rename_data` reaches this classification only once write quorum is met — a
+/// sub-quorum commit returns `Err` and never produces a convergence — so every
+/// variant describes an already-durable, already-ACKable write.
+///
+/// # Extension point for #1312 (commit fencing)
+///
+/// #1312 layers a per-object fencing epoch onto the SAME `rename_data` path.
+/// An epoch rejection is a *commit* failure surfaced through the existing
+/// `Result::Err` channel (the write never lands), which is orthogonal to this
+/// post-commit convergence signal (the write landed; do the replicas need
+/// heal). The two therefore compose: fencing gates whether we reach a
+/// convergence at all, and this enum classifies the convergence once reached.
+/// A future fence-aware variant, if ever needed, is an additive change here.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(in crate::set_disk) enum RenameConvergence {
+    /// Every attempted disk committed and reported an identical, known version
+    /// signature. The committed replicas are already converged; no heal needed.
+    AllSuccessIdentical,
+    /// Write quorum was met but at least one attempted disk failed to commit
+    /// (error or offline). The committed set may be short a replica; the
+    /// laggard must be converged by heal.
+    PartialCommit,
+    /// Every attempted disk committed but their version signatures diverge (or
+    /// mix signed <=10-version disks with unsigned >10-version disks, itself a
+    /// version-count divergence). Heal must reconcile the committed replicas.
+    SignatureDivergent,
+    /// Every attempted disk committed but none produced a version signature
+    /// (all observed >10 versions, where the signature is deliberately
+    /// omitted). Divergence can neither be proven nor ruled out here, so any
+    /// latent divergence is left to the scanner backstop rather than enqueued
+    /// for heal.
+    Unknown,
+}
+
+impl RenameConvergence {
+    /// Whether this commit outcome must enqueue an object heal to converge the
+    /// committed replicas.
+    ///
+    /// `Unknown` and `AllSuccessIdentical` return `false`: the former is
+    /// scanner-backstopped (see the variant docs), the latter is already
+    /// converged. This is the single decision point that replaced the old
+    /// `Option<Vec<u8>>::is_some()` heuristic, under which a healthy MPU
+    /// (identical signatures on every disk) always looked like it needed heal.
+    pub(in crate::set_disk) fn needs_heal(&self) -> bool {
+        matches!(self, Self::PartialCommit | Self::SignatureDivergent)
+    }
 }

 impl SetDisks {
@@ -2551,7 +2603,7 @@ impl SetDisks {
        write_quorum: usize,
    ) -> disk::error::Result<(
        Vec<Option<DiskStore>>,
-        Option<Vec<u8>>,
+        RenameConvergence,
        Option<Uuid>,
        Vec<Option<DiskStore>>,
        Option<OldCurrentSize>,
@@ -2574,6 +2626,11 @@ impl SetDisks {
            let dst_bucket = dst_bucket.clone();

            futures.push(tokio::spawn(async move {
+                // Test-only introspection guard: counts this task as in-flight for
+                // the whole body. Compiles to `()` in production (no behavior).
+                #[allow(clippy::let_unit_value)]
+                let _fanout_task_guard = Self::rename_fanout_task_guard(&dst_object);
+
                if file_info.erasure.index == 0 {
                    file_info.erasure.index = i + 1;
                }
@@ -2582,6 +2639,10 @@ impl SetDisks {
                    return Err(DiskError::FileCorrupt);
                }

+                // Test-only awaitable pause point right before the disk rename.
+                // A no-op immediately-ready future in production.
+                Self::rename_fanout_barrier(&dst_object, i, rename_fanout_barrier_phase::RENAME).await;
+
                if let Some(disk) = disk {
                    disk.rename_data(&src_bucket, &src_object, file_info, &dst_bucket, &dst_object)
                        .await
@@ -2705,7 +2766,7 @@ impl SetDisks {
        }

        let data_dir = Self::reduce_common_data_dir(&data_dirs, write_quorum);
-        let versions = Self::select_rename_data_versions(&disk_versions, &errs, write_quorum);
+        let convergence = Self::classify_rename_convergence(&disk_versions, &errs);
        let old_current_size = Self::reduce_common_old_current_size(&old_current_sizes, write_quorum);
        let online_disks = Self::eval_disks(disks, &errs);
        let cleanup_disks = if let Some(data_dir) = data_dir {
@@ -2725,7 +2786,7 @@ impl SetDisks {
            vec![None; disks.len()]
        };

-        Ok((online_disks, versions, data_dir, cleanup_disks, old_current_size))
+        Ok((online_disks, convergence, data_dir, cleanup_disks, old_current_size))
    }

    /// rustfs/backlog#1009: reduce the per-disk observations of the
@@ -2756,61 +2817,62 @@ impl SetDisks {
        if max >= write_quorum { old_current_size } else { None }
    }

-    pub(in crate::set_disk) fn reduce_common_versions(disk_versions: &[Option<Vec<u8>>], write_quorum: usize) -> Option<Vec<u8>> {
-        let mut versions_count = HashMap::new();
-
-        for versions in disk_versions.iter().flatten() {
-            if let Some(key) = rename_data_versions_key(versions) {
-                *versions_count.entry(key).or_insert(0usize) += 1;
-            }
-        }
-
-        let (common_versions, max_count) = versions_count
-            .into_iter()
-            .max_by_key(|(_, count)| *count)
-            .unwrap_or(([0; 8], 0));
-
-        if max_count < write_quorum {
-            return None;
-        }
-
-        disk_versions
-            .iter()
-            .flatten()
-            .find(|versions| rename_data_versions_key(versions).is_some_and(|key| key == common_versions))
-            .cloned()
-    }
-
-    pub(in crate::set_disk) fn select_rename_data_versions(
+    /// Classify a *post-quorum* `rename_data` commit into an explicit
+    /// convergence outcome (rustfs/backlog#1321). This runs only after the
+    /// write-quorum gate has already passed, so every returned variant
+    /// describes an already-durable, already-ACKable write; the decision here
+    /// is purely "do the committed replicas need heal to converge".
+    ///
+    /// The per-disk version signature (`disk_versions[i]`) is used only as
+    /// comparison material — it is deliberately NOT overloaded to also mean
+    /// "needs heal". A `Some(bytes)` signature carries the concatenated
+    /// version-id bytes of a disk that observed <=10 versions; a `None`
+    /// signature is a disk that committed but deliberately produced no
+    /// signature (>10 versions). A `None` entry for a *failed* disk never
+    /// reaches this point as a signature because failures are handled first.
+    pub(in crate::set_disk) fn classify_rename_convergence(
        disk_versions: &[Option<Vec<u8>>],
        errs: &[Option<DiskError>],
-        write_quorum: usize,
-    ) -> Option<Vec<u8>> {
-        let mut versions = Self::reduce_common_versions(disk_versions, write_quorum);
-        for (dversions, err) in disk_versions.iter().zip(errs.iter()) {
-            if err.is_some() {
-                continue;
-            }
-            let Some(dversions) = dversions.as_ref().filter(|versions| !versions.is_empty()) else {
+    ) -> RenameConvergence {
+        // Any failed / offline disk that got past the write-quorum gate means a
+        // committed replica is missing or stale: converge it via heal,
+        // regardless of what the surviving disks' signatures say.
+        if errs.iter().any(|err| err.is_some()) {
+            return RenameConvergence::PartialCommit;
+        }
+
+        // Every disk committed. Compare their reported version signatures.
+        let mut seen: Option<&Vec<u8>> = None;
+        let mut signed_count = 0usize;
+        let mut divergent = false;
+        for signature in disk_versions.iter() {
+            let Some(sig) = signature.as_ref() else {
                continue;
            };
-
-            match versions.as_ref() {
-                Some(current_versions) if dversions != current_versions => {
-                    if dversions.len() > current_versions.len() {
-                        versions = Some(dversions.clone());
-                    }
-                    break;
-                }
+            signed_count += 1;
+            match seen {
+                None => seen = Some(sig),
+                Some(prev) if prev != sig => divergent = true,
                Some(_) => {}
-                None => {
-                    versions = Some(dversions.clone());
-                    break;
-                }
            }
        }

-        versions
+        if divergent {
+            return RenameConvergence::SignatureDivergent;
+        }
+        if signed_count == 0 {
+            // No disk produced a signature (all observed >10 versions): a
+            // latent divergence cannot be proven or ruled out here, so it is
+            // left to the scanner backstop rather than enqueued for heal.
+            return RenameConvergence::Unknown;
+        }
+        if signed_count < disk_versions.len() {
+            // A mix of signed (<=10 versions) and unsigned (>10 versions) disks
+            // is itself a version-count divergence between committed replicas:
+            // reconcile it via heal.
+            return RenameConvergence::SignatureDivergent;
+        }
+        RenameConvergence::AllSuccessIdentical
    }

    /// Reclaim the old (now dereferenced) `object/<old_data_dir>` on the disks
@@ -2870,6 +2932,12 @@ impl SetDisks {
            let disk = disk.clone();
            let object_for_fault = object_for_fault.clone();
            tokio::spawn(async move {
+                // Test-only introspection guard + awaitable pause point for the
+                // old-data-dir cleanup fan-out. Both compile away in production.
+                #[allow(clippy::let_unit_value)]
+                let _fanout_task_guard = Self::rename_fanout_task_guard(&object_for_fault);
+                Self::rename_fanout_barrier(&object_for_fault, idx, rename_fanout_barrier_phase::CLEANUP).await;
+
                if let Some(err) = Self::cleanup_injected_error(&object_for_fault, idx) {
                    return Some(err);
                }
@@ -2910,6 +2978,56 @@ impl SetDisks {
        None
    }

+    /// Test-only seam that records one per-disk `read_version` metadata RPC for
+    /// the call-counter registry (backlog#1325). In production this is inlined to
+    /// nothing and adds no behavior; only the `#[cfg(test)]` variant touches the
+    /// registry. Placed inside the fan-out spawn tasks so counts are observed
+    /// even though the increments run on arbitrary runtime worker threads.
+    #[cfg(test)]
+    #[inline]
+    fn record_read_version_call(object: &str, disk_index: usize) {
+        disk_call_counters::record(object, disk_call_counters::KIND_READ_VERSION, disk_index);
+    }
+
+    #[cfg(not(test))]
+    #[inline(always)]
+    fn record_read_version_call(_object: &str, _disk_index: usize) {}
+
+    /// Test-only awaitable pause point for the rename/commit fan-out (backlog#1325,
+    /// serving the barrier-style acceptances of #1312 / #1319 / #1313). `phase` is
+    /// [`rename_fanout_barrier::PHASE_RENAME`] or `PHASE_CLEANUP`. When a test has
+    /// armed a barrier for `object` at this `(disk_index, phase)`, the spawned
+    /// fan-out task blocks here until the test releases it, so the test can await
+    /// the pause point and then introspect in-flight background disk work. In
+    /// production this awaits an immediately-ready no-op future — no yield, no
+    /// registry access, no behavior change.
+    #[cfg(test)]
+    #[inline]
+    async fn rename_fanout_barrier(object: &str, disk_index: usize, phase: &'static str) {
+        rename_fanout_barrier::checkpoint(object, disk_index, phase).await;
+    }
+
+    #[cfg(not(test))]
+    #[inline(always)]
+    #[allow(clippy::unused_async)]
+    async fn rename_fanout_barrier(_object: &str, _disk_index: usize, _phase: &'static str) {}
+
+    /// Test-only RAII counter for one in-flight rename/commit fan-out task
+    /// (backlog#1325). Held for the whole spawned task body so a test observing
+    /// `object` can assert "background disk writes are still running" while the
+    /// fan-out is paused and "no background disk writes remain" once it drains —
+    /// the exact signal #1312 needs after a lock release. In production this
+    /// returns `()` and touches no registry.
+    #[cfg(test)]
+    #[inline]
+    fn rename_fanout_task_guard(object: &str) -> rename_fanout_barrier::TaskGuard {
+        rename_fanout_barrier::task_guard(object)
+    }
+
+    #[cfg(not(test))]
+    #[inline(always)]
+    fn rename_fanout_task_guard(_object: &str) {}
+
    /// Report a post-commit old-data-dir cleanup receipt: emit metrics, warn on
    /// residue/below-quorum, and — on residue — enqueue an object heal.
    ///
@@ -3905,6 +4023,340 @@ pub(in crate::set_disk) mod cleanup_fault_injection {
    }
 }

+/// Test-only per-disk call counters for the metadata fan-out (backlog#1325,
+/// serving the RPC-count assertions of #1309 / #1314 / #1315).
+///
+/// The metadata fan-out issues each per-disk `read_version` inside a
+/// `tokio::spawn` task, so the increments happen on arbitrary runtime worker
+/// threads. A thread-local `metrics::Recorder` (see `test_metrics.rs`) cannot
+/// observe those spawned increments; this registry is a process-global keyed by
+/// object name, so counts recorded inside spawned tasks are visible from the
+/// test thread that installed the observing scope.
+///
+/// The whole module is `#[cfg(test)]`, so it never compiles into production.
+/// Parallel tests stay isolated by observing distinct object names — an
+/// unobserved object records nothing, keeping the registry bounded, and each
+/// [`CallCounterScope`] clears only its own object's counts on drop.
+#[cfg(test)]
+pub(in crate::set_disk) mod disk_call_counters {
+    use std::collections::{HashMap, HashSet};
+    use std::sync::{Mutex, OnceLock};
+
+    /// Kind label for the per-disk `read_version` metadata RPC.
+    pub const KIND_READ_VERSION: &str = "read_version";
+
+    /// Registry key: (object, kind, disk_index).
+    type CountKey = (String, String, usize);
+
+    #[derive(Default)]
+    struct Registry {
+        /// Object names with an active observing scope. Only these accumulate,
+        /// so unrelated concurrent tests never inflate one another's counts.
+        observed: HashSet<String>,
+        /// (object, kind, disk_index) -> call count.
+        counts: HashMap<CountKey, u64>,
+    }
+
+    fn registry() -> &'static Mutex<Registry> {
+        static REG: OnceLock<Mutex<Registry>> = OnceLock::new();
+        REG.get_or_init(|| Mutex::new(Registry::default()))
+    }
+
+    /// Record one call of `kind` against `object`'s `disk_index`. A no-op unless
+    /// an observing scope for `object` is currently installed.
+    pub(super) fn record(object: &str, kind: &str, disk_index: usize) {
+        let mut reg = registry().lock().expect("disk call-counter registry poisoned");
+        if !reg.observed.contains(object) {
+            return;
+        }
+        *reg.counts
+            .entry((object.to_string(), kind.to_string(), disk_index))
+            .or_insert(0) += 1;
+    }
+
+    /// RAII scope that observes call counts for a single `object`. Counts recorded
+    /// while the scope is alive — including those recorded inside spawned tasks —
+    /// are queryable; the scope clears its own counts on drop.
+    #[must_use]
+    pub struct CallCounterScope {
+        object: String,
+    }
+
+    /// Begin observing per-disk call counts for `object`.
+    pub fn observe(object: &str) -> CallCounterScope {
+        let mut reg = registry().lock().expect("disk call-counter registry poisoned");
+        reg.observed.insert(object.to_string());
+        CallCounterScope {
+            object: object.to_string(),
+        }
+    }
+
+    impl CallCounterScope {
+        /// Total number of `kind` calls across all disks for the observed object.
+        pub fn total(&self, kind: &str) -> u64 {
+            let reg = registry().lock().expect("disk call-counter registry poisoned");
+            reg.counts
+                .iter()
+                .filter(|((obj, k, _), _)| obj == &self.object && k == kind)
+                .map(|(_, count)| *count)
+                .sum()
+        }
+
+        /// Number of `kind` calls recorded against a specific `disk_index`.
+        pub fn for_disk(&self, kind: &str, disk_index: usize) -> u64 {
+            let reg = registry().lock().expect("disk call-counter registry poisoned");
+            reg.counts
+                .get(&(self.object.clone(), kind.to_string(), disk_index))
+                .copied()
+                .unwrap_or(0)
+        }
+    }
+
+    impl Drop for CallCounterScope {
+        fn drop(&mut self) {
+            let mut reg = registry().lock().expect("disk call-counter registry poisoned");
+            reg.observed.remove(&self.object);
+            reg.counts.retain(|(obj, _, _), _| obj != &self.object);
+        }
+    }
+}
+
+/// Fan-out phase labels for the rename/commit barrier seam (backlog#1325).
+///
+/// These are referenced from the production fan-out call sites (as inert string
+/// literals passed to a no-op seam), so — unlike the `#[cfg(test)]` barrier
+/// registry itself — they are defined unconditionally. In production builds the
+/// seam ignores them entirely.
+pub(in crate::set_disk) mod rename_fanout_barrier_phase {
+    /// The per-disk `rename_data` phase of the write-commit fan-out.
+    pub const RENAME: &str = "rename";
+    /// The per-disk old-data-dir cleanup phase of the commit fan-out.
+    pub const CLEANUP: &str = "cleanup";
+}
+
+/// Test-only awaitable pause barrier + background-task introspection for the
+/// rename/commit fan-out (backlog#1325, the second facility block after the
+/// per-disk call counters). Serves the barrier-style white-box acceptances of
+/// #1312 (commit fencing: "abort at the first-disk rename barrier, assert the
+/// coordinator still holds the lock, assert no background disk write remains
+/// after release"), #1319, and #1313.
+///
+/// Two independent, object-keyed mechanisms share one process-global registry:
+///
+/// 1. **Awaitable pause barrier.** A test [`arm`]s a barrier for `(object,
+///    disk_index, phase)`. The matching spawned fan-out task blocks at its
+///    [`checkpoint`] until the test releases it. The test awaits the pause via
+///    [`BarrierHandle::wait_until_paused`] (a deterministic `Notify` handshake —
+///    no sleeps) and resumes it via [`BarrierHandle::release`]. At most one
+///    barrier is armed per object at a time, matching the single-scope style of
+///    `disk_call_counters`.
+///
+/// 2. **Background-task introspection.** A test [`observe_tasks`] for `object`;
+///    each instrumented fan-out task then holds a [`TaskGuard`] for its whole
+///    body, so [`TaskTrackerScope::running`] reports how many rename/cleanup
+///    background disk tasks are still in flight. This is the concrete "is there
+///    still a background disk write?" signal #1312 asserts after a lock release.
+///
+/// Both are keyed by object and only accumulate for armed/observed objects, so
+/// concurrent tests using distinct object names stay fully isolated. The whole
+/// module is `#[cfg(test)]` and never compiles into production; the fan-out call
+/// sites reach it only through the `#[cfg(not(test))]` no-op seams on `SetDisks`.
+///
+/// Scope of this block (white-box, in-process only): coordinator lock-holding is
+/// asserted by the test at the store/coordinator layer via the guard it already
+/// holds — `io_primitives` has no handle to that namespace lock, and fabricating
+/// a lock-state registry here would be a look-alike rather than the real lock.
+/// Cross-process/black-box fault injection (toxiproxy, blackhole peers, 2-pool)
+/// is a later cluster-harness block, not this one.
+#[cfg(test)]
+pub(in crate::set_disk) mod rename_fanout_barrier {
+    use std::collections::HashMap;
+    use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+    use std::sync::{Arc, Mutex, OnceLock};
+    use tokio::sync::Notify;
+
+    pub use super::rename_fanout_barrier_phase::{CLEANUP as PHASE_CLEANUP, RENAME as PHASE_RENAME};
+
+    /// One armed barrier: the fan-out task matching `(disk_index, phase)` pauses.
+    struct Armed {
+        disk_index: usize,
+        phase: &'static str,
+        /// Signalled (task -> test) when the target task reaches the checkpoint.
+        arrived: Arc<Notify>,
+        /// Signalled (test -> task) to release the paused task.
+        release: Arc<Notify>,
+        /// Set once the target task is parked at the checkpoint.
+        paused: Arc<AtomicBool>,
+    }
+
+    #[derive(Default)]
+    struct Registry {
+        /// object -> armed barrier (at most one per object).
+        armed: HashMap<String, Armed>,
+        /// object -> live in-flight fan-out task count, only for observed objects.
+        observed: HashMap<String, Arc<AtomicUsize>>,
+    }
+
+    fn registry() -> &'static Mutex<Registry> {
+        static REG: OnceLock<Mutex<Registry>> = OnceLock::new();
+        REG.get_or_init(|| Mutex::new(Registry::default()))
+    }
+
+    fn lock() -> std::sync::MutexGuard<'static, Registry> {
+        registry().lock().expect("rename fan-out barrier registry poisoned")
+    }
+
+    /// RAII handle for one armed barrier. Dropping it disarms the barrier and
+    /// releases any still-parked task, so a panicking or forgetful test can never
+    /// leave a spawned fan-out task wedged.
+    #[must_use]
+    pub struct BarrierHandle {
+        object: String,
+        arrived: Arc<Notify>,
+        release: Arc<Notify>,
+        paused: Arc<AtomicBool>,
+    }
+
+    /// Arm a barrier: the fan-out task for `object` at `(disk_index, phase)` will
+    /// pause at its checkpoint until the returned handle is released or dropped.
+    pub fn arm(object: &str, disk_index: usize, phase: &'static str) -> BarrierHandle {
+        let arrived = Arc::new(Notify::new());
+        let release = Arc::new(Notify::new());
+        let paused = Arc::new(AtomicBool::new(false));
+        lock().armed.insert(
+            object.to_string(),
+            Armed {
+                disk_index,
+                phase,
+                arrived: arrived.clone(),
+                release: release.clone(),
+                paused: paused.clone(),
+            },
+        );
+        BarrierHandle {
+            object: object.to_string(),
+            arrived,
+            release,
+            paused,
+        }
+    }
+
+    impl BarrierHandle {
+        /// Await until the armed fan-out task has parked at the checkpoint. Uses a
+        /// stored-permit `Notify`, so this is race-free regardless of whether the
+        /// task reaches the checkpoint before or after this call — no sleeps.
+        pub async fn wait_until_paused(&self) {
+            if self.paused.load(Ordering::SeqCst) {
+                return;
+            }
+            self.arrived.notified().await;
+        }
+
+        /// Whether the target task is currently parked at the checkpoint.
+        pub fn is_paused(&self) -> bool {
+            self.paused.load(Ordering::SeqCst)
+        }
+
+        /// Release the parked task so the fan-out can proceed.
+        pub fn release(&self) {
+            self.release.notify_one();
+        }
+    }
+
+    impl Drop for BarrierHandle {
+        fn drop(&mut self) {
+            // Unblock any task still parked at the checkpoint before disarming, so
+            // a dropped handle can never wedge a spawned fan-out task.
+            self.release.notify_one();
+            lock().armed.remove(&self.object);
+        }
+    }
+
+    /// Seam entry point invoked from inside each spawned fan-out task. A no-op
+    /// unless a barrier is armed for exactly this `(object, disk_index, phase)`.
+    /// The registry mutex is released before awaiting, so it is never held across
+    /// the pause.
+    pub(in crate::set_disk) async fn checkpoint(object: &str, disk_index: usize, phase: &'static str) {
+        let hooks = {
+            let reg = lock();
+            match reg.armed.get(object) {
+                Some(a) if a.disk_index == disk_index && a.phase == phase => {
+                    Some((a.arrived.clone(), a.release.clone(), a.paused.clone()))
+                }
+                _ => None,
+            }
+        };
+        if let Some((arrived, release, paused)) = hooks {
+            paused.store(true, Ordering::SeqCst);
+            arrived.notify_one();
+            release.notified().await;
+        }
+    }
+
+    /// RAII scope that observes in-flight fan-out task counts for a single
+    /// `object`. Counts accrue only while the scope is alive; it clears its own
+    /// entry on drop.
+    #[must_use]
+    pub struct TaskTrackerScope {
+        object: String,
+    }
+
+    /// Begin observing in-flight rename/cleanup fan-out task counts for `object`.
+    pub fn observe_tasks(object: &str) -> TaskTrackerScope {
+        lock()
+            .observed
+            .entry(object.to_string())
+            .or_insert_with(|| Arc::new(AtomicUsize::new(0)));
+        TaskTrackerScope {
+            object: object.to_string(),
+        }
+    }
+
+    impl TaskTrackerScope {
+        /// Number of instrumented fan-out tasks for the observed object that are
+        /// currently in flight (task body entered, guard not yet dropped).
+        pub fn running(&self) -> usize {
+            lock()
+                .observed
+                .get(&self.object)
+                .map(|c| c.load(Ordering::SeqCst))
+                .unwrap_or(0)
+        }
+    }
+
+    impl Drop for TaskTrackerScope {
+        fn drop(&mut self) {
+            lock().observed.remove(&self.object);
+        }
+    }
+
+    /// RAII guard held for the lifetime of one spawned fan-out task. Increments
+    /// the observed counter on creation (if the object is observed) and decrements
+    /// it on drop. Holding the `Arc` keeps the decrement sound even if the scope
+    /// is dropped while a task is still draining.
+    #[must_use]
+    pub struct TaskGuard {
+        counter: Option<Arc<AtomicUsize>>,
+    }
+
+    /// Create a task guard for `object`. A no-op guard unless `object` is observed.
+    pub(in crate::set_disk) fn task_guard(object: &str) -> TaskGuard {
+        let counter = lock().observed.get(object).cloned();
+        if let Some(c) = &counter {
+            c.fetch_add(1, Ordering::SeqCst);
+        }
+        TaskGuard { counter }
+    }
+
+    impl Drop for TaskGuard {
+        fn drop(&mut self) {
+            if let Some(c) = &self.counter {
+                c.fetch_sub(1, Ordering::SeqCst);
+            }
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
    use super::*;
@@ -4039,6 +4491,226 @@ mod tests {
        });
    }

+    /// Builds `count` empty local disks (each in its own tempdir) for the given
+    /// bucket. Returns the tempdir guards (which must outlive the disks) and the
+    /// disk slot vector expected by the metadata fan-out.
+    async fn call_counter_local_disks(bucket: &str, count: usize) -> (Vec<TempDir>, Vec<Option<DiskStore>>) {
+        let mut dirs = Vec::with_capacity(count);
+        let mut disks = Vec::with_capacity(count);
+        for _ in 0..count {
+            let (dir, disk) = read_multiple_test_disk(bucket, &[]).await;
+            dirs.push(dir);
+            disks.push(Some(disk));
+        }
+        (dirs, disks)
+    }
+
+    /// Demo / regression guard for the backlog#1325 per-disk call counters.
+    ///
+    /// The metadata fan-out issues each `read_version` inside its own
+    /// `tokio::spawn` task; on a multi-thread runtime those tasks land on
+    /// arbitrary worker threads. This asserts the process-global registry still
+    /// observes every per-disk increment — the exact gap that makes the
+    /// thread-local `CapturingRecorder` unusable for #1309 / #1314 counting.
+    ///
+    /// Reverting `record_read_version_call` to a no-op drops both the total and
+    /// the per-disk counts to zero and fails this test.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    async fn read_version_call_counter_observes_spawned_fanout() {
+        const DISKS: usize = 4;
+        let bucket = "counter-bucket";
+        let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+
+        let object = "counter-object";
+        let scope = disk_call_counters::observe(object);
+
+        // read_data=false, observe=false -> deterministic full-wait fan-out.
+        let _ = SetDisks::read_all_fileinfo(&disks, bucket, bucket, object, "", false, false, false).await;
+
+        assert_eq!(
+            scope.total(disk_call_counters::KIND_READ_VERSION),
+            DISKS as u64,
+            "every online disk must record exactly one read_version, even from a spawned task"
+        );
+        for idx in 0..DISKS {
+            assert_eq!(
+                scope.for_disk(disk_call_counters::KIND_READ_VERSION, idx),
+                1,
+                "disk {idx} should record exactly one read_version"
+            );
+        }
+
+        // A second observed fan-out accumulates onto the same scope.
+        let _ = SetDisks::read_all_fileinfo(&disks, bucket, bucket, object, "", false, false, false).await;
+        assert_eq!(scope.total(disk_call_counters::KIND_READ_VERSION), (DISKS * 2) as u64);
+
+        drop(dirs);
+    }
+
+    /// Isolation guard: unobserved objects record nothing (so parallel tests do
+    /// not inflate one another), and a scope clears its own counts on drop.
+    #[tokio::test]
+    async fn call_counter_isolates_unobserved_objects_and_clears_on_drop() {
+        let bucket = "counter-bucket-iso";
+        let (dirs, disks) = call_counter_local_disks(bucket, 1).await;
+        let object = "iso-object";
+
+        // No active scope: the fan-out records nothing.
+        let _ = SetDisks::read_all_fileinfo(&disks, bucket, bucket, object, "", false, false, false).await;
+
+        {
+            let scope = disk_call_counters::observe(object);
+            assert_eq!(
+                scope.total(disk_call_counters::KIND_READ_VERSION),
+                0,
+                "reads before the scope existed must not be counted"
+            );
+            let _ = SetDisks::read_all_fileinfo(&disks, bucket, bucket, object, "", false, false, false).await;
+            assert_eq!(scope.total(disk_call_counters::KIND_READ_VERSION), 1);
+        }
+
+        // Previous scope dropped -> counts cleared; a fresh scope starts at zero.
+        let scope = disk_call_counters::observe(object);
+        assert_eq!(
+            scope.total(disk_call_counters::KIND_READ_VERSION),
+            0,
+            "dropping a scope must clear its counts"
+        );
+
+        drop(dirs);
+    }
+
+    /// Bound for the pause handshake. This is a hang-guard, not a timing
+    /// dependency: under a working barrier `wait_until_paused` returns via the
+    /// `Notify` handshake far below this bound regardless of IO pressure, so the
+    /// assertions never depend on the value. It only turns a neutralized-barrier
+    /// hang into a deterministic failure instead of an infinite wait.
+    const BARRIER_PAUSE_GUARD: std::time::Duration = std::time::Duration::from_secs(10);
+
+    fn rename_barrier_fileinfos(object: &str, count: usize) -> Vec<FileInfo> {
+        (0..count).map(|_| metadata_test_fileinfo(object)).collect()
+    }
+
+    /// Demo / regression guard for the backlog#1325 rename fan-out pause barrier
+    /// and background-task introspection. Serves the barrier-style acceptance of
+    /// #1312 ("assert no background disk write remains after release").
+    ///
+    /// It drives the real `SetDisks::rename_data` fan-out: a barrier is armed at
+    /// the first disk's `rename` phase, and the test awaits that pause point,
+    /// asserts a background disk task is still in flight, releases it, and then
+    /// asserts the in-flight count drains to zero once the fan-out completes.
+    ///
+    /// Neutralizing the barrier seam (`rename_fanout_barrier` -> immediate no-op)
+    /// makes `wait_until_paused` never wake, so the guarded await elapses and the
+    /// test fails. Neutralizing the task guard (`rename_fanout_task_guard` ->
+    /// `()`) pins `running()` at zero, so the "still in flight" assertion fails.
+    #[tokio::test]
+    async fn rename_fanout_barrier_pauses_and_reports_running_background_tasks() {
+        const DISKS: usize = 4;
+        let bucket = "rename-barrier-bucket";
+        let object = "rename-barrier-object";
+        let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+        let file_infos = rename_barrier_fileinfos(object, DISKS);
+
+        let tracker = rename_fanout_barrier::observe_tasks(object);
+        assert_eq!(tracker.running(), 0, "no fan-out tasks before rename starts");
+
+        let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_RENAME);
+
+        // Run the real rename fan-out concurrently with the control flow so we can
+        // introspect while it is parked at the first disk's rename checkpoint.
+        let rename_fut = SetDisks::rename_data(&disks, bucket, object, &file_infos, bucket, object, DISKS - 1);
+        let control_fut = async {
+            tokio::time::timeout(BARRIER_PAUSE_GUARD, barrier.wait_until_paused())
+                .await
+                .expect("fan-out must reach the armed rename barrier");
+            assert!(barrier.is_paused(), "target task must be parked at the checkpoint");
+            assert!(tracker.running() >= 1, "a background rename task must still be in flight while paused");
+            barrier.release();
+        };
+        let (_rename_res, ()) = tokio::join!(rename_fut, control_fut);
+
+        // The fan-out has fully joined -> every task guard has dropped.
+        assert_eq!(tracker.running(), 0, "no background rename task may remain once the fan-out has drained");
+
+        drop(dirs);
+    }
+
+    /// Demo / regression guard for the barrier on the commit (old-data-dir)
+    /// cleanup fan-out. Serves the same #1312/#1319 "no background disk write
+    /// after release" shape, on the reclamation path that runs *after* a write is
+    /// ACKed — the classic detached background delete #1312 fences against.
+    ///
+    /// It stages a real `object/<old_data_dir>` on two disks and drives the real
+    /// `commit_rename_data_dir` fan-out, pausing the first disk's cleanup delete.
+    #[tokio::test]
+    async fn commit_cleanup_fanout_barrier_pauses_background_deletes() {
+        let bucket = "cleanup-barrier-bucket";
+        let object = "cleanup-barrier-object";
+        let old_data_dir = "11111111-1111-1111-1111-111111111111";
+        let committed_data_dir = "22222222-2222-2222-2222-222222222222";
+        let path = format!("{object}/{old_data_dir}/part.1");
+        let (_dir1, disk1) = read_multiple_test_disk(bucket, &[(&path, b"one".as_slice())]).await;
+        let (_dir2, disk2) = read_multiple_test_disk(bucket, &[(&path, b"two".as_slice())]).await;
+        let set = io_primitives_test_set(vec![Some(disk1.clone()), Some(disk2.clone())], 1).await;
+        let disks = [Some(disk1.clone()), Some(disk2.clone())];
+
+        let tracker = rename_fanout_barrier::observe_tasks(object);
+        let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_CLEANUP);
+
+        let cleanup_fut = set.commit_rename_data_dir(&disks, bucket, object, old_data_dir, committed_data_dir, 2);
+        let control_fut = async {
+            tokio::time::timeout(BARRIER_PAUSE_GUARD, barrier.wait_until_paused())
+                .await
+                .expect("cleanup fan-out must reach the armed cleanup barrier");
+            assert!(tracker.running() >= 1, "a background cleanup delete must still be in flight while paused");
+            barrier.release();
+        };
+        let (cleanup, ()) = tokio::join!(cleanup_fut, control_fut);
+
+        assert_eq!(tracker.running(), 0, "no background cleanup task may remain after drain");
+        assert_eq!(cleanup.attempted, 2);
+        assert_eq!(cleanup.reclaimed, 2, "the real old-data-dir must still be reclaimed after release");
+
+        drop((disk1, disk2));
+    }
+
+    /// Isolation guard: an armed barrier / observed object only affects its own
+    /// object. A fan-out for a different (unobserved, unarmed) object must not be
+    /// paused and must not accrue any tracked task count — so concurrent tests
+    /// using distinct object names never interfere.
+    #[tokio::test]
+    async fn barrier_and_task_tracker_isolate_by_object() {
+        const DISKS: usize = 3;
+        let bucket = "barrier-iso-bucket";
+        let observed_object = "iso-observed-object";
+        let other_object = "iso-other-object";
+        let (dirs, disks) = call_counter_local_disks(bucket, DISKS).await;
+
+        // Observe + arm one object, then run the fan-out for a *different* object.
+        let tracker = rename_fanout_barrier::observe_tasks(observed_object);
+        let _barrier = rename_fanout_barrier::arm(observed_object, 0, rename_fanout_barrier::PHASE_RENAME);
+
+        let file_infos = rename_barrier_fileinfos(other_object, DISKS);
+        // This must run to completion without ever pausing (no barrier for it) and
+        // must not touch the observed object's counter. A hang here (e.g. if the
+        // barrier ignored the object key) would surface as a timeout.
+        let _ = tokio::time::timeout(
+            BARRIER_PAUSE_GUARD,
+            SetDisks::rename_data(&disks, bucket, other_object, &file_infos, bucket, other_object, DISKS - 1),
+        )
+        .await
+        .expect("fan-out for an unarmed object must not pause");
+
+        assert_eq!(
+            tracker.running(),
+            0,
+            "another object's fan-out must not accrue tracked tasks for the observed object"
+        );
+
+        drop(dirs);
+    }
+
    #[tokio::test]
    async fn multipart_codec_streaming_reader_zero_buffer_is_noop() {
        let reader = tokio::io::BufReader::new(Cursor::new(b"payload".to_vec()));
@@ -140,7 +140,7 @@ use rustfs_lock::local_lock::LocalLock;
 use rustfs_lock::{FastLockGuard, LockManager, NamespaceLock, NamespaceLockGuard, NamespaceLockWrapper, ObjectKey};
 use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem, Infos};
 use rustfs_object_capacity::capacity_scope::{
-    CapacityScope, CapacityScopeDisk, record_capacity_scope, record_global_dirty_scope,
+    CapacityScope, CapacityScopeDisk, current_dirty_generation, record_capacity_scope, record_global_dirty_scope,
 };
 use rustfs_s3_types::EventName;
 use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
@@ -164,6 +164,7 @@ use std::hash::Hash;
 use std::mem::{self};
 use std::pin::Pin;
 use std::sync::OnceLock;
+use std::sync::atomic::{AtomicU64, Ordering};
 use std::task::{Context, Poll};
 use std::time::{Instant, SystemTime, UNIX_EPOCH};
 use std::{
@@ -349,10 +350,145 @@ fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -
        || metadata.contains_key(SSEC_KEY_MD5_HEADER)
 }

+/// Per-set memoized capacity dirty scope.
+///
+/// The disk endpoint/path identity of each slot in a set is immutable for the
+/// set's lifetime (heal replaces the [`DiskStore`] instance but keeps the same
+/// endpoint and root), so the dirty scope only needs to be built once instead
+/// of allocating a `String` per online disk on every successful write
+/// (backlog#1315). Slots are filled lazily as disks are observed online; once
+/// every slot has contributed, `complete` latches and the hot path returns the
+/// shared `Arc` without any scan.
+#[derive(Default, Debug)]
+struct CapacityScopeCache {
+    /// Per-slot resolved disk identity; `None` slots have not been seen online.
+    slots: Vec<Option<CapacityScopeDisk>>,
+    /// Deduplicated scope covering every slot resolved so far.
+    scope: Option<Arc<CapacityScope>>,
+    /// Latches once every slot is resolved; the scope is then stable.
+    complete: bool,
+}
+
+impl CapacityScopeCache {
+    /// Whether `disks` contains an online disk whose slot has not been recorded
+    /// yet. Read-only, allocation-free (used under the read lock).
+    fn has_unresolved_slot(&self, disks: &[Option<DiskStore>]) -> bool {
+        disks
+            .iter()
+            .enumerate()
+            .any(|(idx, disk)| disk.is_some() && self.slots.get(idx).is_none_or(|slot| slot.is_none()))
+    }
+}
+
+impl SetDisks {
+    /// Return the memoized dirty scope for this set, resolving any newly-online
+    /// slots first. Steady-state writes hit the fast path (a read lock and an
+    /// `Arc` clone) with no per-disk `String` allocation.
+    fn capacity_scope(&self, disks: &[Option<DiskStore>]) -> Arc<CapacityScope> {
+        {
+            let cache = self.capacity_scope_cache.read().unwrap_or_else(|p| p.into_inner());
+            if let Some(scope) = cache.scope.as_ref()
+                && (cache.complete || !cache.has_unresolved_slot(disks))
+            {
+                return scope.clone();
+            }
+        }
+        self.resolve_capacity_scope(disks)
+    }
+
+    /// Slow path: fill newly-observed slots and rebuild the deduplicated scope.
+    fn resolve_capacity_scope(&self, disks: &[Option<DiskStore>]) -> Arc<CapacityScope> {
+        let mut cache = self.capacity_scope_cache.write().unwrap_or_else(|p| p.into_inner());
+        if cache.slots.len() < self.set_drive_count {
+            cache.slots.resize(self.set_drive_count, None);
+        }
+
+        let mut changed = false;
+        for (idx, disk) in disks.iter().enumerate() {
+            let Some(slot) = cache.slots.get_mut(idx) else {
+                break;
+            };
+            if slot.is_some() {
+                continue;
+            }
+            if let Some(disk) = disk {
+                *slot = Some(CapacityScopeDisk {
+                    endpoint: disk.endpoint().to_string(),
+                    drive_path: disk.to_string(),
+                });
+                changed = true;
+            }
+        }
+
+        if changed || cache.scope.is_none() {
+            let mut unique = HashSet::with_capacity(cache.slots.len());
+            let mut scoped_disks = Vec::with_capacity(cache.slots.len());
+            for slot in cache.slots.iter().flatten() {
+                if unique.insert(slot.clone()) {
+                    scoped_disks.push(slot.clone());
+                }
+            }
+            cache.complete = !cache.slots.is_empty() && cache.slots.iter().all(|slot| slot.is_some());
+            cache.scope = Some(Arc::new(CapacityScope { disks: scoped_disks }));
+        }
+
+        cache.scope.clone().unwrap_or_else(|| Arc::new(CapacityScope::default()))
+    }
+
+    /// Record the set's dirty scope after a successful write.
+    ///
+    /// The global dirty registry is only upgraded on the first write of each
+    /// registry generation: once this set has marked its disks, subsequent
+    /// writes skip the global mutex until a refresh drain advances the
+    /// generation, forcing a re-mark (backlog#1315). The scope-token registry
+    /// (multipart completion) still records per token so the app-side write
+    /// settle can consume it.
+    fn record_capacity_scope_if_needed(&self, scope_token: Option<Uuid>, disks: &[Option<DiskStore>]) {
+        let scope = self.capacity_scope(disks);
+        if scope.disks.is_empty() {
+            return;
+        }
+
+        let generation = current_dirty_generation();
+        if self.capacity_dirty_generation.load(Ordering::Acquire) != generation {
+            // First write of this generation (or after a drain): upgrade the
+            // global registry and cache the generation observed under its lock.
+            let observed = record_global_dirty_scope((*scope).clone());
+            self.capacity_dirty_generation.store(observed, Ordering::Release);
+        }
+
+        if let Some(token) = scope_token {
+            record_capacity_scope(token, (*scope).clone());
+        }
+    }
+
+    /// Mark healed disks dirty from the (infrequent) heal path.
+    ///
+    /// Heal passes disks in erasure-distribution order (`shuffle_disks`), not
+    /// physical-slot order, so they must not seed the slot-indexed memo — doing
+    /// so could record a disk under the wrong slot and drop another from the
+    /// steady-state scope. Heal therefore builds an ad-hoc scope from the disks
+    /// it actually rewrote and marks the global registry directly. This runs at
+    /// heal frequency, so it does not use the per-generation skip fast-path
+    /// (backlog#1315).
+    fn record_healed_capacity_scope(&self, disks: &[Option<DiskStore>]) {
+        let scope = capacity_scope_from_disks(disks);
+        if scope.disks.is_empty() {
+            return;
+        }
+        // Do not advance the set's generation marker here: this ad-hoc scope is
+        // only the subset of disks heal rewrote, whereas the marker asserts the
+        // full set was marked. Leaving the marker untouched keeps the next write
+        // free to upgrade the full-set scope if it has not been marked yet.
+        let _ = record_global_dirty_scope(scope);
+    }
+}
+
+/// Build an ad-hoc, deduplicated dirty scope from `disks`. Used by the heal
+/// path where disks are not in physical-slot order (backlog#1315).
 fn capacity_scope_from_disks(disks: &[Option<DiskStore>]) -> CapacityScope {
    let mut unique = HashSet::with_capacity(disks.len());
    let mut scoped_disks = Vec::with_capacity(disks.len());
-
    for disk in disks.iter().flatten() {
        let scope_disk = CapacityScopeDisk {
            endpoint: disk.endpoint().to_string(),
@@ -362,23 +498,9 @@ fn capacity_scope_from_disks(disks: &[Option<DiskStore>]) -> CapacityScope {
            scoped_disks.push(scope_disk);
        }
    }
-
    CapacityScope { disks: scoped_disks }
 }

-fn record_capacity_scope_if_needed(scope_token: Option<Uuid>, disks: &[Option<DiskStore>]) {
-    let scope = capacity_scope_from_disks(disks);
-    if scope.disks.is_empty() {
-        return;
-    }
-
-    record_global_dirty_scope(scope.clone());
-
-    if let Some(token) = scope_token {
-        record_capacity_scope(token, scope);
-    }
-}
-
 /// Get the duplex buffer size from environment variable or use default.
 ///
 /// This function reads `RUSTFS_DUPLEX_BUFFER_SIZE` environment variable
@@ -1628,6 +1750,15 @@ pub struct SetDisks {
    /// Slice 3 sources `local_lock_manager` from this context to give each
    /// instance its own lock namespace.
    ctx: Arc<InstanceContext>,
+    /// Memoized capacity dirty scope so successful writes reuse a prebuilt
+    /// `Arc` instead of allocating a `String` per online disk (backlog#1315).
+    /// `Arc` so clones of a set share one memo.
+    capacity_scope_cache: Arc<std::sync::RwLock<CapacityScopeCache>>,
+    /// Last global dirty-registry generation this set marked. `u64::MAX` until
+    /// the first write; equality with the current generation lets steady-state
+    /// writes skip the global registry mutex (backlog#1315). `Arc` so clones of
+    /// a set share one generation marker.
+    capacity_dirty_generation: Arc<AtomicU64>,
 }

 #[derive(Clone, Debug, Eq, Hash, PartialEq)]
@@ -1828,6 +1959,8 @@ impl SetDisks {
            // process lock-manager singleton, so this is unchanged.
            local_lock_manager: ctx.lock_manager(),
            ctx,
+            capacity_scope_cache: Arc::new(std::sync::RwLock::new(CapacityScopeCache::default())),
+            capacity_dirty_generation: Arc::new(AtomicU64::new(u64::MAX)),
        })
    }

@@ -3518,6 +3651,13 @@ fn complete_part_checksum(part: &CompletePart, checksum_type: rustfs_rio::Checks
        rustfs_rio::ChecksumType::CRC32 => Some(part.checksum_crc32.clone()),
        rustfs_rio::ChecksumType::CRC32C => Some(part.checksum_crc32c.clone()),
        rustfs_rio::ChecksumType::CRC64_NVME => Some(part.checksum_crc64nvme.clone()),
+        // XXHash3/64/128 and SHA-512 (AWS 2026-04): s3s CompletePart has no typed
+        // field to carry a client-supplied per-part value in the
+        // CompleteMultipartUpload request, so accept the type with no double-check
+        // value (the part was already verified server-side at UploadPart). This
+        // mirrors the missing-value path of the five typed algorithms. Reject only
+        // genuinely unset/invalid types.
+        base if base.is_set() => Some(None),
        _ => None,
    }
 }
@@ -4621,6 +4761,109 @@ mod tests {
        .await
    }

+    // backlog#1315: the memoized dirty scope must be byte-for-byte identical to
+    // the ad-hoc scope the previous per-write construction produced, otherwise
+    // dirty-disk keys diverge from the disk-cache keys and capacity counts drift.
+    #[tokio::test]
+    #[serial]
+    async fn capacity_scope_memo_matches_adhoc_and_is_reused() {
+        use rustfs_object_capacity::capacity_scope::drain_global_dirty_scopes;
+
+        let (dir_a, disk_a) = make_single_local_disk().await;
+        let (dir_b, disk_b) = make_single_local_disk().await;
+        let disks = vec![Some(disk_a), Some(disk_b)];
+        let set = make_set_disks_with(disks.clone()).await;
+
+        let expected = capacity_scope_from_disks(&disks);
+        let first = set.capacity_scope(&disks);
+        assert_eq!(*first, expected, "memoized scope must equal the ad-hoc per-disk construction bit-for-bit");
+
+        // Second call on a fully-resolved set returns the very same Arc (no new
+        // allocation): proving steady-state writes do not rebuild String/HashSet.
+        let second = set.capacity_scope(&disks);
+        assert!(Arc::ptr_eq(&first, &second), "steady-state scope must reuse the cached Arc");
+
+        let _ = drain_global_dirty_scopes();
+        drop((dir_a, dir_b));
+    }
+
+    // backlog#1315: the global registry mutex must be upgraded only on the first
+    // write of each generation; steady-state writes skip it. Reverting the
+    // generation skip makes the upgrade count grow per write and fails this test.
+    #[tokio::test]
+    #[serial]
+    async fn record_capacity_scope_upgrades_registry_once_per_generation() {
+        use rustfs_object_capacity::capacity_scope::{drain_global_dirty_scopes, global_dirty_upgrade_count};
+
+        let (dir_a, disk_a) = make_single_local_disk().await;
+        let (dir_b, disk_b) = make_single_local_disk().await;
+        let disks = vec![Some(disk_a), Some(disk_b)];
+        let set = make_set_disks_with(disks.clone()).await;
+
+        // Start from a clean registry generation.
+        let _ = drain_global_dirty_scopes();
+        let before = global_dirty_upgrade_count();
+
+        // First write of this generation upgrades the registry exactly once.
+        set.record_capacity_scope_if_needed(None, &disks);
+        assert_eq!(
+            global_dirty_upgrade_count(),
+            before + 1,
+            "first write of a generation must upgrade the global registry"
+        );
+
+        // Subsequent writes in the same generation must not touch the mutex.
+        for _ in 0..16 {
+            set.record_capacity_scope_if_needed(None, &disks);
+        }
+        assert_eq!(
+            global_dirty_upgrade_count(),
+            before + 1,
+            "steady-state writes must reuse the generation mark, not re-upgrade"
+        );
+
+        // A drain advances the generation; the next write must re-mark so the
+        // disks it wrote are captured by the following refresh (no lost update).
+        let drained = drain_global_dirty_scopes();
+        assert_eq!(drained.len(), 2, "both set disks must have been recorded dirty");
+        set.record_capacity_scope_if_needed(None, &disks);
+        assert_eq!(
+            global_dirty_upgrade_count(),
+            before + 2,
+            "the first write after a drain must re-upgrade the registry"
+        );
+
+        let _ = drain_global_dirty_scopes();
+        drop((dir_a, dir_b));
+    }
+
+    // backlog#1315: an offline slot must not force the per-write slow path, and
+    // the resolved scope must still cover every online disk.
+    #[tokio::test]
+    #[serial]
+    async fn capacity_scope_tolerates_offline_slot_without_reallocating() {
+        use rustfs_object_capacity::capacity_scope::drain_global_dirty_scopes;
+
+        let (dir_a, disk_a) = make_single_local_disk().await;
+        // Slot 1 is permanently offline (None); the set never resolves it.
+        let disks = vec![Some(disk_a), None];
+        let set = make_set_disks_with(disks.clone()).await;
+
+        let first = set.capacity_scope(&disks);
+        assert_eq!(first.disks.len(), 1, "only the online disk contributes to the scope");
+        // Even though the set is not "complete" (slot 1 unresolved), repeated
+        // writes with the same online set reuse the cached Arc via the
+        // no-unresolved-slot fast path.
+        let second = set.capacity_scope(&disks);
+        assert!(
+            Arc::ptr_eq(&first, &second),
+            "a stable online subset must reuse the cached Arc even with an offline slot"
+        );
+
+        let _ = drain_global_dirty_scopes();
+        drop(dir_a);
+    }
+
    // issue #4189: an orphan directory tree (empty dirs, no xl.meta) must be purged.
    #[tokio::test]
    async fn purge_orphan_dir_object_removes_empty_tree() {
@@ -6281,33 +6524,107 @@ mod tests {
        signature
    }

+    // backlog#1321: `classify_rename_convergence` is the single decision that
+    // replaced the old `Option<Vec<u8>>::is_some()` heal gate. These are the
+    // revert-fails white-box cases behind the issue acceptance matrix — if the
+    // decision regressed to "a signature exists => needs heal", the healthy
+    // 4/4 and 8/8 cases below would flip from `AllSuccessIdentical` to a
+    // heal-worthy variant and fail.
    #[test]
-    fn test_reduce_common_versions_requires_write_quorum() {
-        let common = rename_versions_signature(1, 1);
-        let other = rename_versions_signature(2, 1);
+    fn test_classify_rename_convergence_healthy_identical_needs_no_heal() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        let sig = rename_versions_signature(1, 3);

-        let disk_versions = vec![Some(common.clone()), Some(common.clone()), Some(other)];
-        let result = SetDisks::reduce_common_versions(&disk_versions, 2);
-        assert_eq!(result, Some(common));
+        // Healthy 4/4: every disk committed with an identical, known signature.
+        let disk_versions = vec![Some(sig.clone()), Some(sig.clone()), Some(sig.clone()), Some(sig.clone())];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::AllSuccessIdentical);
+        assert!(!convergence.needs_heal(), "a fully converged healthy MPU must not enqueue heal");

-        let split_versions = vec![
-            Some(rename_versions_signature(1, 1)),
-            Some(rename_versions_signature(2, 1)),
-            None,
-        ];
-        let result = SetDisks::reduce_common_versions(&split_versions, 2);
-        assert_eq!(result, None);
+        // Healthy 8/8: same property at a wider set width.
+        let disk_versions = vec![Some(sig); 8];
+        let errs = vec![None; 8];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::AllSuccessIdentical);
+        assert!(!convergence.needs_heal());
    }

    #[test]
-    fn test_select_rename_data_versions_keeps_longer_success_disparity() {
-        let common = rename_versions_signature(1, 1);
-        let longer = rename_versions_signature(2, 2);
-        let disk_versions = vec![Some(common.clone()), Some(common), Some(longer.clone())];
-        let errs = vec![None, None, None];
+    fn test_classify_rename_convergence_three_agree_one_diverges_heals() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        let common = rename_versions_signature(1, 2);
+        let odd = rename_versions_signature(2, 2);

-        let result = SetDisks::select_rename_data_versions(&disk_versions, &errs, 2);
-        assert_eq!(result, Some(longer));
+        // 3-same, 1-divergent, all committed: reconcile the odd replica.
+        let disk_versions = vec![Some(common.clone()), Some(common.clone()), Some(common), Some(odd)];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::SignatureDivergent);
+        assert!(convergence.needs_heal());
+    }
+
+    #[test]
+    fn test_classify_rename_convergence_failed_or_offline_disk_heals() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        let sig = rename_versions_signature(1, 2);
+
+        // 1 disk failed/offline while the rest committed identically: past the
+        // write-quorum gate this is a `PartialCommit` — a replica is missing.
+        let disk_versions = vec![Some(sig.clone()), Some(sig.clone()), Some(sig), None];
+        let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::PartialCommit);
+        assert!(convergence.needs_heal());
+    }
+
+    #[test]
+    fn test_classify_rename_convergence_no_common_quorum_heals() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        // No signature holds a majority (2/2 split), every disk committed:
+        // divergence with no common quorum still reconciles via heal.
+        let a = rename_versions_signature(1, 1);
+        let b = rename_versions_signature(2, 1);
+        let disk_versions = vec![Some(a.clone()), Some(a), Some(b.clone()), Some(b)];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::SignatureDivergent);
+        assert!(convergence.needs_heal());
+    }
+
+    #[test]
+    fn test_classify_rename_convergence_over_ten_versions_by_success() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        // >10 versions: every disk deliberately omits the signature (`None`).
+        // All committed => `Unknown` => scanner-backstopped, no self-enqueue.
+        let disk_versions = vec![None, None, None, None];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::Unknown);
+        assert!(
+            !convergence.needs_heal(),
+            ">10-version healthy commit relies on the scanner, not self-enqueue"
+        );
+
+        // Same >10-version shape but with a failed disk: a failure is
+        // conservative-heal regardless of signatures being unavailable.
+        let errs = vec![None, None, None, Some(DiskError::FileCorrupt)];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::PartialCommit);
+        assert!(convergence.needs_heal());
+    }
+
+    #[test]
+    fn test_classify_rename_convergence_mixed_signed_unsigned_heals() {
+        use crate::set_disk::core::io_primitives::RenameConvergence;
+        // A committed replica with <=10 versions (signed) alongside one with
+        // >10 versions (unsigned) is itself a version-count divergence.
+        let sig = rename_versions_signature(1, 2);
+        let disk_versions = vec![Some(sig.clone()), Some(sig.clone()), Some(sig), None];
+        let errs = vec![None, None, None, None];
+        let convergence = SetDisks::classify_rename_convergence(&disk_versions, &errs);
+        assert_eq!(convergence, RenameConvergence::SignatureDivergent);
+        assert!(convergence.needs_heal());
    }

    #[test]
@@ -6718,6 +7035,23 @@ mod tests {
            ..Default::default()
        };
        assert_eq!(complete_part_checksum(&part, full_object_crc32), Some(Some("AAAAAA==".to_string())));
+
+        // AWS 2026-04 additional algorithms have no CompletePart field, so they are
+        // accepted with no double-check value (verified at UploadPart) — Some(None),
+        // NOT the outer None that would fail the multipart completion (#1261).
+        for ct in [
+            rustfs_rio::ChecksumType::XXHASH3,
+            rustfs_rio::ChecksumType::XXHASH64,
+            rustfs_rio::ChecksumType::XXHASH128,
+            rustfs_rio::ChecksumType::SHA512,
+            rustfs_rio::ChecksumType::MD5,
+        ] {
+            assert_eq!(complete_part_checksum(&CompletePart::default(), ct), Some(None), "{ct:?}");
+        }
+
+        // Genuinely unset / invalid types must still be rejected (outer None).
+        assert_eq!(complete_part_checksum(&CompletePart::default(), rustfs_rio::ChecksumType::NONE), None);
+        assert_eq!(complete_part_checksum(&CompletePart::default(), rustfs_rio::ChecksumType::INVALID), None);
    }

    fn direct_memory_test_metadata(size: i64) -> (ObjectInfo, FileInfo, ObjectOptions) {
@@ -682,7 +682,7 @@ impl SetDisks {
                            ));
                        }

-                        record_capacity_scope_if_needed(None, &out_dated_disks);
+                        self.record_healed_capacity_scope(&out_dated_disks);

                        // The object is healthy here; sweep any data dirs left behind
                        // by pre-#3510 unversioned overwrites, which the dangling paths
@@ -21,6 +21,7 @@
 //! unchanged; method bodies are moved verbatim and runtime behavior is the same.

 use super::super::*;
+use crate::crash_inject::{self, CrashPoint};
 use std::future::Future;
 use std::time::Duration;
 use tokio::task::JoinSet;
@@ -1420,10 +1421,19 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {

        let complete_tail_stage_start = rustfs_io_metrics::put_stage_metrics_enabled().then(Instant::now);

+        // Crash-consistency injection: hard power loss after the upload is fully
+        // staged and locked but before the authoritative rename_data commit. No
+        // disk has moved the staged data, so a crash here must leave any prior
+        // committed version byte-for-byte intact (rustfs/backlog#864) and the
+        // upload fully retryable. Compiles to a no-op outside `#[cfg(test)]`.
+        if crash_inject::should_crash_at(CrashPoint::MultipartBeforeCommitRename, object) {
+            return Err(StorageError::Unexpected);
+        }
+
        // The trailing `_` drops the rename_data old-size backfill
        // (rustfs/backlog#1009): CompleteMultipartUpload keeps its pre-commit
        // `get_object_info` lookup, so the backfill has no consumer here yet.
-        let (online_disks, versions, op_old_dir, cleanup_disks, _) = Self::rename_data(
+        let (online_disks, convergence, op_old_dir, cleanup_disks, _) = Self::rename_data(
            &shuffle_disks,
            RUSTFS_META_MULTIPART_BUCKET,
            &upload_id_path,
@@ -1434,6 +1444,16 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
        )
        .await?;

+        // Crash-consistency injection: hard power loss after the authoritative
+        // rename_data commit succeeded but before the stale part.N.meta cleanup.
+        // The new version is durably committed and visible, so a crash here must
+        // leave the object readable as the new version; the un-reclaimed staging
+        // parts are swept by a retried completion or upload GC (rustfs/backlog#946).
+        // Compiles to a no-op outside `#[cfg(test)]`.
+        if crash_inject::should_crash_at(CrashPoint::MultipartAfterCommitBeforePartsCleanup, object) {
+            return Err(StorageError::Unexpected);
+        }
+
        // backlog#946: reclaim the stale per-part metadata (and any superfluous
        // part.N data files no longer in the completed set) only *after* the
        // authoritative rename_data commit above has succeeded. If rename_data
@@ -1465,17 +1485,46 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {

        drop(object_lock_guard); // drop object lock guard to release the lock

-        if let Some(versions) = versions {
-            let _ =
-                rustfs_common::heal_channel::send_heal_request(rustfs_common::heal_channel::create_heal_request_with_options(
-                    bucket.to_string(),
-                    Some(object.to_string()),
-                    false,
-                    Some(HealChannelPriority::Normal),
-                    Some(self.pool_index),
-                    Some(self.set_index),
-                ))
+        // backlog#1321: enqueue heal only when the committed replicas actually
+        // need to converge — a partial commit (some disk failed/offline) or a
+        // signature divergence between committed replicas. A fully healthy MPU
+        // (identical signatures on every disk) is `AllSuccessIdentical` and
+        // submits nothing, which is the fix: the old `Option::is_some()` gate
+        // treated the mere existence of a version signature as "needs heal", so
+        // every healthy <=10-version completion self-enqueued.
+        //
+        // The submit is detached (`tokio::spawn`) so it stays off the ACK
+        // critical path AND survives cancellation of the completion future: the
+        // write is already durable and ACK-worthy, so the heal admission must
+        // not ride the client's request lifetime. The admission itself is
+        // bounded / deduplicated / observable (`send_heal_request` ->
+        // `HealAdmissionResult`), so this emits at most one submit per
+        // completion and coalesces with any in-flight heal for the same object.
+        //
+        // Scanner backstop (backlog#1321 patch): a `PartialCommit` whose
+        // completion is cancelled in the narrow window after the durable commit
+        // but before this spawn runs is not lost — the divergence it would have
+        // healed is exactly what the background scanner reconciles. `Unknown`
+        // (>10 versions, no signature produced) likewise relies on the scanner
+        // rather than self-enqueuing.
+        if convergence.needs_heal() {
+            let bucket = bucket.to_string();
+            let object = object.to_string();
+            let pool_index = self.pool_index;
+            let set_index = self.set_index;
+            tokio::spawn(async move {
+                let _ = rustfs_common::heal_channel::send_heal_request(
+                    rustfs_common::heal_channel::create_heal_request_with_options(
+                        bucket,
+                        Some(object),
+                        false,
+                        Some(HealChannelPriority::Normal),
+                        Some(pool_index),
+                        Some(set_index),
+                    ),
+                )
                .await;
+            });
        }

        let upload_id_path = upload_id_path.clone();
@@ -1493,7 +1542,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
            }
        }

-        record_capacity_scope_if_needed(opts.capacity_scope_token, &online_disks);
+        self.record_capacity_scope_if_needed(opts.capacity_scope_token, &online_disks);

        fi.is_latest = true;

@@ -2125,4 +2174,222 @@ mod tests {
        let paged: Vec<&str> = first.iter().chain(second.iter()).map(|u| u.upload_id.as_str()).collect();
        assert_eq!(paged, vec!["u0", "u1", "u2", "u3"]);
    }
+
+    /// Crash-consistency for the two `complete_multipart_upload` commit windows.
+    ///
+    /// rustfs/backlog#864: a fault that interrupts a completion must never mutate
+    /// an already-committed object; the object must read back as the whole old
+    /// version or the whole new version, never a torn mix, and the upload must
+    /// stay recoverable.
+    ///
+    /// [`CrashPoint::MultipartBeforeCommitRename`] fires after the upload is fully
+    /// staged and locked but before the authoritative `rename_data` commit: a
+    /// prior committed version survives byte-for-byte and the upload is retryable.
+    /// [`CrashPoint::MultipartAfterCommitBeforePartsCleanup`] fires after the
+    /// commit lands but before the stale `part.N.meta` cleanup: the new version is
+    /// readable and the un-reclaimed staging is harmless and reclaimable.
+    mod crash_consistency {
+        use super::*;
+        use crate::crash_inject::{self, CrashPoint};
+        use crate::storage_api_contracts::object::ObjectIO as _;
+        use http::HeaderMap;
+        use tokio::io::AsyncReadExt as _;
+
+        /// 1 MiB keeps every object off the 128 KiB inline fast path, so the
+        /// commit moves real erasure shards through `rename_data`.
+        const PART_SIZE: usize = 1 << 20;
+
+        fn payload(fill: u8) -> Vec<u8> {
+            vec![fill; PART_SIZE]
+        }
+
+        async fn make_bucket_on_all(disks: &[DiskStore], bucket: &str) {
+            for disk in disks {
+                disk.make_volume(bucket).await.expect("bucket volume should be created");
+            }
+        }
+
+        /// Stage a single-part multipart upload without completing it. A single
+        /// part is the last part, so the 5 MiB minimum-part-size gate does not
+        /// apply. Returns the upload id and the `CompletePart` list.
+        async fn stage_upload(
+            set_disks: &Arc<SetDisks>,
+            bucket: &str,
+            object: &str,
+            content: &[u8],
+        ) -> (String, Vec<CompletePart>) {
+            let upload = set_disks
+                .new_multipart_upload(bucket, object, &ObjectOptions::default())
+                .await
+                .expect("multipart upload should be created");
+            let mut reader = PutObjReader::new(
+                HashReader::from_stream(
+                    Cursor::new(content.to_vec()),
+                    content.len() as i64,
+                    content.len() as i64,
+                    None,
+                    None,
+                    false,
+                )
+                .expect("hash reader should be constructed"),
+            );
+            let part = set_disks
+                .put_object_part(bucket, object, &upload.upload_id, 1, &mut reader, &ObjectOptions::default())
+                .await
+                .expect("uploading the part should succeed");
+            (
+                upload.upload_id,
+                vec![CompletePart {
+                    part_num: part.part_num,
+                    etag: part.etag,
+                    ..Default::default()
+                }],
+            )
+        }
+
+        async fn complete(
+            set_disks: &Arc<SetDisks>,
+            bucket: &str,
+            object: &str,
+            upload_id: &str,
+            parts: Vec<CompletePart>,
+        ) -> Result<ObjectInfo> {
+            set_disks
+                .clone()
+                .complete_multipart_upload(bucket, object, upload_id, parts, &ObjectOptions::default())
+                .await
+        }
+
+        /// Read the whole committed object back; returns `(body, etag)`. The full
+        /// body read also proves the served length matches (never a torn/short body).
+        async fn read_object(set_disks: &Arc<SetDisks>, bucket: &str, object: &str) -> (Vec<u8>, Option<String>) {
+            let mut reader = set_disks
+                .get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
+                .await
+                .expect("object reader should open");
+            let etag = reader.object_info.etag.clone();
+            let mut body = Vec::new();
+            reader
+                .stream
+                .read_to_end(&mut body)
+                .await
+                .expect("object should stream fully");
+            (body, etag)
+        }
+
+        async fn upload_is_listed(set_disks: &Arc<SetDisks>, bucket: &str, object: &str, upload_id: &str) -> bool {
+            let page = set_disks
+                .list_multipart_uploads(bucket, object, None, None, None, 1000)
+                .await
+                .expect("listing multipart uploads should succeed");
+            page.uploads.iter().any(|u| u.upload_id == upload_id)
+        }
+
+        #[tokio::test]
+        #[serial]
+        async fn pre_commit_crash_keeps_committed_old_version() {
+            let (temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
+            let bucket = "multipart-crash-pre";
+            let object = "crash-pre-object";
+            make_bucket_on_all(&disk_stores, bucket).await;
+
+            // Commit an OLD version through the multipart path.
+            let old = payload(0xA1);
+            let (u_old, parts_old) = stage_upload(&set_disks, bucket, object, &old).await;
+            complete(&set_disks, bucket, object, &u_old, parts_old)
+                .await
+                .expect("the old version should commit");
+            let (old_body, old_etag) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(old_body, old, "the old version must round-trip before the crash");
+
+            // Stage a replacement upload with NEW content, then crash before commit.
+            let new = payload(0xB2);
+            let (u_new, parts_new) = stage_upload(&set_disks, bucket, object, &new).await;
+            crash_inject::arm(CrashPoint::MultipartBeforeCommitRename, object);
+            let crashed = complete(&set_disks, bucket, object, &u_new, parts_new.clone()).await;
+            assert!(
+                matches!(crashed, Err(StorageError::Unexpected)),
+                "the armed pre-commit crash point must be the failure that surfaced, got {crashed:?}"
+            );
+
+            // rustfs/backlog#864: the committed old version is untouched, byte-for-byte.
+            let (after_body, after_etag) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(after_body, old, "the interrupted completion must not mutate the committed old version");
+            assert_eq!(after_etag, old_etag, "the old version's ETag must be unchanged");
+
+            // The staged upload survived intact (part.N.meta present on quorum), so
+            // a retried completion commits the new version cleanly (retryable).
+            assert!(
+                total_part1_meta(&temp_dirs).await > 0,
+                "the un-committed upload must keep its staging parts for retry"
+            );
+            crash_inject::disarm(CrashPoint::MultipartBeforeCommitRename, object);
+            complete(&set_disks, bucket, object, &u_new, parts_new)
+                .await
+                .expect("a retried completion must succeed");
+            let (retry_body, retry_etag) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(retry_body, new, "the retried completion must publish the whole new version");
+            assert_ne!(retry_etag, old_etag, "the new version must carry a distinct ETag");
+        }
+
+        #[tokio::test]
+        #[serial]
+        async fn post_commit_crash_publishes_new_version_and_leaves_reclaimable_upload() {
+            let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
+            let bucket = "multipart-crash-post";
+            let object = "crash-post-object";
+            make_bucket_on_all(&disk_stores, bucket).await;
+
+            // Stage an upload and crash AFTER the commit rename but BEFORE cleanup.
+            let new = payload(0xC3);
+            let (u_new, parts_new) = stage_upload(&set_disks, bucket, object, &new).await;
+            let parts_retry = parts_new.clone();
+            crash_inject::arm(CrashPoint::MultipartAfterCommitBeforePartsCleanup, object);
+            let crashed = complete(&set_disks, bucket, object, &u_new, parts_new).await;
+            assert!(
+                matches!(crashed, Err(StorageError::Unexpected)),
+                "the armed post-commit crash point must be the failure that surfaced, got {crashed:?}"
+            );
+            crash_inject::disarm(CrashPoint::MultipartAfterCommitBeforePartsCleanup, object);
+
+            // The commit landed: the new version reads back whole and correct.
+            let (body, _etag) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(body, new, "a post-commit crash must leave the whole new version readable");
+
+            // The completion never reached its cleanup, so the upload's staging
+            // directory is still listable — leftover that must be reclaimable.
+            assert!(
+                upload_is_listed(&set_disks, bucket, object, &u_new).await,
+                "the post-commit crash must leave the upload listable for reclamation"
+            );
+
+            // A retried CompleteMultipartUpload is answered deterministically: the
+            // commit rename already consumed the upload's metadata, so the retry
+            // resolves to InvalidUploadID (NoSuchUpload to the S3 client, the
+            // standard answer for a completed-then-retried upload) — never a torn
+            // state, and the committed object is untouched by the retry.
+            let retried = complete(&set_disks, bucket, object, &u_new, parts_retry).await;
+            assert!(
+                matches!(retried, Err(StorageError::InvalidUploadID(..))),
+                "a retried complete after the commit landed must resolve to InvalidUploadID, got {retried:?}"
+            );
+            let (body_after_retry, _) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(body_after_retry, new, "the failed retry must not disturb the committed object");
+
+            // Reclaim the leftover exactly as the production tail does: delete_all
+            // on the upload path (abort_multipart_upload cannot — the upload's
+            // metadata is gone, so it too answers InvalidUploadID).
+            let upload_dir = SetDisks::get_upload_id_dir(bucket, object, &u_new);
+            set_disks
+                .delete_all(RUSTFS_META_MULTIPART_BUCKET, &upload_dir)
+                .await
+                .expect("sweeping the leftover upload staging should succeed");
+            assert!(
+                !upload_is_listed(&set_disks, bucket, object, &u_new).await,
+                "reclaiming the upload must drop it from the listing"
+            );
+            let (body_after, _) = read_object(&set_disks, bucket, object).await;
+            assert_eq!(body_after, new, "reclaiming the leftover upload must not disturb the committed object");
+        }
+    }
 }
@@ -1076,7 +1076,7 @@ impl SetDisks {
            if fi.is_compressed() {
                record_compression_total_memory(actual_size as u64, w_size as u64).await;
            }
-            record_capacity_scope_if_needed(opts.capacity_scope_token, &online_disks);
+            self.record_capacity_scope_if_needed(opts.capacity_scope_token, &online_disks);

            fi.replication_state_internal = Some(replication_state_to_filemeta(&opts.put_replication_state()));

@@ -1801,7 +1801,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
            }
        }

-        record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
+        self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);

        let mut rollback_futures = Vec::new();
        for fi_vers in &vers {
@@ -1997,7 +1997,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                .map_err(|e| to_object_err(e, vec![bucket, object]))?;

            let disks = self.disk_inventory().await;
-            record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
+            self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);

            let mut oi = ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended);
            oi.replication_decision = goi.replication_decision;
@@ -2027,7 +2027,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
            .map_err(|e| to_object_err(e, vec![bucket, object]))?;

        let disks = self.disk_inventory().await;
-        record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
+        self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);

        let mut obj_info = ObjectInfo::from_file_info(&dfi, bucket, object, opts.versioned || opts.version_suspended);
        obj_info.size = goi.size;
@@ -2326,7 +2326,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                "transition completed on remote tier but source cleanup failed; skipping external lifecycle transition notification"
            );
        } else {
-            record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
+            self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
        }

        for disk in disks.iter() {
@@ -2384,7 +2384,16 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
        let (actual_fi, _, _) = fi?;

        oi = ObjectInfo::from_file_info(&actual_fi, bucket, object, opts.versioned || opts.version_suspended);
-        let ropts = put_restore_opts(bucket, object, &opts.transition.restore_request, &oi).await?;
+        let mut ropts = put_restore_opts(bucket, object, &opts.transition.restore_request, &oi).await?;
+        // The restore copy-back re-writes this same object via put_object /
+        // new_multipart_upload / complete_multipart_upload, each of which takes
+        // the object write lock in its commit phase. The caller
+        // (handle_restore_transitioned_object, #4877) already holds that write
+        // lock for the whole restore and forwards no_lock=true, so the inner
+        // writes must inherit it or they self-deadlock on the lock we already
+        // hold and time out. put_restore_opts builds fresh options that default
+        // no_lock=false, so propagate it explicitly here.
+        ropts.no_lock = opts.no_lock;
        if oi.parts.len() == 1 {
            let mut opts = opts.clone();
            opts.part_number = Some(1);
@@ -2502,6 +2511,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                uploaded_parts,
                &ObjectOptions {
                    mod_time: oi.mod_time,
+                    // Inherit the restore write lock (see ropts.no_lock above):
+                    // the commit phase re-acquires this object's write lock.
+                    no_lock: opts.no_lock,
                    ..Default::default()
                },
            )
@@ -4095,6 +4095,27 @@ mod tests {
        assert_eq!(&out[..n], [b"aaaa", b"bbbb", b"cccc", b"dddd"][fallback_index]);
    }

+    #[tokio::test]
+    async fn bitrot_reader_setup_skips_deferred_slots_without_a_source() {
+        let setup = setup_inline_bitrot_readers_with_env(
+            vec![Some(b"aaaa"), Some(b"bbbb"), None, Some(b"dddd")],
+            2,
+            2,
+            BitrotReaderSetupMode::ReadQuorum,
+            true,
+        )
+        .await;
+
+        assert!(setup.has_setup_quorum(2, 2, BitrotReaderSetupMode::ReadQuorum));
+        assert_eq!(setup.available_shards(), 2);
+        assert_eq!(setup.deferred_shards(), 1);
+        assert!(setup.readers[2].is_none(), "a slot without inline data or a disk has no usable source");
+        assert!(matches!(setup.errors[2], Some(DiskError::DiskNotFound)));
+        assert!(setup.deferred_stripe_handles[2].is_none());
+        assert!(setup.readers[3].is_some(), "an inline shard remains available as a deferred fallback");
+        assert!(setup.deferred_stripe_handles[3].is_some());
+    }
+
    #[tokio::test]
    async fn bitrot_reader_setup_preference_uses_data_blocks_first_without_env() {
        let setup = setup_inline_bitrot_readers_with_preference(
@@ -1308,11 +1308,19 @@ impl ECStore {
        opts: &ObjectOptions,
    ) -> Result<()> {
        let object = encode_dir_object(object);
+        let mut opts = opts.clone();
+        let _object_lock_guard = self
+            .acquire_object_write_lock_if_needed("restore_transitioned_object", bucket, &object, &mut opts)
+            .await?;
+
        if self.single_pool() {
-            return self.pools[0].clone().restore_transitioned_object(bucket, &object, opts).await;
+            return self.pools[0]
+                .clone()
+                .restore_transitioned_object(bucket, &object, &opts)
+                .await;
        }

-        let opts = transition_restore_pool_opts(opts);
+        let opts = transition_restore_pool_opts(&opts);
        let (_, idx) = self
            .get_latest_accessible_object_info_with_idx(bucket, object.as_str(), &opts)
            .await?;
@@ -2171,6 +2179,31 @@ mod tests {
        );
    }

+    #[tokio::test]
+    #[serial_test::serial]
+    async fn restore_transitioned_object_waits_for_existing_reader() {
+        let store = Arc::new(new_read_lock_test_store().await);
+        let ns_lock = store
+            .handle_new_ns_lock("bucket", "object")
+            .await
+            .expect("namespace lock should be created");
+        let _reader = ns_lock
+            .get_read_lock(Duration::from_secs(1))
+            .await
+            .expect("reader should acquire the object lock");
+
+        let err = temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("1"))], async {
+            store
+                .clone()
+                .handle_restore_transitioned_object("bucket", "object", &ObjectOptions::default())
+                .await
+                .expect_err("restore must wait behind an existing reader")
+        })
+        .await;
+
+        assert!(matches!(err, StorageError::Lock(rustfs_lock::LockError::Timeout { .. })));
+    }
+
    #[tokio::test]
    #[serial_test::serial]
    async fn reader_lock_is_held_when_optimization_is_disabled() {
@@ -280,14 +280,20 @@ async fn rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext() {

 async fn assert_fixture_round_trip(case_id: &str, expected_size: i64) {
    let (object_info, encrypted, expected_sha256) = load_fixture_reader_input(case_id).await;
-    let object_size = object_info.size;
+    // `ObjectInfo.size` is the on-disk size. For SSE objects that is the
+    // DARE-encrypted size (plaintext + 32 bytes per 64 KiB block), which is
+    // deliberately larger than the logical object size. The size a client sees
+    // (and what MinIO records via `x-*-internal-actual-size`) comes from
+    // `decrypted_size()`/`get_actual_size()`, so assert against that — the raw
+    // `size` field would never equal the plaintext length for encrypted objects.
+    let decrypted_size = object_info.decrypted_size().expect("decrypted size from MinIO metadata");
    let kms_key_b64 = minio_static_kms_key_b64();

    let plaintext = read_fixture_plaintext(encrypted, object_info, kms_key_b64)
        .await
        .expect("fixture must restore with the configured KMS key");

-    assert_eq!(object_size, expected_size);
+    assert_eq!(decrypted_size, expected_size);
    assert_eq!(plaintext.len(), expected_size as usize);
    assert_eq!(sha256_hex(&plaintext), expected_sha256);
 }
@@ -28,11 +28,11 @@ categories = ["web-programming", "development-tools"]
 doctest = false

 [dependencies]
-serde.workspace = true
+serde = { workspace = true, features = ["derive"] }
 thiserror.workspace = true

 [dev-dependencies]
-serde_json.workspace = true
+serde_json = { workspace = true, features = ["raw_value"] }

 [lints]
 workspace = true
@@ -34,22 +34,22 @@ hotpath = { workspace = true, optional = true }
 crc-fast = { workspace = true }
 rmp.workspace = true
 rmp-serde.workspace = true
-serde.workspace = true
-time.workspace = true
-uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
-tokio = { workspace = true, features = ["io-util", "macros", "sync"] }
-xxhash-rust = { workspace = true, features = ["xxh64"] }
-bytes.workspace = true
+serde = { workspace = true, features = ["derive"] }
+time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
+uuid = { workspace = true, features = ["v4", "fast-rng", "serde", "macro-diagnostics"] }
+tokio = { workspace = true, features = ["io-util", "macros", "sync", "fs", "rt-multi-thread"] }
+xxhash-rust = { workspace = true, features = ["xxh64", "xxh3"] }
+bytes = { workspace = true, features = ["serde"] }
 rustfs-utils = { workspace = true, features = ["hash", "http"] }
 byteorder = { workspace = true }
 tracing.workspace = true
 thiserror.workspace = true
-s3s.workspace = true
+s3s = { workspace = true, features = ["minio"] }
 regex.workspace = true
 arc-swap.workspace = true

 [dev-dependencies]
-criterion = { workspace = true }
+criterion = { workspace = true, features = ["html_reports"] }
 tempfile = { workspace = true }
 proptest = "1"

@@ -0,0 +1,209 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Structured version-graph marshal -> load roundtrip property (backlog#1151
+//! sec-10).
+//!
+//! xl.meta roundtrip coverage was a fixed example test plus byte-level no-panic
+//! fuzz properties; nothing generated STRUCTURED version graphs, so a lossy
+//! encoding bug in the version header, ordering, delete-marker, or
+//! dual-internal-metadata-key handling would only be caught if the fixed
+//! example happened to hit it. These properties generate multi-version graphs
+//! (objects + delete markers, distinct and colliding mod_times, user metadata,
+//! and the AGENTS.md dual `x-rustfs-internal-*` / `x-minio-internal-*` keys)
+//! and assert `FileMeta::marshal_msg` -> `FileMeta::load` preserves them
+//! exactly: version count, order, headers, and fully decoded per-version
+//! content.
+
+use proptest::collection::vec;
+use proptest::prelude::*;
+use rustfs_filemeta::{ChecksumAlgo, ErasureAlgo, FileMeta, FileMetaVersion, MetaDeleteMarker, MetaObject, VersionType};
+use rustfs_utils::http::insert_bytes;
+use std::collections::HashMap;
+use time::OffsetDateTime;
+use uuid::Uuid;
+
+/// Internal metadata suffixes to write under BOTH internal prefixes (the
+/// cross-cutting dual-key invariant). Realistic suffixes used by production
+/// code paths.
+const SYS_SUFFIXES: &[&str] = &["replication-status", "transition-status", "tier-name"];
+
+#[derive(Debug, Clone)]
+struct GenVersion {
+    is_delete_marker: bool,
+    version_id: u128,
+    /// Seconds offset added to the base timestamp. Small range on purpose so
+    /// some versions share a mod_time and the tie-break ordering is exercised.
+    mod_offset: i64,
+    size: i64,
+    user_meta: Vec<(String, String)>,
+    sys_suffix_idx: usize,
+    sys_value: Vec<u8>,
+}
+
+fn gen_version_strategy() -> impl Strategy<Value = GenVersion> {
+    (
+        proptest::bool::weighted(0.3),
+        1u128..u128::MAX,
+        0i64..8,
+        1i64..(1 << 30),
+        vec(("[a-z]{1,8}", "[a-zA-Z0-9 ]{0,16}"), 0..3),
+        0usize..SYS_SUFFIXES.len(),
+        vec(any::<u8>(), 1..24),
+    )
+        .prop_map(
+            |(is_delete_marker, version_id, mod_offset, size, user_meta, sys_suffix_idx, sys_value)| GenVersion {
+                is_delete_marker,
+                version_id,
+                mod_offset,
+                size,
+                user_meta,
+                sys_suffix_idx,
+                sys_value,
+            },
+        )
+}
+
+const BASE_TS: i64 = 1_700_000_000;
+
+fn build_version(g: &GenVersion) -> FileMetaVersion {
+    let mod_time = OffsetDateTime::from_unix_timestamp(BASE_TS + g.mod_offset).expect("valid timestamp");
+    let mut meta_sys = HashMap::new();
+    // Dual internal metadata keys: insert_bytes writes BOTH the
+    // x-rustfs-internal-* and x-minio-internal-* forms.
+    insert_bytes(&mut meta_sys, SYS_SUFFIXES[g.sys_suffix_idx], g.sys_value.clone());
+
+    if g.is_delete_marker {
+        FileMetaVersion {
+            version_type: VersionType::Delete,
+            delete_marker: Some(MetaDeleteMarker {
+                version_id: Some(Uuid::from_u128(g.version_id)),
+                mod_time: Some(mod_time),
+                meta_sys,
+            }),
+            write_version: 0,
+            ..Default::default()
+        }
+    } else {
+        FileMetaVersion {
+            version_type: VersionType::Object,
+            object: Some(MetaObject {
+                version_id: Some(Uuid::from_u128(g.version_id)),
+                data_dir: Some(Uuid::from_u128(g.version_id.wrapping_add(1).max(1))),
+                erasure_algorithm: ErasureAlgo::ReedSolomon,
+                erasure_m: 2,
+                erasure_n: 2,
+                erasure_block_size: 1 << 20,
+                erasure_index: 1,
+                erasure_dist: vec![1, 2, 3, 4],
+                bitrot_checksum_algo: ChecksumAlgo::HighwayHash,
+                part_numbers: vec![1],
+                part_etags: vec![],
+                part_sizes: vec![g.size as usize],
+                part_actual_sizes: vec![g.size],
+                part_indices: vec![],
+                size: g.size,
+                mod_time: Some(mod_time),
+                meta_sys,
+                meta_user: g.user_meta.iter().cloned().collect(),
+            }),
+            write_version: 0,
+            ..Default::default()
+        }
+    }
+}
+
+proptest! {
+    /// A generated version graph survives marshal -> load exactly: count,
+    /// order, headers, and fully decoded version content (objects, delete
+    /// markers, user metadata, and both internal metadata keys).
+    #[test]
+    fn version_graph_marshal_load_roundtrip(gens in vec(gen_version_strategy(), 1..8)) {
+        let mut fm = FileMeta::new();
+        for g in &gens {
+            fm.add_version_filemata(build_version(g)).expect("add generated version");
+        }
+
+        let encoded = fm.marshal_msg().expect("marshal generated version graph");
+        let loaded = FileMeta::load(&encoded).expect("load marshaled version graph");
+
+        // Count and metadata version survive.
+        prop_assert_eq!(loaded.versions.len(), fm.versions.len(), "version count must survive the roundtrip");
+        prop_assert_eq!(loaded.meta_ver, fm.meta_ver, "meta_ver must survive the roundtrip");
+
+        // Version ORDER survives exactly: add_version_filemata sorted the
+        // in-memory graph (mod_time desc with documented tie-breaks, delete
+        // markers interleaved); load must reproduce that same sequence, not
+        // re-derive a different one.
+        let pre: Vec<_> = fm.versions.iter().map(|v| (v.header.version_id, v.header.version_type.clone())).collect();
+        let post: Vec<_> = loaded.versions.iter().map(|v| (v.header.version_id, v.header.version_type.clone())).collect();
+        prop_assert_eq!(pre, post, "version (id, type) sequence must survive the roundtrip");
+
+        for (orig, back) in fm.versions.iter().zip(loaded.versions.iter()) {
+            // Full header equality (mod_time, flags, signature, ec fields).
+            prop_assert_eq!(&orig.header, &back.header, "version header must survive the roundtrip");
+
+            // Fully decoded version content equality.
+            let orig_v = orig.parse_version_meta().expect("decode original version");
+            let back_v = back.parse_version_meta().expect("decode roundtripped version");
+            prop_assert_eq!(&orig_v, &back_v, "decoded version content must survive the roundtrip");
+
+            // Delete-marker semantics preserved.
+            if back.header.version_type == VersionType::Delete {
+                prop_assert!(back_v.delete_marker.is_some(), "delete marker payload must survive");
+                prop_assert!(back_v.object.is_none(), "a delete marker must not grow an object payload");
+            } else {
+                prop_assert!(back_v.object.is_some(), "object payload must survive");
+            }
+
+            // Dual internal metadata keys: BOTH prefixed forms survive with the
+            // same value (losing either breaks MinIO interop; losing both loses
+            // replication/tier state).
+            let meta_sys = match (&back_v.object, &back_v.delete_marker) {
+                (Some(o), _) => &o.meta_sys,
+                (_, Some(d)) => &d.meta_sys,
+                _ => unreachable!("version decoded above"),
+            };
+            let orig_sys = match (&orig_v.object, &orig_v.delete_marker) {
+                (Some(o), _) => &o.meta_sys,
+                (_, Some(d)) => &d.meta_sys,
+                _ => unreachable!("version decoded above"),
+            };
+            for (k, v) in orig_sys {
+                prop_assert_eq!(
+                    meta_sys.get(k).map(|b| b.as_slice()),
+                    Some(v.as_slice()),
+                    "internal metadata key {} must survive the roundtrip with an identical value",
+                    k
+                );
+            }
+        }
+    }
+
+    /// Idempotence: a second marshal of the loaded graph is byte-identical.
+    /// Catches encoders that mutate or reorder state on the way out (a lossy
+    /// first pass shows up as a divergent second pass).
+    #[test]
+    fn version_graph_marshal_is_idempotent(gens in vec(gen_version_strategy(), 1..6)) {
+        let mut fm = FileMeta::new();
+        for g in &gens {
+            fm.add_version_filemata(build_version(g)).expect("add generated version");
+        }
+
+        let first = fm.marshal_msg().expect("first marshal");
+        let loaded = FileMeta::load(&first).expect("load first marshal");
+        let second = loaded.marshal_msg().expect("second marshal");
+        prop_assert_eq!(first, second, "marshal(load(marshal(x))) must be byte-identical to marshal(x)");
+    }
+}
@@ -34,27 +34,27 @@ rustfs-storage-api = { workspace = true }
 rustfs-common = { workspace = true }
 rustfs-madmin = { workspace = true }
 rustfs-utils = { workspace = true }
-tokio = { workspace = true, features = ["sync","io-util","time","macros"] }
-tokio-util = { workspace = true }
+tokio = { workspace = true, features = ["sync", "io-util", "time", "macros", "fs", "rt-multi-thread"] }
+tokio-util = { workspace = true, features = ["io", "compat"] }
 tracing = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
-serde_json = { workspace = true }
+serde_json = { workspace = true, features = ["raw_value"] }
 thiserror = { workspace = true }
-uuid = { workspace = true, features = ["v4", "serde"] }
+uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnostics"] }
 async-trait = { workspace = true }
 futures = { workspace = true }
 metrics = { workspace = true }
 base64 = { workspace = true }

 [dev-dependencies]
-serde_json = { workspace = true }
+serde_json = { workspace = true, features = ["raw_value"] }
+rustfs-test-utils = { workspace = true }
 serial_test = { workspace = true }
-tracing-subscriber = { workspace = true }
 tempfile = { workspace = true }
 walkdir = { workspace = true }
 http = { workspace = true }
 temp-env = { workspace = true }
-tokio = { workspace = true, features = ["test-util","fs"] }
+tokio = { workspace = true, features = ["test-util", "fs", "rt-multi-thread"] }

 [lib]
 doctest = false
@@ -99,6 +99,54 @@ impl Error {
    pub fn transient_skip(message: impl Into<String>) -> Self {
        Error::TransientSkip { message: message.into() }
    }
+
+    /// Whether a heal operation can be retried without changing its inputs.
+    pub(crate) fn is_recoverable_heal(&self) -> bool {
+        match self {
+            Error::TaskCancelled => false,
+            Error::TaskTimeout | Error::TransientSkip { .. } => true,
+            Error::Storage(err) => {
+                err.is_quorum_error()
+                    || matches!(err, EcstoreError::SlowDown | EcstoreError::OperationCanceled | EcstoreError::Lock(_))
+                    || is_recoverable_heal_error_message(&err.to_string())
+            }
+            Error::Disk(err) => {
+                matches!(
+                    err,
+                    DiskError::ErasureReadQuorum
+                        | DiskError::ErasureWriteQuorum
+                        | DiskError::Timeout
+                        | DiskError::SourceStalled
+                        | DiskError::FaultyRemoteDisk
+                        | DiskError::FaultyDisk
+                ) || is_recoverable_heal_error_message(&err.to_string())
+            }
+            Error::TaskExecutionFailed { message } | Error::IO(message) | Error::Other(message) => {
+                is_recoverable_heal_error_message(message)
+            }
+            Error::Io(err) => is_recoverable_heal_error_message(&err.to_string()),
+            _ => false,
+        }
+    }
+}
+
+fn is_recoverable_heal_error_message(error: &str) -> bool {
+    let error = error.to_ascii_lowercase();
+    [
+        "failed to acquire read lock",
+        "lock acquisition failed",
+        "lock acquisition timeout",
+        "remote lock rpc timed out",
+        "deadline has elapsed",
+        "timed out",
+        "transport error",
+        "network error",
+        "connection refused",
+        "operation canceled",
+        "quorum not reached",
+    ]
+    .iter()
+    .any(|pattern| error.contains(pattern))
 }

 impl From<Error> for std::io::Error {
@@ -34,7 +34,7 @@ use tokio::{
 use tokio_util::sync::CancellationToken;
 use tracing::{debug, error, info, warn};

-use super::{DiskError, EcstoreError, HealDiskExt as _, local_disk_map_read};
+use super::{DiskError, HealDiskExt as _, local_disk_map_read};

 const KEEP_HEAL_TASK_STATUS_DURATION: Duration = Duration::from_secs(10 * 60);
 const LOG_COMPONENT_HEAL: &str = "heal";
@@ -565,9 +565,12 @@ fn retry_request_for_result(task: &HealTask, result: &Result<()>) -> Option<(Hea
    if task.retry_attempts >= MAX_RECOVERABLE_HEAL_RETRIES {
        return None;
    }
+    if task.has_batch_failure() {
+        return None;
+    }

    let error = err.to_string();
-    if !is_recoverable_heal_error(err, &error) {
+    if !err.is_recoverable_heal() {
        return None;
    }

@@ -582,54 +585,6 @@ fn recoverable_heal_retry_delay(retry_attempt: u32) -> Duration {
    delay.min(MAX_RECOVERABLE_HEAL_RETRY_DELAY)
 }

-fn is_recoverable_heal_error(err: &Error, error: &str) -> bool {
-    match err {
-        Error::TaskCancelled => false,
-        Error::TaskTimeout | Error::TransientSkip { .. } => true,
-        Error::Storage(err) => is_recoverable_storage_heal_error(err) || is_recoverable_heal_error_message(error),
-        Error::Disk(err) => is_recoverable_disk_heal_error(err) || is_recoverable_heal_error_message(error),
-        Error::TaskExecutionFailed { .. } | Error::Io(_) | Error::IO(_) | Error::Other(_) => {
-            is_recoverable_heal_error_message(error)
-        }
-        _ => false,
-    }
-}
-
-fn is_recoverable_storage_heal_error(err: &EcstoreError) -> bool {
-    err.is_quorum_error() || matches!(err, EcstoreError::SlowDown | EcstoreError::OperationCanceled | EcstoreError::Lock(_))
-}
-
-fn is_recoverable_disk_heal_error(err: &DiskError) -> bool {
-    matches!(
-        err,
-        DiskError::ErasureReadQuorum
-            | DiskError::ErasureWriteQuorum
-            | DiskError::Timeout
-            | DiskError::SourceStalled
-            | DiskError::FaultyRemoteDisk
-            | DiskError::FaultyDisk
-    )
-}
-
-fn is_recoverable_heal_error_message(error: &str) -> bool {
-    let error = error.to_ascii_lowercase();
-    [
-        "failed to acquire read lock",
-        "lock acquisition failed",
-        "lock acquisition timeout",
-        "remote lock rpc timed out",
-        "deadline has elapsed",
-        "timed out",
-        "transport error",
-        "network error",
-        "connection refused",
-        "operation canceled",
-        "quorum not reached",
-    ]
-    .iter()
-    .any(|pattern| error.contains(pattern))
-}
-
 /// Heal config
 #[derive(Debug, Clone)]
 pub struct HealConfig {
@@ -2917,8 +2872,9 @@ fn can_schedule_request(request: &HealRequest, running_per_set: &HashMap<String,
 #[cfg(test)]
 mod tests {
    use super::*;
+    use crate::heal::EcstoreError;
    use crate::heal::storage::{HealObjectInfo, HealStorageAPI};
-    use crate::heal::task::{HealOptions, HealPriority, HealRequest, HealTask, HealType};
+    use crate::heal::task::{BatchHealFailure, HealOptions, HealPriority, HealRequest, HealTask, HealType};
    use rustfs_common::heal_channel::{HealOpts, HealRequestSource};
    use rustfs_concurrency::{WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot};
    use rustfs_madmin::heal_commands::HealResultItem;
@@ -3678,6 +3634,24 @@ mod tests {
        assert!(retry_request_for_result(&task, &result).is_none());
    }

+    #[tokio::test]
+    async fn test_retry_request_does_not_rescan_batch_after_object_retries_exhausted() {
+        let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
+        let task = HealTask::from_request(HealRequest::bucket("bucket".to_string()), storage);
+        let result = Err(task
+            .record_batch_failure(BatchHealFailure {
+                scope: "bucket:bucket".to_string(),
+                failed: 1,
+                retryable: 1,
+                permanent: 0,
+                first_object: "object".to_string(),
+                first_error: "Lock acquisition timeout".to_string(),
+            })
+            .await);
+
+        assert!(retry_request_for_result(&task, &result).is_none());
+    }
+
    #[test]
    fn test_heal_type_matches_path_normalizes_prefix_trailing_slash() {
        let heal_type = HealType::Prefix {
@@ -25,7 +25,10 @@ use rustfs_utils::path::SLASH_SEPARATOR;
 use serde::{Deserialize, Serialize};
 use std::{
    future::Future,
-    sync::Arc,
+    sync::{
+        Arc,
+        atomic::{AtomicBool, Ordering},
+    },
    time::{Duration, Instant, SystemTime},
 };
 use tokio::sync::RwLock;
@@ -41,6 +44,8 @@ const EVENT_HEAL_TASK_STATE: &str = "heal_task_state";
 const EVENT_HEAL_OBJECT_STAGE: &str = "heal_object_stage";
 const EVENT_HEAL_OBJECT_MISSING: &str = "heal_object_missing";
 const EVENT_HEAL_OBJECT_RESULT: &str = "heal_object_result";
+const MAX_BUCKET_OBJECT_HEAL_RETRIES: u32 = 3;
+const MAX_BUCKET_FAILURE_LOG_SAMPLES: u64 = 5;
 const EVENT_HEAL_BUCKET_STAGE: &str = "heal_bucket_stage";
 const EVENT_HEAL_BUCKET_RESULT: &str = "heal_bucket_result";
 const EVENT_HEAL_METADATA_STAGE: &str = "heal_metadata_stage";
@@ -181,6 +186,26 @@ pub enum HealTaskStatus {
    Timeout,
 }

+#[derive(Debug)]
+pub(crate) struct BatchHealFailure {
+    pub(crate) scope: String,
+    pub(crate) failed: u64,
+    pub(crate) retryable: u64,
+    pub(crate) permanent: u64,
+    pub(crate) first_object: String,
+    pub(crate) first_error: String,
+}
+
+impl std::fmt::Display for BatchHealFailure {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(
+            f,
+            "Heal batch failed for {}: {} failed ({} retryable, {} permanent); first failure at {}: {}",
+            self.scope, self.failed, self.retryable, self.permanent, self.first_object, self.first_error
+        )
+    }
+}
+
 /// Heal request
 #[derive(Debug, Clone)]
 pub struct HealRequest {
@@ -281,6 +306,8 @@ pub struct HealTask {
    pub progress: Arc<RwLock<HealProgress>>,
    /// Result items collected from storage heal calls.
    pub result_items: Arc<RwLock<Vec<HealResultItem>>>,
+    batch_failure: Arc<RwLock<Option<BatchHealFailure>>>,
+    batch_failure_recorded: Arc<AtomicBool>,
    /// Created time
    pub created_at: SystemTime,
    /// Queue admission time
@@ -310,6 +337,8 @@ impl HealTask {
            status: Arc::new(RwLock::new(HealTaskStatus::Pending)),
            progress: Arc::new(RwLock::new(HealProgress::new())),
            result_items: Arc::new(RwLock::new(Vec::new())),
+            batch_failure: Arc::new(RwLock::new(None)),
+            batch_failure_recorded: Arc::new(AtomicBool::new(false)),
            created_at: request.created_at,
            enqueued_at: request.enqueued_at,
            started_at: Arc::new(RwLock::new(None)),
@@ -348,6 +377,21 @@ impl HealTask {
        }
    }

+    pub(crate) fn has_batch_failure(&self) -> bool {
+        self.batch_failure_recorded.load(Ordering::Acquire)
+    }
+
+    pub(crate) async fn record_batch_failure(&self, failure: BatchHealFailure) -> Error {
+        self.batch_failure_recorded.store(true, Ordering::Release);
+        let message = failure.to_string();
+        *self.batch_failure.write().await = Some(failure);
+        Error::TaskExecutionFailed { message }
+    }
+
+    async fn take_batch_failure(&self) -> Option<BatchHealFailure> {
+        self.batch_failure.write().await.take()
+    }
+
    pub fn metric_set_label(&self) -> String {
        match &self.heal_type {
            HealType::ErasureSet { set_disk_id, .. } => set_disk_id.clone(),
@@ -477,6 +521,15 @@ impl HealTask {
        }
    }

+    fn bucket_object_retry_delay(&self, retry_attempt: u32) -> Duration {
+        let base = Duration::from_secs(2_u64.saturating_pow(retry_attempt.clamp(1, MAX_BUCKET_OBJECT_HEAL_RETRIES)));
+        let jitter_seed = self
+            .id
+            .bytes()
+            .fold(0_u64, |acc, byte| acc.wrapping_mul(31).wrapping_add(u64::from(byte)));
+        Duration::from_millis(jitter_seed % 500).saturating_add(base)
+    }
+
    fn should_return_typed_heal_error(err: &Error) -> bool {
        matches!(err, Error::Storage(_) | Error::Disk(_))
    }
@@ -1263,9 +1316,61 @@ impl HealTask {
        );

        let bucket_infos = self.await_with_control(self.storage.list_buckets()).await?;
+        let mut failed = 0_u64;
+        let mut retryable = 0_u64;
+        let mut permanent = 0_u64;
+        let mut first_object = None;
+        let mut first_error = None;
        for bucket_info in bucket_infos {
            self.check_control_flags().await?;
-            self.heal_bucket(&bucket_info.name).await?;
+            let mut retry_attempt = 0_u32;
+            loop {
+                match self.heal_bucket(&bucket_info.name).await {
+                    Ok(()) => break,
+                    Err(Error::TaskCancelled) => return Err(Error::TaskCancelled),
+                    Err(Error::TaskTimeout) => return Err(Error::TaskTimeout),
+                    Err(err) => {
+                        if let Some(failure) = self.take_batch_failure().await {
+                            failed = failed.saturating_add(failure.failed);
+                            retryable = retryable.saturating_add(failure.retryable);
+                            permanent = permanent.saturating_add(failure.permanent);
+                            first_object.get_or_insert(failure.first_object);
+                            first_error.get_or_insert(failure.first_error);
+                            break;
+                        }
+                        if err.is_recoverable_heal() && retry_attempt < MAX_BUCKET_OBJECT_HEAL_RETRIES {
+                            retry_attempt = retry_attempt.saturating_add(1);
+                            self.await_with_control(async {
+                                tokio::time::sleep(self.bucket_object_retry_delay(retry_attempt)).await;
+                                Ok(())
+                            })
+                            .await?;
+                            continue;
+                        }
+                        failed = failed.saturating_add(1);
+                        if err.is_recoverable_heal() {
+                            retryable = retryable.saturating_add(1);
+                        } else {
+                            permanent = permanent.saturating_add(1);
+                        }
+                        first_object.get_or_insert(bucket_info.name.clone());
+                        first_error.get_or_insert_with(|| err.to_string());
+                        break;
+                    }
+                }
+            }
+        }
+
+        if failed > 0 {
+            let failure = BatchHealFailure {
+                scope: "cluster".to_string(),
+                failed,
+                retryable,
+                permanent,
+                first_object: first_object.unwrap_or_default(),
+                first_error: first_error.unwrap_or_default(),
+            };
+            return Err(self.record_batch_failure(failure).await);
        }

        Ok(())
@@ -1292,7 +1397,12 @@ impl HealTask {
        let mut scanned = 0u64;
        let mut healed = 0u64;
        let mut failed = 0u64;
+        let mut retryable_failed = 0u64;
+        let mut permanent_failed = 0u64;
        let mut bytes = 0u64;
+        let mut first_failed_object = None;
+        let mut first_error = None;
+        let mut failure_samples_logged = 0_u64;

        let heal_opts = HealOpts {
            recursive: false,
@@ -1315,34 +1425,44 @@ impl HealTask {
                )
                .await?;

-            for item in objects {
-                self.check_control_flags().await?;
-                let object = item.name.as_str();
-                scanned += 1;
-                {
-                    let mut progress = self.progress.write().await;
-                    progress.set_current_object(Some(format!("{bucket}/{object}")));
-                    progress.update_progress(scanned, healed, failed, bytes);
+            let mut pending = objects;
+            let mut retry_attempt = 0_u32;
+            while !pending.is_empty() {
+                if retry_attempt > 0 {
+                    self.await_with_control(async {
+                        tokio::time::sleep(self.bucket_object_retry_delay(retry_attempt)).await;
+                        Ok(())
+                    })
+                    .await?;
                }
-
-                // Heal each enumerated version directly. There is no latest-only
-                // existence gate: genuine absence surfaces through heal_object's
-                // not-found result and is handled below.
-                match self
-                    .await_with_control(
-                        self.storage
-                            .heal_object(bucket, object, item.version_id.as_deref(), &heal_opts),
-                    )
-                    .await
-                {
-                    Ok((result, None)) => {
-                        healed += 1;
-                        bytes = bytes.saturating_add(result.object_size as u64);
-                        self.record_result_item(result).await;
+                let mut retry = Vec::with_capacity(pending.len());
+                for item in pending {
+                    self.check_control_flags().await?;
+                    let object = item.name.as_str();
+                    if retry_attempt == 0 {
+                        scanned = scanned.saturating_add(1);
                    }
-                    Ok((_, Some(err))) => {
-                        if is_missing_object_dir_heal_result(object, &err) {
-                            healed += 1;
+                    {
+                        let mut progress = self.progress.write().await;
+                        progress.set_current_object(Some(format!("{bucket}/{object}")));
+                        progress.update_progress(scanned, healed, failed, bytes);
+                    }
+
+                    let error = match self
+                        .await_with_control(
+                            self.storage
+                                .heal_object(bucket, object, item.version_id.as_deref(), &heal_opts),
+                        )
+                        .await
+                    {
+                        Ok((result, None)) => {
+                            healed = healed.saturating_add(1);
+                            bytes = bytes.saturating_add(u64::try_from(result.object_size).unwrap_or_default());
+                            self.record_result_item(result).await;
+                            None
+                        }
+                        Ok((_, Some(err))) if is_missing_object_dir_heal_result(object, &err) => {
+                            healed = healed.saturating_add(1);
                            debug!(
                                target: "rustfs::heal::task",
                                event = EVENT_HEAL_BUCKET_RESULT,
@@ -1350,77 +1470,77 @@ impl HealTask {
                                subsystem = LOG_SUBSYSTEM_TASK,
                                task_id = %self.id,
                                bucket,
-                                object = %object,
+                                object,
                                result = "object_dir_not_found_skipped",
                                "Heal bucket object-dir candidate skipped after not-found result"
                            );
-                            continue;
+                            None
                        }
-                        if Self::should_skip_data_usage_cache_heal_error(bucket, object, &err) {
-                            warn!(
-                                target: "rustfs::heal::task",
-                                event = EVENT_HEAL_BUCKET_RESULT,
-                                component = LOG_COMPONENT_HEAL,
-                                subsystem = LOG_SUBSYSTEM_TASK,
-                                task_id = %self.id,
-                                bucket,
-                                object = %object,
-                                result = "transient_skip",
-                                error = %err,
-                                "Heal bucket object repair skipped due to transient error"
-                            );
-                        } else {
-                            failed += 1;
-                            warn!(
-                                target: "rustfs::heal::task",
-                                event = EVENT_HEAL_BUCKET_RESULT,
-                                component = LOG_COMPONENT_HEAL,
-                                subsystem = LOG_SUBSYSTEM_TASK,
-                                task_id = %self.id,
-                                bucket,
-                                object = %object,
-                                result = "object_failed",
-                                error = %err,
-                                "Heal bucket object repair failed"
-                            );
-                        }
-                    }
-                    Err(err) => {
-                        if Self::should_skip_data_usage_cache_heal_error(bucket, object, &err) {
-                            warn!(
-                                target: "rustfs::heal::task",
-                                event = EVENT_HEAL_BUCKET_RESULT,
-                                component = LOG_COMPONENT_HEAL,
-                                subsystem = LOG_SUBSYSTEM_TASK,
-                                task_id = %self.id,
-                                bucket,
-                                object = %object,
-                                result = "transient_skip",
-                                error = %err,
-                                "Heal bucket object repair skipped due to transient error"
-                            );
-                        } else {
-                            failed += 1;
-                            warn!(
-                                target: "rustfs::heal::task",
-                                event = EVENT_HEAL_BUCKET_RESULT,
-                                component = LOG_COMPONENT_HEAL,
-                                subsystem = LOG_SUBSYSTEM_TASK,
-                                task_id = %self.id,
-                                bucket,
-                                object = %object,
-                                result = "object_failed",
-                                error = %err,
-                                "Heal bucket object repair failed"
-                            );
-                        }
-                    }
-                }
+                        Ok((_, Some(err))) | Err(err) => Some(err),
+                    };
+
+                    if let Some(err) = error {
+                        if Self::should_skip_data_usage_cache_heal_error(bucket, object, &err) {
+                            warn!(
+                                target: "rustfs::heal::task",
+                                event = EVENT_HEAL_BUCKET_RESULT,
+                                component = LOG_COMPONENT_HEAL,
+                                subsystem = LOG_SUBSYSTEM_TASK,
+                                task_id = %self.id,
+                                bucket,
+                                object,
+                                result = "transient_skip",
+                                error = %err,
+                                "Heal bucket object repair skipped due to transient metadata error"
+                            );
+                        } else if err.is_recoverable_heal() && retry_attempt < MAX_BUCKET_OBJECT_HEAL_RETRIES {
+                            debug!(
+                                target: "rustfs::heal::task",
+                                event = EVENT_HEAL_BUCKET_RESULT,
+                                component = LOG_COMPONENT_HEAL,
+                                subsystem = LOG_SUBSYSTEM_TASK,
+                                task_id = %self.id,
+                                bucket,
+                                object,
+                                retry_attempt = retry_attempt.saturating_add(1),
+                                error = %err,
+                                result = "object_retry_scheduled",
+                                "Heal bucket object retry scheduled"
+                            );
+                            retry.push(item);
+                        } else {
+                            failed = failed.saturating_add(1);
+                            if err.is_recoverable_heal() {
+                                retryable_failed = retryable_failed.saturating_add(1);
+                            } else {
+                                permanent_failed = permanent_failed.saturating_add(1);
+                            }
+                            first_failed_object.get_or_insert_with(|| object.to_string());
+                            first_error.get_or_insert_with(|| err.to_string());
+                            if failure_samples_logged < MAX_BUCKET_FAILURE_LOG_SAMPLES {
+                                failure_samples_logged = failure_samples_logged.saturating_add(1);
+                                warn!(
+                                    target: "rustfs::heal::task",
+                                    event = EVENT_HEAL_BUCKET_RESULT,
+                                    component = LOG_COMPONENT_HEAL,
+                                    subsystem = LOG_SUBSYSTEM_TASK,
+                                    task_id = %self.id,
+                                    bucket,
+                                    object,
+                                    retry_attempt,
+                                    error = %err,
+                                    result = "object_failed",
+                                    "Heal bucket object repair failed"
+                                );
+                            }
+                        }
+                    }

-                {
                    let mut progress = self.progress.write().await;
                    progress.update_progress(scanned, healed, failed, bytes);
                }
+                pending = retry;
+                retry_attempt = retry_attempt.saturating_add(1);
            }

            if !is_truncated {
@@ -1435,9 +1555,15 @@ impl HealTask {
        }

        if failed > 0 {
-            return Err(Error::TaskExecutionFailed {
-                message: format!("Failed to heal {failed} object(s) in bucket {bucket}"),
-            });
+            let failure = BatchHealFailure {
+                scope: format!("bucket:{bucket}"),
+                failed,
+                retryable: retryable_failed,
+                permanent: permanent_failed,
+                first_object: first_failed_object.unwrap_or_default(),
+                first_error: first_error.unwrap_or_default(),
+            };
+            return Err(self.record_batch_failure(failure).await);
        }

        debug!(
@@ -2223,7 +2349,7 @@ mod tests {
    use super::*;
    use crate::heal::storage::{DiskStatus, HealListItem, HealObjectInfo};
    use rustfs_madmin::heal_commands::HealResultItem;
-    use std::collections::HashMap;
+    use std::collections::{HashMap, VecDeque};
    use std::sync::Mutex;

    use super::super::storage_api::status::BucketInfo;
@@ -2238,11 +2364,15 @@ mod tests {
        object_exists: Mutex<Option<bool>>,
        object_exists_by_name: Mutex<HashMap<String, MockObjectExists>>,
        heal_object_outcome: Mutex<Option<MockHealObjectOutcome>>,
+        heal_object_outcomes: Mutex<HashMap<String, VecDeque<MockHealObjectOutcome>>>,
        deleted_objects: Mutex<Vec<String>>,
        format_no_heal_required: Mutex<bool>,
        listed_prefixes: Mutex<Vec<String>>,
        truncate_without_token: Mutex<bool>,
        include_object_dir_candidate: Mutex<bool>,
+        listed_buckets: Mutex<Option<Vec<String>>>,
+        bucket_heal_errors: Mutex<HashMap<String, VecDeque<&'static str>>>,
+        bucket_heal_calls: Mutex<Vec<String>>,
    }

    /// Build a latest, non-delete-marker heal list item with no version id.
@@ -2257,6 +2387,8 @@ mod tests {
    enum MockHealObjectOutcome {
        OkWithOtherError(&'static str),
        ErrOther(&'static str),
+        RetryableReadQuorum,
+        PermanentOther(&'static str),
    }

    #[derive(Clone, Copy)]
@@ -2326,10 +2458,19 @@ mod tests {
        }

        async fn list_buckets(&self) -> Result<Vec<BucketInfo>> {
-            Ok(vec![BucketInfo {
-                name: "bucket-a".to_string(),
-                ..Default::default()
-            }])
+            let buckets = self
+                .listed_buckets
+                .lock()
+                .unwrap()
+                .clone()
+                .unwrap_or_else(|| vec!["bucket-a".to_string()]);
+            Ok(buckets
+                .into_iter()
+                .map(|name| BucketInfo {
+                    name,
+                    ..Default::default()
+                })
+                .collect())
        }

        async fn object_exists(&self, _bucket: &str, object: &str) -> Result<bool> {
@@ -2364,12 +2505,37 @@ mod tests {
                .unwrap()
                .push(version_id.map(ToString::to_string));
            self.object_heal_opts.lock().unwrap().push(*opts);
+            if let Some(outcome) = self
+                .heal_object_outcomes
+                .lock()
+                .unwrap()
+                .get_mut(object)
+                .and_then(VecDeque::pop_front)
+            {
+                return match outcome {
+                    MockHealObjectOutcome::RetryableReadQuorum => Err(Error::Storage(EcstoreError::InsufficientReadQuorum(
+                        bucket.to_string(),
+                        object.to_string(),
+                    ))),
+                    MockHealObjectOutcome::PermanentOther(message) => Err(Error::other(message)),
+                    MockHealObjectOutcome::OkWithOtherError(message) => {
+                        Ok((HealResultItem::default(), Some(Error::other(message))))
+                    }
+                    MockHealObjectOutcome::ErrOther(message) => Err(Error::other(message)),
+                };
+            }
            if let Some(outcome) = self.heal_object_outcome.lock().unwrap().take() {
                return match outcome {
                    MockHealObjectOutcome::OkWithOtherError(message) => {
                        Ok((HealResultItem::default(), Some(Error::other(message))))
                    }
-                    MockHealObjectOutcome::ErrOther(message) => Err(Error::other(message)),
+                    MockHealObjectOutcome::ErrOther(message) | MockHealObjectOutcome::PermanentOther(message) => {
+                        Err(Error::other(message))
+                    }
+                    MockHealObjectOutcome::RetryableReadQuorum => Err(Error::Storage(EcstoreError::InsufficientReadQuorum(
+                        bucket.to_string(),
+                        object.to_string(),
+                    ))),
                };
            }
            if bucket == RUSTFS_META_BUCKET && object == format!("{BUCKET_META_PREFIX}/{DATA_USAGE_CACHE_NAME}") {
@@ -2393,8 +2559,18 @@ mod tests {
            ))
        }

-        async fn heal_bucket(&self, _bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
+        async fn heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
+            self.bucket_heal_calls.lock().unwrap().push(bucket.to_string());
            self.bucket_heal_opts.lock().unwrap().push(*opts);
+            if let Some(message) = self
+                .bucket_heal_errors
+                .lock()
+                .unwrap()
+                .get_mut(bucket)
+                .and_then(VecDeque::pop_front)
+            {
+                return Err(Error::other(message));
+            }
            Ok(HealResultItem::default())
        }

@@ -2570,6 +2746,152 @@ mod tests {
        assert!(matches!(task.get_status().await, HealTaskStatus::Completed));
    }

+    #[tokio::test(start_paused = true)]
+    async fn test_recursive_bucket_heal_retries_only_retryable_objects() {
+        let storage = Arc::new(MockStorage::default());
+        storage
+            .heal_object_outcomes
+            .lock()
+            .unwrap()
+            .insert("object-a".to_string(), VecDeque::from([MockHealObjectOutcome::RetryableReadQuorum]));
+        let request = HealRequest::new(
+            HealType::Bucket {
+                bucket: "bucket-a".to_string(),
+            },
+            HealOptions {
+                recursive: true,
+                timeout: None,
+                ..Default::default()
+            },
+            HealPriority::Normal,
+        );
+        let task = HealTask::from_request(request, storage.clone());
+
+        task.heal_bucket("bucket-a")
+            .await
+            .expect("retryable object failure should be retried within the listing page");
+
+        assert_eq!(
+            storage.heal_object_calls.lock().unwrap().as_slice(),
+            ["object-a".to_string(), "object-b".to_string(), "object-a".to_string()]
+        );
+        let progress = task.get_progress().await;
+        assert_eq!(progress.objects_scanned, 2);
+        assert_eq!(progress.objects_healed, 2);
+        assert_eq!(progress.objects_failed, 0);
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_recursive_bucket_heal_reports_typed_exhausted_and_permanent_failures() {
+        let storage = Arc::new(MockStorage::default());
+        storage.heal_object_outcomes.lock().unwrap().insert(
+            "object-a".to_string(),
+            VecDeque::from([
+                MockHealObjectOutcome::RetryableReadQuorum,
+                MockHealObjectOutcome::RetryableReadQuorum,
+                MockHealObjectOutcome::RetryableReadQuorum,
+                MockHealObjectOutcome::RetryableReadQuorum,
+            ]),
+        );
+        storage.heal_object_outcomes.lock().unwrap().insert(
+            "object-b".to_string(),
+            VecDeque::from([MockHealObjectOutcome::PermanentOther("invalid metadata")]),
+        );
+        let request = HealRequest::new(
+            HealType::Bucket {
+                bucket: "bucket-a".to_string(),
+            },
+            HealOptions {
+                recursive: true,
+                timeout: None,
+                ..Default::default()
+            },
+            HealPriority::Normal,
+        );
+        let task = HealTask::from_request(request, storage.clone());
+
+        let err = task
+            .heal_bucket("bucket-a")
+            .await
+            .expect_err("exhausted retryable and permanent failures must fail the task");
+
+        assert!(matches!(err, Error::TaskExecutionFailed { .. }));
+        let failure = task
+            .take_batch_failure()
+            .await
+            .expect("batch failure details should be retained on the task");
+        assert_eq!(failure.failed, 2);
+        assert_eq!(failure.retryable, 1);
+        assert_eq!(failure.permanent, 1);
+        assert_eq!(failure.first_object, "object-b");
+        let calls = storage.heal_object_calls.lock().unwrap();
+        assert_eq!(calls.iter().filter(|object| object.as_str() == "object-a").count(), 4);
+        assert_eq!(calls.iter().filter(|object| object.as_str() == "object-b").count(), 1);
+    }
+
+    #[tokio::test]
+    async fn test_cluster_heal_continues_after_bucket_failure() {
+        let storage = Arc::new(MockStorage {
+            listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
+            bucket_heal_errors: Mutex::new(HashMap::from([("bucket-a".to_string(), VecDeque::from(["metadata unavailable"]))])),
+            ..Default::default()
+        });
+        let request = HealRequest::new(
+            HealType::Cluster,
+            HealOptions {
+                recursive: true,
+                timeout: None,
+                ..Default::default()
+            },
+            HealPriority::Normal,
+        );
+        let task = HealTask::from_request(request, storage.clone());
+
+        let err = task.execute().await.expect_err("cluster task must report the failed bucket");
+
+        assert!(matches!(err, Error::TaskExecutionFailed { .. }));
+        let failure = task
+            .take_batch_failure()
+            .await
+            .expect("cluster failure details should be retained on the task");
+        assert_eq!(failure.failed, 1);
+        assert_eq!(
+            storage.bucket_heal_calls.lock().unwrap().as_slice(),
+            ["bucket-a".to_string(), "bucket-b".to_string()]
+        );
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_cluster_heal_retries_only_recoverable_bucket() {
+        let storage = Arc::new(MockStorage {
+            listed_buckets: Mutex::new(Some(vec!["bucket-a".to_string(), "bucket-b".to_string()])),
+            bucket_heal_errors: Mutex::new(HashMap::from([(
+                "bucket-a".to_string(),
+                VecDeque::from(["lock acquisition timeout"]),
+            )])),
+            ..Default::default()
+        });
+        let request = HealRequest::new(
+            HealType::Cluster,
+            HealOptions {
+                recursive: true,
+                timeout: None,
+                ..Default::default()
+            },
+            HealPriority::Normal,
+        );
+        let task = HealTask::from_request(request, storage.clone());
+
+        task.execute()
+            .await
+            .expect("cluster task should retry a recoverable bucket failure");
+
+        assert_eq!(
+            storage.bucket_heal_calls.lock().unwrap().as_slice(),
+            ["bucket-a".to_string(), "bucket-a".to_string(), "bucket-b".to_string()]
+        );
+    }
+
    #[tokio::test]
    async fn test_recursive_bucket_heal_does_not_remove_bucket_metadata() {
        let storage = Arc::new(MockStorage::default());
@@ -18,7 +18,7 @@
 //!
 //! These drive the REAL `ECStoreHealStorage` (not a mock) against a real 4-disk
 //! `ECStore`, mirroring `heal_integration_test.rs`. Every test is `#[serial]`
-//! and re-runs the full `setup_test_env` init (which sets the process-global
+//! and re-runs the full `heal_env` init (which sets the process-global
 //! bucket-metadata-sys OnceCell) — under `cargo nextest` each test runs
 //! in its own process so the OnceCell never collides.

@@ -34,20 +34,15 @@ use rustfs_heal::heal::{
 use serial_test::serial;
 use std::{
    path::{Path, PathBuf},
-    sync::{Arc, Once},
+    sync::Arc,
    time::Duration,
 };
-use tokio::fs;
-use tokio_util::sync::CancellationToken;
 use tracing::info;
 use walkdir::WalkDir;

 mod storage_api;

-use storage_api::integration::{
-    BucketOperations, BucketOptions, ECStore, Endpoint, EndpointServerPools, Endpoints, MakeBucketOptions, ObjectIO as _,
-    ObjectOperations as _, PoolEndpoints, init_bucket_metadata_sys, init_local_disks,
-};
+use storage_api::integration::{BucketOperations, ECStore, MakeBucketOptions, ObjectIO as _, ObjectOperations as _};

 /// 256 KiB + change: large enough to be stored as non-inline erasure shards
 /// (so each data version materializes as an on-disk `part.*` file we can assert
@@ -62,78 +57,16 @@ fn versioned_test_data(seed: u8) -> Vec<u8> {
        .collect()
 }

-static INIT: Once = Once::new();
-
-fn init_tracing() {
-    INIT.call_once(|| {
-        let _ = tracing_subscriber::fmt()
-            .with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
-            .with_timer(tracing_subscriber::fmt::time::UtcTime::rfc_3339())
-            .with_thread_names(true)
-            .try_init();
-    });
-}
-
-/// Build a real 4-disk `ECStore` + `ECStoreHealStorage`. Mirrors
-/// `heal_integration_test::setup_test_env`.
-async fn setup_test_env() -> (Vec<PathBuf>, Arc<ECStore>, Arc<ECStoreHealStorage>) {
-    init_tracing();
-
-    let test_base_dir = format!("/tmp/rustfs_heal_b5_test_{}", uuid::Uuid::new_v4());
-    let temp_dir = PathBuf::from(&test_base_dir);
-    if temp_dir.exists() {
-        fs::remove_dir_all(&temp_dir).await.ok();
-    }
-    fs::create_dir_all(&temp_dir).await.unwrap();
-
-    let disk_paths = vec![
-        temp_dir.join("disk1"),
-        temp_dir.join("disk2"),
-        temp_dir.join("disk3"),
-        temp_dir.join("disk4"),
-    ];
-    for disk_path in &disk_paths {
-        fs::create_dir_all(disk_path).await.unwrap();
-    }
-
-    let mut endpoints = Vec::new();
-    for (i, disk_path) in disk_paths.iter().enumerate() {
-        let mut endpoint = Endpoint::try_from(disk_path.to_str().unwrap()).unwrap();
-        endpoint.set_pool_index(0);
-        endpoint.set_set_index(0);
-        endpoint.set_disk_index(i);
-        endpoints.push(endpoint);
-    }
-
-    let pool_endpoints = PoolEndpoints {
-        legacy: false,
-        set_count: 1,
-        drives_per_set: 4,
-        endpoints: Endpoints::from(endpoints),
-        cmd_line: "test".to_string(),
-        platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
-    };
-    let endpoint_pools = EndpointServerPools::from(vec![pool_endpoints]);
-
-    init_local_disks(endpoint_pools.clone()).await.unwrap();
-
-    let server_addr: std::net::SocketAddr = "127.0.0.1:0".parse().unwrap();
-    let ecstore = ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
-        .await
-        .unwrap();
-
-    let buckets_list = ecstore
-        .list_bucket(&BucketOptions {
-            no_metadata: true,
-            ..Default::default()
-        })
-        .await
-        .unwrap();
-    let buckets = buckets_list.into_iter().map(|v| v.name).collect();
-    init_bucket_metadata_sys(ecstore.clone(), buckets).await;
-
-    let heal_storage = Arc::new(ECStoreHealStorage::new(ecstore.clone()));
-    (disk_paths, ecstore, heal_storage)
+/// Build a real 4-disk `ECStore` + `ECStoreHealStorage` via the shared
+/// rustfs-test-utils environment (backlog#1153 infra-1). Mirrors
+/// `heal_integration_test::heal_env`.
+async fn heal_env() -> (Vec<PathBuf>, Arc<ECStore>, Arc<ECStoreHealStorage>) {
+    let env = rustfs_test_utils::TestECStoreEnv::builder()
+        .prefix("rustfs_heal_b5_test")
+        .build()
+        .await;
+    let heal_storage = Arc::new(ECStoreHealStorage::new(env.ecstore.clone()));
+    (env.disk_paths, env.ecstore, heal_storage)
 }

 /// Create a bucket with S3 versioning ENABLED at creation time. Without this the
@@ -308,7 +241,7 @@ mod serial_tests {
    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    #[serial]
    async fn test_enumeration_includes_all_versions_and_delete_marker_real_fixture() {
-        let (_disk_paths, ecstore, heal_storage) = setup_test_env().await;
+        let (_disk_paths, ecstore, heal_storage) = heal_env().await;
        let bucket = "b5-enum-versions";
        let object = "obj.bin";
        create_versioned_bucket(&ecstore, bucket).await;
@@ -345,7 +278,7 @@ mod serial_tests {
    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    #[serial]
    async fn test_heal_old_nonlatest_version_after_disk_wipe() {
-        let (disk_paths, ecstore, heal_storage) = setup_test_env().await;
+        let (disk_paths, ecstore, heal_storage) = heal_env().await;
        let bucket = "b5-old-version-heal";
        let object = "obj.bin";
        create_versioned_bucket(&ecstore, bucket).await;
@@ -397,7 +330,7 @@ mod serial_tests {
    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    #[serial]
    async fn test_heal_delete_marker_latest_enumerated_and_healed() {
-        let (disk_paths, ecstore, heal_storage) = setup_test_env().await;
+        let (disk_paths, ecstore, heal_storage) = heal_env().await;
        let bucket = "b5-dm-latest-heal";
        let object = "obj.bin";
        create_versioned_bucket(&ecstore, bucket).await;
@@ -463,7 +396,7 @@ mod serial_tests {
    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    #[serial]
    async fn test_version_id_normalization_null_and_unversioned_real_fixture() {
-        let (_disk_paths, ecstore, heal_storage) = setup_test_env().await;
+        let (_disk_paths, ecstore, heal_storage) = heal_env().await;
        let bucket = "b5-unversioned-normalize";
        create_unversioned_bucket(&ecstore, bucket).await;

@@ -499,7 +432,7 @@ mod serial_tests {
    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    #[serial]
    async fn test_heal_unversioned_bucket_e2e() {
-        let (disk_paths, ecstore, heal_storage) = setup_test_env().await;
+        let (disk_paths, ecstore, heal_storage) = heal_env().await;
        let bucket = "b5-unversioned-e2e";
        create_unversioned_bucket(&ecstore, bucket).await;

@@ -584,7 +517,7 @@ mod serial_tests {
    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    #[serial]
    async fn test_heal_resume_across_page_boundary_e2e() {
-        let (disk_paths, ecstore, heal_storage) = setup_test_env().await;
+        let (disk_paths, ecstore, heal_storage) = heal_env().await;
        let bucket = "b5-resume-e2e";
        create_versioned_bucket(&ecstore, bucket).await;

@@ -617,28 +550,37 @@ mod serial_tests {
        };
        let heal_manager = HealManager::new(heal_storage.clone(), Some(cfg));
        heal_manager.start().await.unwrap();
-        let request = HealRequest::new(
-            HealType::ErasureSet {
-                buckets: vec![bucket.to_string()],
-                set_disk_id: SET_DISK_ID.to_string(),
-            },
-            HealOptions {
-                recursive: true,
-                recreate_missing: true,
-                scan_mode: HealScanMode::Normal,
-                timeout: Some(Duration::from_secs(300)),
-                ..Default::default()
-            },
-            HealPriority::Normal,
-        );
-        let task_id = request.id.clone();
-        let admission = heal_manager
-            .submit_heal_request(request)
-            .await
-            .expect("failed to submit erasure-set heal");
-        assert!(admission.is_admitted(), "erasure-set heal must be admitted");

-        wait_for_task(&heal_manager, &task_id, Duration::from_secs(120)).await;
+        // The erasure-set healer defers any per-version heal that hits a
+        // transient error (unmet quorum, DiskNotFound, a slow-disk read under
+        // load) to a later heal cycle: it persists its resume/checkpoint state
+        // and returns a terminal `Failed { .. "retry scheduled" }` so a *fresh*
+        // heal run re-drives the still-unhealed versions (see the finalize step
+        // in `ErasureSetHealer::heal_bucket_with_resume`). In production the
+        // background scanner is that next run; here we supply it ourselves by
+        // re-submitting the idempotent heal. This keeps the e2e faithful to the
+        // resume design without making it hostage to a rare single-version
+        // transient hiccup — the strict data-restoration assertions below still
+        // run only after a genuine `Completed`.
+        let build_request = |force_start: bool| {
+            let mut request = HealRequest::new(
+                HealType::ErasureSet {
+                    buckets: vec![bucket.to_string()],
+                    set_disk_id: SET_DISK_ID.to_string(),
+                },
+                HealOptions {
+                    recursive: true,
+                    recreate_missing: true,
+                    scan_mode: HealScanMode::Normal,
+                    timeout: Some(Duration::from_secs(300)),
+                    ..Default::default()
+                },
+                HealPriority::Normal,
+            );
+            request.force_start = force_start;
+            request
+        };
+        drive_heal_to_completion(&heal_manager, build_request, Duration::from_secs(120), 3).await;

        // Every data version is readable end-to-end and physically restored on the
        // wiped disk. (Resume machinery drove the full per-version heal.)
@@ -664,20 +606,74 @@ mod serial_tests {

    /// Poll a heal task to a terminal state, panicking on failure/timeout.
    async fn wait_for_task(heal_manager: &HealManager, task_id: &str, timeout: Duration) {
+        match await_terminal_status(heal_manager, task_id, timeout).await {
+            HealTaskStatus::Completed => {}
+            HealTaskStatus::Failed { error } => panic!("heal task failed: {error}"),
+            HealTaskStatus::Cancelled => panic!("heal task was cancelled"),
+            other => panic!("heal task reached unexpected terminal state: {other:?}"),
+        }
+    }
+
+    /// Poll a heal task until it reaches a terminal state and return that state.
+    /// Panics only on timeout — callers decide how to treat each terminal state.
+    async fn await_terminal_status(heal_manager: &HealManager, task_id: &str, timeout: Duration) -> HealTaskStatus {
        let deadline = tokio::time::Instant::now() + timeout;
        loop {
-            if let Ok(status) = heal_manager.get_task_status(task_id).await {
-                match status {
-                    HealTaskStatus::Completed => return,
-                    HealTaskStatus::Failed { ref error } => panic!("heal task failed: {error}"),
-                    HealTaskStatus::Cancelled => panic!("heal task was cancelled"),
-                    _ => {}
-                }
+            if let Ok(status) = heal_manager.get_task_status(task_id).await
+                && matches!(
+                    status,
+                    HealTaskStatus::Completed
+                        | HealTaskStatus::Failed { .. }
+                        | HealTaskStatus::Cancelled
+                        | HealTaskStatus::Timeout
+                )
+            {
+                return status;
            }
            if tokio::time::Instant::now() >= deadline {
-                panic!("heal task {task_id} did not complete within {timeout:?}");
+                panic!("heal task {task_id} did not reach a terminal state within {timeout:?}");
            }
            tokio::time::sleep(Duration::from_millis(100)).await;
        }
    }
+
+    /// Submit an erasure-set heal and drive it to a genuine `Completed`,
+    /// tolerating the healer's by-design "retry scheduled" deferral.
+    ///
+    /// When a per-version heal hits a transient error the erasure-set healer
+    /// persists its resume state and returns a terminal `Failed` carrying
+    /// `retry scheduled` (retry budget still remaining) — expecting a later heal
+    /// run to finish the job. We re-submit ourselves (an idempotent re-heal),
+    /// mirroring the production background scanner. A `Failed` WITHOUT that
+    /// marker (e.g. `exhausted retries`) or any other non-`Completed` terminal
+    /// state is a real failure and panics.
+    async fn drive_heal_to_completion(
+        heal_manager: &HealManager,
+        build_request: impl Fn(bool) -> HealRequest,
+        per_attempt_timeout: Duration,
+        max_redrives: usize,
+    ) {
+        for attempt in 0..=max_redrives {
+            let request = build_request(attempt > 0);
+            let task_id = request.id.clone();
+            let admission = heal_manager
+                .submit_heal_request(request)
+                .await
+                .expect("failed to submit erasure-set heal");
+            assert!(admission.is_admitted(), "erasure-set heal must be admitted");
+
+            match await_terminal_status(heal_manager, &task_id, per_attempt_timeout).await {
+                HealTaskStatus::Completed => return,
+                HealTaskStatus::Failed { error } if error.contains("retry scheduled") => {
+                    info!(attempt, error, "erasure-set heal deferred a transient version; re-driving to completion");
+                    // Brief settle before the next idempotent re-heal.
+                    tokio::time::sleep(Duration::from_millis(200)).await;
+                }
+                HealTaskStatus::Failed { error } => panic!("heal task failed (non-retryable): {error}"),
+                HealTaskStatus::Cancelled => panic!("heal task was cancelled"),
+                other => panic!("heal task reached unexpected terminal state: {other:?}"),
+            }
+        }
+        panic!("erasure-set heal did not reach Completed within {} attempt(s)", max_redrives + 1);
+    }
 }
--- a/Show More
+++ b/Show More