fix(heal): wait for held legacy responsibility in replay test

fix(scanner): diagnose and verify pause backlog recovery (#8293 )
* fix(scanner): diagnose and verify pause backlog recovery Expose the retained replica snapshot and claimed membership in abnormal admin status responses. Keep diagnostics off metrics updates and verify single-pool recovery and conflicting-proof preservation across 24 sets. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(scanner): move replica snapshots into diagnostics Consume the terminal admin read snapshot in a single state match and move membership, revision, and error buffers into the response. Verify buffer handoff and the unchanged JSON contract without altering ledger authority. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 14:27:10 +00:00 · 2026-10-02 03:13:15 +08:00 · 2026-10-02 01:09:00 +08:00 · 2026-10-02 01:00:04 +08:00 · 2026-10-02 00:54:32 +08:00 · 2026-10-02 00:46:57 +08:00
292 changed files with 27016 additions and 3199 deletions
@@ -1,6 +1,6 @@
 ---
 name: rustfs-release-publish
-description: "Run the RustFS console gate, source version bump, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
+description: "Run the RustFS console gate, source version bump, release-branch CI, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
 ---
 # RustFS Release Publish (preview-validated pipeline)

@@ -19,7 +19,9 @@ Pipeline shape:
 check console main against its latest Release
  -> if ahead: publish console -> wait for Release asset + latest API
  -> bump Cargo.toml and Cargo.lock to <target> -> merge
-  -> tag <preview-tag> at that commit -> CI green
+  -> cut release/<target> from a selected main commit containing the bump
+  -> CI green for that exact release-branch commit
+  -> tag <preview-tag> at that commit
  -> verify preview Release assets -> run binary locally + console checks
  -> validate with latest rc client
  -> report preview acceptance results -> STOP for explicit human confirmation
@@ -29,7 +31,7 @@ check console main against its latest Release
  -> update installation references on main -> update rustfs.com announcement
 ```

-On preview validation failure: fix lands on main via normal PR (Cargo versions are already at `<target>`, no new source bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2. Installation references remain on the previous published deliverable throughout preview validation.
+On candidate or preview validation failure: fix lands on main via normal PR, then backport only the needed fix to `release/<target>` through a reviewed PR. Re-run release-branch CI at the new commit and, if a preview was already tagged, tag `<preview-tag N+1>` there and restart acceptance. Installation references remain on the previous published deliverable throughout preview validation.

 ## Required inputs

@@ -68,14 +70,14 @@ Rules:
 - Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
 - Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
 - Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
+- The release branch is the candidate source during multi-day acceptance. Do not merge main wholesale into it, force-push it, or advance it with unrelated changes. The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD or an advanced release-branch HEAD, and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
 - When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
 - Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure blocks dependent steps. For a preview/source acceptance failure before the final tag exists, land the fix, verify Cargo still matches the confirmed target, and restart from Phase 2 with the next preview at the fixed commit. If main has advanced to another target, resolve that source/version mismatch before another preview.
+- Phases run in order; a failure blocks dependent steps. For a candidate or preview/source acceptance failure before the final tag exists, land the fix on main, backport it to the release branch, verify Cargo there still matches the confirmed target, and restart from release-branch CI in Phase 2. Use the next preview iteration if any preview was already tagged. Main may advance to another target without changing this release candidate.
 - Once the final tag exists, preserve its commit. Retry failed publication jobs for that exact tag; do not recreate the tag or restart preview on newer main. If a source change is required after final-tag publication, report the failure and obtain a new target version. Phase 7 failures resume only the failed follow-up after rechecking artifact availability and whether a newer release has superseded it.
 - Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
 - Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
+- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision. Do not delete the release branch or preview tags as part of abandonment without an explicit decision.
 - User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.

 ## Phase 0 — Preflight
@@ -83,6 +85,7 @@ Rules:
 - `git status --short` clean; `git fetch origin main --tags`.
 - `gh auth status` works; confirm you can view `gh release list -L 3`.
 - Confirm the exact final target version with the user if not explicit.
+- Check whether `release/<target>` already exists before creating it; a resumed release reuses the existing branch and its history. For an in-flight release that already has a preview tag but no branch, anchor the new branch to that tag's commit only after verifying its target versions and prior acceptance evidence.

 ### Console release gate

@@ -90,18 +93,19 @@ Read and complete [the Console gate](references/console-gate.md) before Phase 1.

 ## Phase 1 — Source version bump to the final target (once)

- If Cargo.toml and all workspace members in Cargo.lock already read `<target>` (e.g. this is a restart after a failed preview), verify both and skip the source bump. Installation versions intentionally differ during preview; do not treat them as leftovers. If a previous preparation advanced installation references to an unavailable target, restore those references to the verified published baseline before selecting the next preview commit.
- Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- Get the PR merged into main. Record the resulting main commit:
+- If `release/<target>` exists, verify its Cargo.toml and workspace members in Cargo.lock read `<target>` and skip the source bump, even if main has advanced. If the branch versions differ, stop and resolve the mismatch without downgrading main.
+- If no branch exists but an unfinished preview tag does, verify its commit has `<target>` in both version files and skip the bump; Phase 2 anchors the branch to that commit. Stop if those files disagree. Otherwise, if main already has `<target>` in both files, skip the bump. Installation references intentionally retain the previous published deliverable during preview; if they were advanced to an unavailable target, restore that baseline before choosing the next candidate.
+- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
+- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.

-```bash
-git fetch origin main
-PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
-```
+Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.

-`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
+## Phase 2 — Pin and validate the release candidate, then publish the preview tag

-## Phase 2 — Publish the preview tag
+- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
+- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
+- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
+- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.

 ```bash
 git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
@@ -112,11 +116,12 @@ Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `d

 The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.

-On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
+On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.

 ## Phase 3 — CI and preview Release verification

 - Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
+- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
 - Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
 - Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
 - Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
@@ -128,13 +133,13 @@ Read and complete [preview acceptance](references/preview-acceptance.md): verify

 ### Manual confirmation gate

-After every Phase 3–5 check passes, report the target, preview tag, `PREVIEW_HASH`, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
+After every Phase 3–5 check passes, verify the release branch still points to `PREVIEW_HASH`. Report the target, release branch, preview tag, `PREVIEW_HASH`, release-branch CI run, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.

 Continue to Phase 6 only after a new user reply explicitly confirms the reported target, preview tag, and commit. A clear affirmative reply to that exact report, such as `确认继续`, is sufficient; if the reply is ambiguous or any reported value changed, ask again.

 ## Phase 6 — Publish the final tag on the validated commit

-No second source version bump. The final tag goes on the exact commit the preview validated:
+No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:

 ```bash
 git fetch origin --tags
@@ -158,7 +163,7 @@ Only after Phase 6 succeeds, complete [post-release updates](references/post-rel
 Always report:

 - Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
+- Target version, release branch, release-branch CI run and validated SHA, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
 - Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
 - Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
 - Post-release installation PR and verification results; website banner target, text, link, PR, and observed deployment state. Report any remaining merge authorization or failed deployment explicitly rather than claiming the banner is live.
@@ -1,2 +1,2 @@
-sha256-linux=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
-sha256-darwin=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
+sha256-linux=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
+sha256-darwin=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
@@ -1,2 +1,2 @@
-sha256-darwin=9e9687ded961aa4e619a15def6e1ee0a5153be05423d009ee7a58126624c83e7
-sha256-linux=364666693c2bf258997643c9d87e07e7b7501e58175f190d201d53c81946eb20
+sha256-darwin=14c8cd76b8cbf643329c371daa70a5edd4357d42259a2fd9c653e548675d4a6d
+sha256-linux=80d0fae627007c31f250fb1fb21c8a952861195d53d7f60b6e71636d5ca65b46
@@ -1,2 +1,2 @@
-sha256-darwin=83a7dcaffd5a789517ae9f02a224f66a9713937885cff96fca2ad7e216f197ae
-sha256-linux=626c10f8c964507ff987b6c86069e9019dc6d2ae7fb02db9be5df5aa8cc5145b
+sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
+sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
@@ -1 +1 @@
-sha256=548a58d74f3cb3b5a1ab9dcd4d3f8625b9ddebe847a2e14de1d08c3b2e6292eb
+sha256=476cb92aa9e385c9bb509ee5df3e5bd07e8c929730040b60d16791b6c0616d00
@@ -52,6 +52,7 @@ script-tests: ## Run shell script tests
 	./scripts/check_embedded_secrets.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
@@ -61,6 +62,7 @@ script-tests: ## Run shell script tests
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
 	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
 	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
@@ -82,6 +82,13 @@ setup = 'lifecycle-large-stack'
 filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
 setup = 'ecstore-large-stack'

+# These real-disk scanner fixtures assert quiet-workload publication and
+# durable MRF admission. Reserve the pool while their bounded CAS/fsync
+# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
+[[profile.default.overrides]]
+filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
+threads-required = "num-test-threads"
+
 [[profile.default.overrides]]
 filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
@@ -107,6 +114,16 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
 test-group = 'ecstore-serial-flaky'

+# Serialize the replacement bucket metadata heal test. It builds a real-disk
+# hermetic erasure set, writes bucket metadata, then asserts byte-identical
+# xl.meta readback after a rejected delayed repair. Under parallel load,
+# concurrent ecstore tests mutate the process-wide disk registry and alter
+# the serialized timestamp/disk-identity fields, flipping the assertion.
+# Preventive serialization only, no retries.
+[[profile.default.overrides]]
+filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
+test-group = 'ecstore-serial-flaky'
+
 # Serialize the metadata-cache generation-retirement pair. Both carry
 # #[serial(metadata_cache_invalidation_probe)] — a no-op across nextest's
 # process boundary — and assert get_object_metadata_cache generation
@@ -132,10 +149,37 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
 test-group = 'embedded-test-ports'

-# The real object probe includes bucket creation and cleanup in its short
-# measurement budget. Keep competing storage fixtures outside that budget.
+# Registration, inventory delivery, and real drive/object probes include
+# durable filesystem IO in short deadlines. Reserve capacity so sibling test
+# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
 [[profile.default.overrides]]
-filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
+filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+threads-required = "num-test-threads"
+
+# This cancellation probe hashes the serving executable before acquiring its
+# collector lease. Keep competing test processes out of its 30s startup budget.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
+threads-required = "num-test-threads"
+
+# These real IAM/state fixtures keep production lock-acquisition limits while
+# peers complete durable IO. Reserve capacity from unrelated test processes;
+# the shared fixture's durable_iam_state_ prefix keeps the whole family together.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
+threads-required = "num-test-threads"
+
+# Real target-repair and rebalance-stop fixtures commit durable metadata while
+# peers wait on production locks or bounded cancellation checks. Reserve IO
+# capacity for each fixture; its internal races and deadlines remain intact.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
+threads-required = "num-test-threads"
+
+# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
+# Their existing deadlines, assertions, and retry policy remain in force.
+[[profile.default.overrides]]
+filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
 threads-required = "num-test-threads"

 # Serialize the durable manual-transition checkpoint test across nextest's
@@ -218,6 +262,13 @@ test-group = 'e2e-cluster-nightly'
 filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
 test-group = 'e2e-vault'

+# These server-heavy smoke cases perform 1k+ sequential S3 writes or run the
+# scanner and a remote tier server together. Reserve the full test budget so
+# unrelated E2E processes cannot starve their storage and polling deadlines.
+[[profile.default.overrides]]
+filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
+threads-required = "num-test-threads"
+
 # This four-disk, 65-member rollback probe already drives up to 32 concurrent
 # durable deletions. Reserve this nextest run's capacity for its progress oracle.
 [[profile.default.overrides]]
@@ -257,6 +308,13 @@ setup = 'lifecycle-large-stack'
 filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
 setup = 'ecstore-large-stack'

+# These real-disk scanner fixtures assert quiet-workload publication and
+# durable MRF admission. Reserve the pool while their bounded CAS/fsync
+# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
+threads-required = "num-test-threads"
+
 # ===========================================================================
 # QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
 #
@@ -282,13 +340,6 @@ retries = 2
 filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
 test-group = 'ecstore-serial-flaky'

-# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
-# on producer/consumer timing windows that stretch past the budget on loaded
-# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
-retries = 2
-
 # Serialize the relocated-pool GET resume regression under the ci profile too
 # (see the matching default-profile override near the top). No longer a
 # quarantine: the fixture race (rustfs#6701/rustfs#6703) was fixed by #6707,
@@ -320,6 +371,11 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
 test-group = 'ecstore-serial-flaky'

+# Serialize the replacement bucket metadata heal test under the ci profile too.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
+test-group = 'ecstore-serial-flaky'
+
 # Serialize the metadata-cache generation-retirement pair under the ci
 # profile too (see the matching default-profile override near the top). Not a
 # quarantine: no retries.
@@ -334,7 +390,30 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
 test-group = 'embedded-test-ports'

 [[profile.ci.overrides]]
-filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
+filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
+threads-required = "num-test-threads"
+
+# Preserve the cancellation probe's startup budget and assertions in CI too.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
+threads-required = "num-test-threads"
+
+# Keep the same IAM/state fixture capacity reservation in CI without changing the
+# production lock deadline, internal concurrency, assertions, or retry policy.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
+threads-required = "num-test-threads"
+
+# Match the real admin fixture capacity reservation without retries or changes
+# to production lock deadlines, cancellation barriers, or assertions.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
+threads-required = "num-test-threads"
+
+# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
+# Their existing deadlines, assertions, and retry policy remain in force.
+[[profile.ci.overrides]]
+filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
 threads-required = "num-test-threads"

 # Serialize the durable manual-transition checkpoint test under the ci profile
@@ -468,6 +547,7 @@ default-filter = """
        | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
        | test(/^reliant::lifecycle::/)
        | test(/^reliant::tiering::/)
+        | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
        | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
    )
 """
@@ -476,13 +556,16 @@ fail-fast = false
 [profile.e2e-smoke.junit]
 path = "junit.xml"

-# The pagination boundary cases can stall when a server/listing regression
-# prevents the continuation request from completing. Keep the timeout scoped
-# to those known failure modes so legitimate lifecycle/tiering waits retain
-# their test-level timing budget.
+# Bound the whole pagination fixture, including startup, seeding, and cleanup;
+# its LIST deadline begins only after the seed objects have been written.
 [[profile.e2e-smoke.overrides]]
 filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
 slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
+threads-required = "num-test-threads"
+
+[[profile.e2e-smoke.overrides]]
+filter = 'package(e2e_test) & test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry)'
+threads-required = "num-test-threads"

 # ---------------------------------------------------------------------------
 # e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
@@ -692,6 +775,7 @@ default-filter = """
    & !test(/^replication_extension_test::/)
    & !test(/^replication_target_matrix_test::/)
    & !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
+    & !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
 """
 fail-fast = false

@@ -720,3 +804,9 @@ test-group = 'e2e-cluster-nightly'
 [[profile.e2e-full.overrides]]
 filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
 test-group = 'e2e-vault'
+
+# Keep the storage-heavy smoke cases isolated in the full suite too. They
+# start multiple server fixtures and issue many sequential S3 operations.
+[[profile.e2e-full.overrides]]
+filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
+threads-required = "num-test-threads"
@@ -100,12 +100,7 @@ runs:

    - name: Check test wiring
      shell: bash
-      run: |
-        python3 ./scripts/check_test_wiring.py --self-test
-        python3 ./scripts/check_scheduled_validation_freshness.py --self-test
-        python3 ./scripts/test_security_workflow.py
-        python3 ./scripts/test_nightly_candidate.py
-        python3 ./scripts/check_test_wiring.py
+      run: python3 ./scripts/check_test_wiring.py

    - name: Check no planning docs committed
      shell: bash
@@ -107,6 +107,8 @@ runs:
    - name: Install cargo-nextest
      if: inputs.install-test-tools == 'true'
      uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
+      with:
+        tool: cargo-nextest@0.9.138

    - name: Setup Rust cache
      uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
@@ -1,61 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Architecture Migration Rules
-
-on:
-  pull_request:
-    types: [ opened, synchronize, reopened, closed ]
-    branches: [ main ]
-    paths:
-      - "ARCHITECTURE.md"
-      - "docs/architecture/**"
-      - "scripts/check_architecture_migration_rules.sh"
-      - ".github/workflows/architecture-migration-rules.yml"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  cancel-closed-pr-runs:
-    name: Cancel Closed PR Runs
-    if: github.event_name == 'pull_request' && github.event.action == 'closed'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - name: Explain cancellation run
-        run: echo "PR closed; this run only cancels older runs in the same concurrency group."
-
-  architecture-migration-rules:
-    name: Architecture Migration Rules
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-
-      - name: Install ripgrep
-        uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
-        with:
-          tool: ripgrep@15.2.0
-
-      - name: Check architecture migration rules
-        run: ./scripts/check_architecture_migration_rules.sh
@@ -204,7 +204,7 @@ jobs:
    # open the Actions tab. Same ci-8 mechanism coverage.yml and
    # e2e-replication-nightly.yml already use.
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -27,7 +27,8 @@
 #   schedule and manual dispatch build the full platform matrix.
 #
 # Manual Parameters:
-# - build_docker: Build and push Docker images (default: true)
+# - publish: Publish binaries/releases for manual runs (default: false)
+# - build_docker: Advisory only; manual runs do not trigger Docker publishing
 # - platforms: Comma-separated platform IDs or 'all' (default: all)

 name: Build and Release
@@ -55,12 +56,16 @@ on:
    - cron: "13 1 * * 0" # Weekly on Sunday 01:13 UTC
  workflow_dispatch:
    inputs:
+      publish:
+        description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
+        required: false
+        default: false
+        type: boolean
      build_docker:
        # Advisory only. docker.yml triggers on workflow_run and its job-level
        # condition requires the triggering event to be a tag push, so a manual
        # dispatch of this workflow never produces images regardless of this
-        # value. Kept because the summary step reports it; wiring it up would
-        # mean teaching docker.yml's version parser a second event shape.
+        # value. Publishing remains controlled by the explicit publish input.
        description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
        required: false
        default: true
@@ -92,6 +97,7 @@ jobs:
    timeout-minutes: 30
    outputs:
      should_build: ${{ steps.check.outputs.should_build }}
+      should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
      build_type: ${{ steps.check.outputs.build_type }}
      version: ${{ steps.check.outputs.version }}
      short_sha: ${{ steps.check.outputs.short_sha }}
@@ -208,9 +214,9 @@ jobs:

          all='{"include":[
            {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":"","features":""},
+            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
            {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope"},
+            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
          ]}'
@@ -402,6 +408,8 @@ jobs:

      - name: Build RustFS
        shell: bash
+        env:
+          BUILD_TARGET: ${{ matrix.target }}
        run: |
          # Force rebuild by touching build.rs
          touch rustfs/build.rs
@@ -418,14 +426,37 @@ jobs:
          # Build one binary at a time so release LTO links cannot exhaust a
          # self-hosted runner by running concurrently.
          for binary in "${BINARY_NAMES[@]}"; do
+            PROFILE_ARGS=()
+            if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+              # The in-process profiler resolves names from the final executable.
+              # Retain function symbols without enabling DWARF source information.
+              PROFILE_ARGS=(--config 'profile.release.strip="none"')
+            fi
            if [[ "${{ matrix.cross }}" == "true" ]]; then
              # All cross targets in the matrix are Linux; zigbuild handles them.
-              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
+              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
            else
-              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
+              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
            fi
          done

+      - name: Generate the CPU symbol catalog
+        if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
+        shell: bash
+        run: |
+          cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
+            --features rustfs/pyroscope > "$RUNNER_TEMP/cpu-catalog-metadata.json"
+          features=$(jq -ce '
+            (.packages[] | select(.name == "rustfs") | .id) as $id |
+            .resolve.nodes[] | select(.id == $id) | .features | map(select(. != "default")) | sort
+          ' "$RUNNER_TEMP/cpu-catalog-metadata.json")
+          release_dir="target/${{ matrix.target }}/release"
+          # These task-owned outputs may remain in a restored target cache.
+          rm -f "$release_dir/rustfs.cpu-symbol-catalog.json" "$release_dir/rustfs.cpu-symbol-catalog.sha256"
+          cargo run --locked -p rustfs-connect-symbol-catalog -- \
+            "$release_dir/rustfs" "$(git rev-parse HEAD)" '${{ matrix.target }}' "$features" \
+            "$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
+
      - name: Create release package
        id: package
        shell: bash
@@ -516,6 +547,10 @@ jobs:
            BINARY_FILES+=("rustfs-cli")
          fi

+          if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+            BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
+          fi
+
          # Verify the binaries exist before packaging
          for binary in "${BINARY_FILES[@]}"; do
            if [[ ! -f "$binary" ]]; then
@@ -625,7 +660,7 @@ jobs:
          echo "🔧 Build type: ${BUILD_TYPE}"
          echo "📊 Version: ${VERSION}"

-      - name: Verify packaged Linux CLI
+      - name: Verify packaged Linux binaries
        if: matrix.platform == 'linux'
        shell: bash
        run: |
@@ -634,21 +669,27 @@ jobs:
          package_file="${{ steps.package.outputs.package_file }}"
          archive_members="$(unzip -Z1 "$package_file" | sort)"
          expected_members=$'rustfs\nrustfs-cli'
+          target="${{ matrix.target }}"
+          features="${{ matrix.features }}"
+          if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
+            expected_members+=$'\nrustfs.cpu-symbol-catalog.json\nrustfs.cpu-symbol-catalog.sha256'
+          fi
          if [[ "$archive_members" != "$expected_members" ]]; then
-            echo "Linux package must contain exactly rustfs and rustfs-cli" >&2
+            echo "Linux package does not contain the expected files" >&2
            printf '%s\n' "$archive_members" >&2
            exit 1
          fi

-          if [[ "${{ matrix.target }}" == "x86_64-unknown-linux-gnu" ]]; then
-            cli_dir="$(mktemp -d)"
-            trap 'rm -rf "$cli_dir"' EXIT
-            unzip -q "$package_file" rustfs-cli -d "$cli_dir"
-            "$cli_dir/rustfs-cli" --help >/dev/null
-          fi
+          # Every Linux target builds natively, so executing the binaries here
+          # proves each artifact matches its runner architecture.
+          cli_dir="$(mktemp -d)"
+          trap 'rm -rf "$cli_dir"' EXIT
+          unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
+          "$cli_dir/rustfs" --version
+          "$cli_dir/rustfs-cli" --help >/dev/null

      - name: Verify packaged console
-        if: matrix.target == 'x86_64-unknown-linux-gnu'
+        if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
        shell: bash
        run: |
          set -euo pipefail
@@ -709,7 +750,7 @@ jobs:
          retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}

      - name: Upload to Cloudflare R2
-        if: env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
+        if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
        env:
          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -726,12 +767,33 @@ jobs:
          fi

          # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
-          # images did). Install it on demand so R2 uploads survive a fresh runner
-          # instead of hard-failing here.
+          # images did). Install the official CLI bundle on demand so R2 uploads
+          # survive a fresh runner without relying on a distro package.
          if ! command -v aws >/dev/null 2>&1; then
            echo "aws CLI not found on runner; installing..."
            if command -v apt-get >/dev/null 2>&1; then
-              sudo apt-get update && sudo apt-get install -y awscli
+              aws_arch="$(uname -m)"
+              case "$aws_arch" in
+                x86_64) aws_arch="x86_64" ;;
+                aarch64|arm64) aws_arch="aarch64" ;;
+                *)
+                  echo "❌ Unsupported Linux architecture for AWS CLI: $aws_arch"
+                  exit 1
+                  ;;
+              esac
+
+              aws_install_root="${RUNNER_TEMP:-$PWD}/aws-cli"
+              rm -rf "$aws_install_root"
+              mkdir -p "$aws_install_root"
+              curl --fail --location --retry 3 \
+                "https://awscli.amazonaws.com/awscli-exe-linux-${aws_arch}.zip" \
+                --output "$aws_install_root/awscliv2.zip"
+              unzip -q "$aws_install_root/awscliv2.zip" -d "$aws_install_root"
+              "$aws_install_root/aws/install" \
+                --install-dir "$aws_install_root/aws-cli" \
+                --bin-dir "$aws_install_root/bin"
+              export PATH="$aws_install_root/bin:$PATH"
+              aws --version
            elif command -v brew >/dev/null 2>&1; then
              brew install awscli
            else
@@ -776,62 +838,35 @@ jobs:
      - name: Build completion summary
        shell: bash
        env:
-          # dispatch input via env: free-form text must not be pasted into the
-          # script for bash to evaluate.
-          INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
+          SHOULD_PUBLISH: ${{ needs.build-check.outputs.should_publish }}
+          BUILD_TYPE: ${{ needs.build-check.outputs.build_type }}
+          VERSION: ${{ needs.build-check.outputs.version }}
+          BUILD_STATUS: ${{ needs.build-rustfs.result }}
+          EVENT_NAME: ${{ github.event_name }}
        run: |
-          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
-          VERSION="${{ needs.build-check.outputs.version }}"
-
-          echo "🎉 Build completed successfully!"
-          echo "📦 Build type: $BUILD_TYPE"
-          echo "🔢 Version: $VERSION"
-          echo ""
-
-          # Check build status
-          BUILD_STATUS="${{ needs.build-rustfs.result }}"
-
-          echo "📊 Build Results:"
-          echo "  📦 All platforms: $BUILD_STATUS"
-          echo ""
-
-          case "$BUILD_TYPE" in
-            "preview")
-              echo "🔍 Preview artifacts are published in a GitHub prerelease"
-              echo "⏭️  Preview releases do not update latest channels"
-              ;;
-            "development")
-              echo "🛠️  Development build artifacts have been uploaded to OSS dev directory"
-              echo "⚠️  This is a development build - not suitable for production use"
-              ;;
-            "release")
-              echo "🚀 Release build artifacts have been uploaded to OSS release directory"
-              echo "✅ This build is ready for production use"
-              echo "🏷️  GitHub Release will be created in this workflow"
-              ;;
-            "prerelease")
-              echo "🧪 Prerelease build artifacts have been uploaded to OSS release directory"
-              echo "⚠️  This is a prerelease build - use with caution"
-              echo "🏷️  GitHub Release will be created in this workflow"
-              ;;
-          esac
-
-          echo ""
-          echo "🐳 Docker Images:"
-          if [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
-            echo "⏭️  Docker image build was skipped (binary only build)"
+          echo "Build type: $BUILD_TYPE"
+          echo "Version: $VERSION"
+          echo "Build result: $BUILD_STATUS"
+          if [[ "$SHOULD_PUBLISH" != "true" ]]; then
+            echo "Artifact-only run: no download channels or GitHub Releases are updated."
+            echo "Successful platform packages are available as workflow artifacts."
          elif [[ "$BUILD_STATUS" == "success" ]]; then
-            echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
+            echo "Publishing enabled; check the upload and release jobs for their results."
          else
-            echo "❌ Docker image build will be skipped due to build failure"
+            echo "Build incomplete; check individual platform jobs for any completed uploads."
+          fi
+          if [[ "$EVENT_NAME" == "push" && "$GITHUB_REF" == refs/tags/* && "$BUILD_STATUS" == "success" ]]; then
+            echo "Docker publishing is eligible through the downstream workflow."
+          else
+            echo "This run does not trigger Docker publishing."
          fi

  # Create GitHub Release for every valid release tag, including previews
  create-release:
    name: Create GitHub Release
    needs: [ build-check, build-rustfs ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: sm-standard-2
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
    timeout-minutes: 30
    permissions:
      contents: write
@@ -891,8 +926,8 @@ jobs:
  upload-release-assets:
    name: Upload Release Assets
    needs: [ build-check, build-rustfs, create-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: sm-standard-2
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
    timeout-minutes: 30
    permissions:
      contents: write
@@ -985,7 +1020,7 @@ jobs:
  update-latest-version:
    name: Update Latest Version
    needs: [ build-check, publish-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
    runs-on: sm-standard-2
    timeout-minutes: 30
    steps:
@@ -1045,8 +1080,8 @@ jobs:
  publish-release:
    name: Publish Release
    needs: [ build-check, create-release, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: sm-standard-2
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
    timeout-minutes: 30
    permissions:
      contents: write
@@ -1083,8 +1118,8 @@ jobs:
  cleanup-preview-releases:
    name: Cleanup Preview Releases
    needs: [ build-check, publish-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: sm-standard-2
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
    timeout-minutes: 30
    permissions:
      contents: write
@@ -1132,7 +1167,7 @@ jobs:
    if: >-
      always() && github.event_name == 'schedule' &&
      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -15,16 +15,16 @@
 # Sole writer of the Rust dependency caches that ci.yml restores.
 #
 # Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
-# concurrency group cancels in-progress runs on main pushes, and merges land far
-# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
+# concurrency group originally cancelled in-progress main runs, while merges
+# landed faster than its 70-minute pipeline. Over 15 consecutive main pushes:
 # 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
 # Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
 # so the writer lanes were saving nothing and every PR paid a cold restore —
 # 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
 #
-# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
-# superseded runs (which is correct — nobody needs test results for a commit
-# that is already three merges behind) while the caches still get written.
+# Keep cache writing separate from validation: only these jobs publish the
+# complete feature closure for each key. Main validation now also finishes its
+# running baseline, while superseded PR attempts can still be cancelled.
 #
 # The group below deliberately does NOT cancel in progress; see the comment on
 # it for how that bounds concurrency and why it is scoped by event.
@@ -256,7 +256,7 @@ jobs:
  # belonged to it.
  warm-ci-uring:
    name: Warm ci-uring
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - name: Checkout repository
@@ -47,12 +47,12 @@ permissions:
  contents: read

 # Concurrency groups are scoped per event so different triggers never cancel
-# each other: PR pushes cancel the previous run of that PR, main pushes keep
-# latest-wins semantics among themselves, and scheduled runs always complete
-# (a shared group used to let every merge kill the weekly scheduled run).
+# each other. PR pushes cancel superseded attempts; main and scheduled runs
+# finish so a busy merge stream cannot starve the complete baseline. For main,
+# GitHub retains one running run and replaces the pending run with the latest.
 concurrency:
  group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: ${{ github.event_name != 'schedule' }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

 env:
  CARGO_TERM_COLOR: always
@@ -128,7 +128,7 @@ jobs:

  test-and-lint:
    name: Workspace Test and Lint
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
    needs: [ quick-checks, classify-changes ]
    runs-on: sm-standard-4
    timeout-minutes: 90
@@ -146,11 +146,9 @@ jobs:
        uses: ./.github/actions/setup
        with:
          rust-version: stable
-          # Every lane in this workflow reads its cache and none writes it.
-          # cache-warm.yml is the sole writer for all four keys: this workflow
-          # cancels superseded runs on main, and a cancelled run never reaches
-          # rust-cache's post step, so writing from here saved nothing (12 of 15
-          # consecutive main-push runs were cancelled). See rustfs/backlog#1600.
+          # Every lane reads its cache; cache-warm.yml remains the sole writer
+          # with the matching feature closure. Tests never compete to replace
+          # shared caches, including when a PR attempt is cancelled.
          cache-shared-key: ci-dev
          cache-save-if: 'false'
          install-build-packaging-tools: 'false'
@@ -195,13 +193,10 @@ jobs:
          # Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the
          # merge path is untouched while the experiment runs.
          #
-          # Dispatch is included because push alone cannot supply the samples:
-          # this workflow cancels superseded runs on main, and only 4 of the last
-          # 20 push-triggered Test and Lint jobs reached a terminal state — at
-          # that rate ten samples would take roughly fifty merges. The
-          # concurrency group is scoped by event_name, so a dispatched run has
-          # its own group and is not cancelled by merge traffic, which makes the
-          # sample collectable on demand rather than by waiting.
+          # During baseline collection, main pushes cancelled superseded runs:
+          # only 4 of 20 Test and Lint jobs reached a terminal state. Running
+          # main validations now finish; dispatch remains an independent group
+          # for collecting additional samples on demand.
          #
          # Baseline over 17 samples at 2:
          # median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2
@@ -250,11 +245,11 @@ jobs:
          mkdir -p artifacts/test-and-lint
          set +e
          timeout --verbose --signal=TERM --kill-after=30s 15m \
-            cargo test --all --doc \
+            cargo test --all --exclude e2e_test --doc \
            2>&1 | tee artifacts/test-and-lint/doctest.log
          status=${PIPESTATUS[0]}
          {
-            echo "command=cargo test --all --doc"
+            echo "command=cargo test --all --exclude e2e_test --doc"
            echo "exit_status=${status}"
            echo "finished_at=$(date --utc --iso-8601=seconds)"
            echo
@@ -281,21 +276,10 @@ jobs:
      - name: Check log-analyzer rule anchors
        run: ./scripts/check_log_analyzer_rules.sh

-      # Explicit gate for migration-critical suites. These tests already ran in
-      # the full nextest pass above; a single filtered nextest invocation keeps
-      # the named gate without rebuilding or re-running them one package at a time.
-      #
-      # The gate selects tests by name substring (data_movement / rebalance /
-      # decommission / source_cleanup / delete_marker), so renames can silently
-      # thin it. The script owns the filter expression and first verifies the
-      # selected-test count against the committed floor in
-      # .config/migration-gate-floor.txt before running the gate; renames or
-      # removals must update that file consciously (see the script header).
-      # Kept on the default profile (no --profile ci): a second --profile ci run
-      # would clobber target/nextest/ci/junit.xml, and none of these tests are
-      # quarantined so they gain nothing from the ci profile's retry overrides.
-      - name: Run rebalance/decommission migration proofs
-        run: ./scripts/check_migration_gate_count.sh
+      # Preserve the migration floor and require successful execution evidence
+      # from the workspace run, without compiling or executing its subset again.
+      - name: Verify rebalance/decommission migration proofs
+        run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml

  # This gate builds into a fresh target directory to isolate the E2E root.
  # Give its cold build a separate budget from workspace tests and migration proofs.
@@ -431,6 +415,7 @@ jobs:
        run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings

      - name: Run rio-v2 feature tests
+        shell: bash
        env:
          # Match the main nextest lane's #5394 link-I/O guard. A cold feature
          # cache otherwise fans out enough rust-lld processes to exhaust this
@@ -440,9 +425,25 @@ jobs:
          # --profile ci so the quarantine list (and its junit flaky markers)
          # covers this leg too; the default profile is the local no-retry
          # profile and silently ignored quarantined flakes here (rustfs#6703).
-          cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2
+          mkdir -p artifacts/test-and-lint
+          rm -f target/nextest/ci/junit.xml
+          ./scripts/ci/resource_sampler.sh start nextest-rio-v2
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2 \
+            2>&1 | tee artifacts/test-and-lint/nextest.log
          cargo test -p rustfs --doc --features rio-v2

+      - name: Upload rio-v2 test reports and diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: junit-test-and-lint-rio-v2-${{ github.run_number }}
+          path: |
+            target/nextest/ci/junit.xml
+            artifacts/test-and-lint
+          retention-days: 3
+          if-no-files-found: error
+
  connect-short-credential-boundary:
    name: Connect Short Credential Boundary
    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
@@ -508,13 +509,9 @@ jobs:
    runs-on: sm-standard-4
    timeout-minutes: 90
    strategy:
-      # On a PR, one failing protocol leg is enough to know the PR is not ready,
-      # so stop the sibling leg instead of paying another ~40 minutes for it.
-      # Everywhere else (main pushes, the merge queue, the weekly schedule) keep
-      # the full signal: there we want to know whether swift AND sftp are broken,
-      # not just whichever failed first. This is the only part of the early-stop
-      # work that also covers fork PRs, since it needs no token.
-      fail-fast: ${{ github.event_name == 'pull_request' }}
+      # Preserve both independent results when one protocol fails, so diagnosis
+      # and a focused fix do not require rebuilding an interrupted sibling.
+      fail-fast: false
      matrix:
        features:
          - name: swift
@@ -537,27 +534,50 @@ jobs:
          cache-save-if: 'false'
          install-build-packaging-tools: 'false'

-      - name: Protect Connect test home
-        run: chmod go-w "$(realpath "$HOME")"
+      - name: Prepare protocol test state
+        run: |
+          rm -f target/nextest/ci/junit.xml
+          chmod go-w "$(realpath "$HOME")"

      - name: Run clippy with ${{ matrix.features.name }}
        run: |
+          ./scripts/ci/resource_sampler.sh start clippy
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
          cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings

      - name: Run tests with ${{ matrix.features.name }}
+        id: protocol-tests
+        shell: bash
        env:
          # Keep feature-test linking under the same bounded concurrency as the
          # main nextest lane; Clippy is metadata-only and needs no such limit.
          CARGO_BUILD_JOBS: "2"
+          SAMPLER_INTERVAL_SECS: "15"
        run: |
          # --profile ci so the quarantine list (and its junit flaky markers)
          # covers this leg too; the default profile is the local no-retry
          # profile and silently ignored quarantined flakes here (rustfs#6703).
-          cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }}
+          mkdir -p artifacts/protocol-tests
+          ./scripts/ci/resource_sampler.sh start nextest
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
+          cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }} \
+            2>&1 | tee artifacts/protocol-tests/nextest.log
+
+      - name: Upload protocol test reports and diagnostics
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: junit-test-and-lint-${{ matrix.features.name }}-${{ github.run_number }}-${{ github.run_attempt }}
+          path: |
+            target/nextest/ci/junit.xml
+            artifacts/protocol-tests
+            artifacts/test-and-lint
+          retention-days: 3
+          if-no-files-found: warn

  build-rustfs-debug-binary:
    name: Build RustFS Debug Binary
-    if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
+    if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
    needs: [ quick-checks, classify-changes ]
    runs-on: sm-standard-4
    timeout-minutes: 30
@@ -810,6 +830,10 @@ jobs:
          python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
          ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"

+      # Keep E2E documentation coverage with its already-built package graph.
+      - name: Run E2E documentation tests
+        run: cargo test -p e2e_test --doc
+
      # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
      # profile.e2e-smoke default-filter in .config/nextest.toml is the single
      # wiring mechanism for e2e tests in CI — extend that filter instead of
@@ -819,18 +843,22 @@ jobs:
        env:
          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
+          SAMPLER_INTERVAL_SECS: "15"
        run: |
+          ./scripts/ci/resource_sampler.sh start e2e-smoke
+          trap './scripts/ci/resource_sampler.sh stop' EXIT
          python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
            --status-level all --final-status-level all --failure-output final

      - name: Upload e2e smoke diagnostics
-        if: failure()
+        if: always()
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: e2e-smoke-diagnostics-${{ github.run_number }}
          path: |
            ${{ runner.temp }}/rustfs-e2e-smoke-logs/
            ${{ runner.temp }}/rustfs-e2e-smoke-list.json
+            artifacts/test-and-lint
          if-no-files-found: warn

      - name: Upload e2e smoke JUnit report
@@ -1005,6 +1033,7 @@ jobs:
          RUSTFS_E2E_STARTUP_CAS_BINARY: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs
          RUSTFS_E2E_STARTUP_CAS_BUILD_MANIFEST: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs.e2e-startup-cas-build.json
          RUSTFS_E2E_STARTUP_CAS_ARTIFACT_DIR: ${{ runner.temp }}/rustfs-startup-cas-evidence
+          RUSTFS_HEAL_CHAOS_LOG_ROOT: ${{ runner.temp }}/rustfs-heal-chaos-logs
        run: python3 scripts/e2e_binary.py run --binary "$RUSTFS_E2E_STARTUP_CAS_BINARY" --features e2e-test-hooks -- cargo nextest run --profile e2e-full -p e2e_test

      - name: Upload junit
@@ -1028,6 +1057,15 @@ jobs:
          if-no-files-found: warn
          retention-days: 7

+      - name: Upload heal cluster logs
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: e2e-full-heal-logs-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-heal-chaos-logs
+          if-no-files-found: warn
+          retention-days: 7
+
  e2e-tests-rio-v2:
    name: End-to-End Tests (rio-v2)
    # Inherits the schedule/dispatch-only gate through needs: on every other
@@ -1270,7 +1308,7 @@ jobs:
    if: >-
      always() && github.event_name == 'schedule' &&
      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -60,7 +60,7 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

@@ -0,0 +1,236 @@
+# Copyright 2026 RustFS Team
+# SPDX-License-Identifier: Apache-2.0
+
+name: Connect bounded health service acceptance
+
+on:
+  workflow_dispatch:
+    inputs:
+      build_run_id:
+        description: Successful main-branch Build and Release run ID
+        required: true
+        type: string
+      artifact_id:
+        description: Linux x86_64 GNU artifact from that run
+        required: true
+        type: string
+      source_sha:
+        description: Exact RustFS source commit
+        required: true
+        type: string
+      artifact_digest:
+        description: GitHub artifact digest including sha256 prefix
+        required: true
+        type: string
+      binary_sha256:
+        description: Independently verified rustfs binary SHA-256
+        required: true
+        type: string
+      connect_sha:
+        description: Exact Connect health service acceptance harness commit
+        required: true
+        type: string
+
+permissions:
+  actions: read
+  contents: read
+
+jobs:
+  health-service:
+    name: Verify bounded health service over mTLS
+    runs-on: dind-sm-standard-2
+    timeout-minutes: 45
+    steps:
+      - name: Checkout exact RustFS source
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          path: rustfs-source
+          persist-credentials: false
+          ref: ${{ inputs.source_sha }}
+
+      - name: Checkout exact Connect harness
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          repository: rustfs/connect
+          path: connect-harness
+          persist-credentials: false
+          ref: ${{ inputs.connect_sha }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+
+      - name: Set up Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 25.8.1
+          cache: npm
+          cache-dependency-path: connect-harness/web/package-lock.json
+
+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
+
+      - name: Verify official source and artifact identity
+        shell: bash
+        env:
+          GH_TOKEN: ${{ github.token }}
+          BUILD_RUN_ID: ${{ inputs.build_run_id }}
+          ARTIFACT_ID: ${{ inputs.artifact_id }}
+          SOURCE_SHA: ${{ inputs.source_sha }}
+          ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_SHA: ${{ inputs.connect_sha }}
+        run: |
+          set -euo pipefail
+          [[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
+          [[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
+          [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
+          [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
+          [[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
+          [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
+          [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
+          [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
+          [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
+          run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
+          jq -e --arg source "$SOURCE_SHA" '
+            .head_sha == $source and .head_branch == "main"
+            and .head_repository.full_name == "rustfs/rustfs"
+            and .name == "Build and Release" and .path == ".github/workflows/build.yml"
+            and .status == "completed" and .conclusion == "success"
+          ' <<<"$run" >/dev/null
+          jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
+          jq -e '
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            | ($matches | length) == 1 and $matches[0].conclusion == "success"
+          ' <<<"$jobs" >/dev/null
+          artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
+          jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
+            .workflow_run.id == $run and .workflow_run.head_sha == $source
+            and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
+            and .digest == $digest and .expired == false
+            and (.expires_at | fromdateiso8601) > now
+            and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
+          ' <<<"$artifact" >/dev/null
+          jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
+          gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
+          [[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
+          printf '%s  official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
+
+      - name: Extract only the verified official service binary
+        shell: bash
+        env:
+          SOURCE_SHA: ${{ inputs.source_sha }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+        run: |
+          set -euo pipefail
+          python3 - <<'PY'
+          import io
+          import os
+          import pathlib
+          import shutil
+          import stat
+          import zipfile
+
+          def regular(entry):
+              mode = entry.external_attr >> 16
+              return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
+
+          package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
+          with zipfile.ZipFile('official-artifact.zip') as outer:
+              entries = outer.infolist()
+              assert 1 <= len(entries) <= 16
+              assert len({entry.filename for entry in entries}) == len(entries)
+              assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
+              assert sum(entry.file_size for entry in entries) <= 2147483648
+              assert outer.testzip() is None
+              with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
+                  entries = inner.infolist()
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
+                  assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
+                  assert inner.testzip() is None
+                  pathlib.Path('binary').mkdir()
+                  with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
+                      header = source.read(20)
+                      assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
+                      destination.write(header)
+                      shutil.copyfileobj(source, destination)
+          PY
+          chmod 0755 binary/rustfs
+          printf '%s  binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
+
+      - name: Run the real health service job
+        shell: bash
+        env:
+          SOURCE_SHA: ${{ inputs.source_sha }}
+        run: |
+          set -euo pipefail
+          git -C connect-harness diff --exit-code
+          printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
+          changed=$(git -C connect-harness diff --name-only)
+          [[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
+          npm --prefix connect-harness/web ci
+          connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
+          make -C connect-harness e2e-connected-dispatch-check
+          (cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+            RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
+            CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
+            make -C connect-harness e2e-connected E2E_SCENARIO=health-service
+
+      - name: Bind and validate sanitized evidence
+        shell: bash
+        env:
+          BUILD_RUN_ID: ${{ inputs.build_run_id }}
+          ARTIFACT_ID: ${{ inputs.artifact_id }}
+          ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_SHA: ${{ inputs.connect_sha }}
+          SOURCE_SHA: ${{ inputs.source_sha }}
+        run: |
+          set -euo pipefail
+          jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
+            .provenance.connectSha == $connect and .provenance.sourceSha == $source
+            and .provenance.binarySha256 == $binary
+          ' health-service-evidence.json >/dev/null
+          jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
+            --arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
+            --arg expiry "$(cat artifact-expires-at)" '
+            .provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry}
+          ' health-service-evidence.json > health-service-evidence.bound.json
+          mv health-service-evidence.bound.json health-service-evidence.json
+          node connect-harness/tests/connectivity/health-service-evidence.mjs health-service-evidence.json --bound
+
+      - name: Upload only verified sanitized evidence
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: connect-health-service-evidence-${{ github.run_id }}
+          path: health-service-evidence.json
+          retention-days: 14
+          if-no-files-found: error
@@ -76,7 +76,7 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

@@ -107,7 +107,7 @@ jobs:
          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]

-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
          jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
          jq -e --arg name "$expected_job" '
            [.[].jobs[] | select(.name == $name)] as $matches
@@ -76,7 +76,7 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

@@ -107,7 +107,7 @@ jobs:
          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]

-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
          jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
          jq -e --arg name "$expected_job" '
            [.[].jobs[] | select(.name == $name)] as $matches
@@ -37,6 +37,10 @@ on:
        description: Expected rustfs binary SHA-256
        required: true
        type: string
+      catalog_sha256:
+        description: Independently reviewed release CPU catalog SHA-256
+        required: true
+        type: string
      connect_sha:
        description: Exact 40-character RustFS Connect harness commit
        required: true
@@ -49,7 +53,7 @@ permissions:
 jobs:
  profile-cpu:
    name: Verify native Linux x86_64 profile.cpu service job
-    runs-on: sm-standard-2
+    runs-on: dind-sm-standard-2
    timeout-minutes: 45
    steps:
      - name: Checkout acceptance harness
@@ -76,7 +80,7 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

@@ -101,20 +105,27 @@ jobs:
          [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
          [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]

-          run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
+          github_api() {
+            curl --fail --silent --show-error \
+              --header "Authorization: Bearer $GH_TOKEN" \
+              --header 'Accept: application/vnd.github+json' \
+              "https://api.github.com/$1"
+          }
+
+          run=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
          [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
          [[ $(jq -r '.head_branch' <<<"$run") == main ]]
          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]

-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
-          jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
          jq -e --arg name "$expected_job" '
-            [.[].jobs[] | select(.name == $name)] as $matches
+            [.jobs[] | select(.name == $name)] as $matches
            | (($matches | length) == 1 and $matches[0].conclusion == "success")
          ' <<<"$jobs" >/dev/null

-          artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
+          artifact=$(github_api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
          [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
          [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
          [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
@@ -129,24 +140,32 @@ jobs:
          run-id: ${{ inputs.build_run_id }}
          github-token: ${{ github.token }}

-      - name: Extract exact RustFS binary
+      - name: Extract exact RustFS binary and reviewed catalog
        shell: bash
        env:
          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
          SOURCE_SHA: ${{ inputs.source_sha }}
        run: |
          set -euo pipefail
+          [[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
+          [[ "$CATALOG_SHA256" =~ ^[0-9a-f]{64}$ ]]
          short_sha=${SOURCE_SHA:0:7}
          package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
          [[ -n "$package" ]]
          mkdir -p binary
-          unzip -qq "$package" rustfs -d binary
+          unzip -qq "$package" rustfs rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256 -d binary
          chmod +x binary/rustfs
          printf '%s  %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict
+          # The packaged digest is corroboration, never the independent trust input.
+          [[ $(cat binary/rustfs.cpu-symbol-catalog.sha256) == "$CATALOG_SHA256" ]]
+          printf '%s  %s\n' "$CATALOG_SHA256" binary/rustfs.cpu-symbol-catalog.json | sha256sum --check --strict
+          chmod 0444 binary/rustfs.cpu-symbol-catalog.json

      - name: Run signed service job acceptance
        shell: bash
        env:
+          CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
          SOURCE_SHA: ${{ inputs.source_sha }}
        run: |
          set -euo pipefail
@@ -156,6 +175,8 @@ jobs:
          make -C connect-harness e2e-connected-dispatch-check
          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
            RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
+            CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
+            CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \
            CONNECT_E2E_PROFILE_EVIDENCE="$GITHUB_WORKSPACE/profile-service-job-evidence.json" \
            make -C connect-harness e2e-connected E2E_SCENARIO=profile

@@ -38,7 +38,7 @@ permissions:
 jobs:
  profile-memory:
    name: Verify signed service memory profile over mTLS
-    runs-on: sm-standard-2
+    runs-on: dind-sm-standard-2
    timeout-minutes: 45
    steps:
      - name: Checkout exact RustFS source
@@ -60,10 +60,43 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
+
      - name: Verify official source and artifact identity
        shell: bash
        env:
@@ -139,7 +172,8 @@ jobs:
              assert outer.testzip() is None
              with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                  entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                  assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                  assert inner.testzip() is None
                  pathlib.Path('binary').mkdir()
@@ -17,6 +17,14 @@ name: Connect profile.threads service-job acceptance
 on:
  workflow_dispatch:
    inputs:
+      thread_scope:
+        description: Thread profile scope
+        required: true
+        type: choice
+        default: native
+        options:
+          - native
+          - runtime
      build_run_id:
        description: Main-branch Build and Release workflow run ID
        required: true
@@ -48,8 +56,8 @@ permissions:

 jobs:
  profile-threads:
-    name: Verify native Linux x86_64 profile.threads service job
-    runs-on: sm-standard-2
+    name: Verify Linux x86_64 profile.threads service job
+    runs-on: dind-sm-standard-2
    timeout-minutes: 45
    steps:
      - name: Checkout acceptance harness
@@ -76,10 +84,43 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
+
      - name: Verify source run and artifact identity
        shell: bash
        env:
@@ -107,7 +148,7 @@ jobs:
          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]

-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
          jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
          jq -e --arg name "$expected_job" '
            [.[].jobs[] | select(.name == $name)] as $matches
@@ -148,8 +189,14 @@ jobs:
        shell: bash
        env:
          SOURCE_SHA: ${{ inputs.source_sha }}
+          THREAD_SCOPE: ${{ inputs.thread_scope }}
        run: |
          set -euo pipefail
+          case "$THREAD_SCOPE" in
+            native) export CONNECT_E2E_THREAD_SCOPE=NATIVE_THREADS ;;
+            runtime) export CONNECT_E2E_THREAD_SCOPE=TOKIO_RUNTIME ;;
+            *) exit 1 ;;
+          esac
          printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
          npm --prefix connect-harness/web ci
          connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
@@ -168,6 +215,7 @@ jobs:
          ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
          CONNECT_SHA: ${{ inputs.connect_sha }}
          SOURCE_SHA: ${{ inputs.source_sha }}
+          THREAD_SCOPE: ${{ inputs.thread_scope }}
        run: |
          set -euo pipefail
          jq \
@@ -177,7 +225,8 @@ jobs:
            --arg artifactDigest "$ARTIFACT_DIGEST" \
            --arg connectSha "$CONNECT_SHA" \
            --arg sourceSha "$SOURCE_SHA" \
-            '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \
+            --arg requestedThreadScope "$THREAD_SCOPE" \
+            '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha, requestedThreadScope: $requestedThreadScope}' \
            profile-threads-service-job-evidence.json >profile-threads-service-job-evidence.bound.json
          mv profile-threads-service-job-evidence.bound.json profile-threads-service-job-evidence.json

@@ -76,7 +76,7 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

@@ -107,7 +107,7 @@ jobs:
          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]

-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
          jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
          jq -e --arg name "$expected_job" '
            [.[].jobs[] | select(.name == $name)] as $matches
@@ -12,21 +12,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-name: Connect top.disk artifact acceptance
+name: Connect top.disk service-job acceptance

 on:
  workflow_dispatch:
    inputs:
      build_run_id:
-        description: Successful main-branch Build and Release workflow run ID
+        description: Successful main-branch Build and Release run ID
        required: true
        type: string
      artifact_id:
-        description: Linux x86_64 GNU artifact ID from that run
+        description: Linux x86_64 GNU artifact from that run
        required: true
        type: string
      source_sha:
-        description: Exact 40-character source commit
+        description: Exact RustFS source commit
        required: true
        type: string
      artifact_digest:
@@ -34,7 +34,11 @@ on:
        required: true
        type: string
      binary_sha256:
-        description: Expected rustfs binary SHA-256
+        description: Independently verified rustfs binary SHA-256
+        required: true
+        type: string
+      connect_sha:
+        description: Exact Connect top.disk service-job acceptance harness commit
        required: true
        type: string

@@ -44,16 +48,67 @@ permissions:

 jobs:
  top-disk:
-    name: Verify native Linux x86_64 top.disk artifact
-    runs-on: sm-standard-2
-    timeout-minutes: 15
+    name: Verify native top.disk service job over mTLS
+    runs-on: dind-sm-standard-2
+    timeout-minutes: 45
    steps:
-      - name: Checkout acceptance harness
+      - name: Checkout exact RustFS source
        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
        with:
+          path: rustfs-source
          persist-credentials: false
+          ref: ${{ inputs.source_sha }}

-      - name: Verify source run and artifact identity
+      - name: Checkout exact Connect harness
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          repository: rustfs/connect
+          path: connect-harness
+          persist-credentials: false
+          ref: ${{ inputs.connect_sha }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+
+      - name: Set up Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 25.8.1
+          cache: npm
+          cache-dependency-path: connect-harness/web/package-lock.json
+
+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
+
+      - name: Verify official source and artifact identity
        shell: bash
        env:
          GH_TOKEN: ${{ github.token }}
@@ -61,82 +116,136 @@ jobs:
          ARTIFACT_ID: ${{ inputs.artifact_id }}
          SOURCE_SHA: ${{ inputs.source_sha }}
          ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_SHA: ${{ inputs.connect_sha }}
        run: |
          set -euo pipefail
-          [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]]
-          [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]]
-          [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
+          [[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
+          [[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
+          [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
          [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
+          [[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
+          [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
+          [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
+          [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
+          [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
+          run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
+          jq -e --arg source "$SOURCE_SHA" '
+            .head_sha == $source and .head_branch == "main"
+            and .head_repository.full_name == "rustfs/rustfs"
+            and .name == "Build and Release" and .path == ".github/workflows/build.yml"
+            and .status == "completed" and .conclusion == "success"
+          ' <<<"$run" >/dev/null
+          jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
+          jq -e '
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            | ($matches | length) == 1 and $matches[0].conclusion == "success"
+          ' <<<"$jobs" >/dev/null
+          artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
+          jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
+            .workflow_run.id == $run and .workflow_run.head_sha == $source
+            and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
+            and .digest == $digest and .expired == false
+            and (.expires_at | fromdateiso8601) > now
+            and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
+          ' <<<"$artifact" >/dev/null
+          jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
+          gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
+          [[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
+          printf '%s  official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict

-          run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
-          [[ $(jq -r '.conclusion' <<<"$run") == success ]]
-          [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
-          [[ $(jq -r '.head_branch' <<<"$run") == main ]]
-          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
-          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
-
-          artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
-          [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
-          [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
-          [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
-          [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]]
-          [[ $(jq -r '.expired' <<<"$artifact") == false ]]
-
-      - name: Download exact build artifact
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-        with:
-          artifact-ids: ${{ inputs.artifact_id }}
-          path: artifact
-          run-id: ${{ inputs.build_run_id }}
-          github-token: ${{ github.token }}
-
-      - name: Extract exact RustFS binary
-        shell: bash
-        env:
-          SOURCE_SHA: ${{ inputs.source_sha }}
-        run: |
-          set -euo pipefail
-          short_sha=${SOURCE_SHA:0:7}
-          package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
-          [[ -n "$package" ]]
-          mkdir -p binary
-          unzip -qq "$package" rustfs -d binary
-          chmod +x binary/rustfs
-
-      - name: Run top.disk acceptance
+      - name: Extract only the verified official service binary
        shell: bash
        env:
          SOURCE_SHA: ${{ inputs.source_sha }}
          BINARY_SHA256: ${{ inputs.binary_sha256 }}
        run: |
          set -euo pipefail
-          scripts/ci/check_connect_top_disk_artifact.sh \
-            binary/rustfs "$SOURCE_SHA" "$BINARY_SHA256" \
-            top-disk-runtime-evidence.json top-disk.zip
+          python3 - <<'PY'
+          import io
+          import os
+          import pathlib
+          import shutil
+          import stat
+          import zipfile

-      - name: Bind workflow and artifact provenance
+          def regular(entry):
+              mode = entry.external_attr >> 16
+              return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
+
+          package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
+          with zipfile.ZipFile('official-artifact.zip') as outer:
+              entries = outer.infolist()
+              assert 1 <= len(entries) <= 16
+              assert len({entry.filename for entry in entries}) == len(entries)
+              assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
+              assert sum(entry.file_size for entry in entries) <= 2147483648
+              assert outer.testzip() is None
+              with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
+                  entries = inner.infolist()
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
+                  assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
+                  assert inner.testzip() is None
+                  pathlib.Path('binary').mkdir()
+                  with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
+                      header = source.read(20)
+                      assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
+                      destination.write(header)
+                      shutil.copyfileobj(source, destination)
+          PY
+          chmod 0755 binary/rustfs
+          printf '%s  binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
+
+      - name: Run the real top.disk service job
+        shell: bash
+        env:
+          SOURCE_SHA: ${{ inputs.source_sha }}
+          CONNECT_E2E_PRODUCER_SOURCE: ${{ inputs.source_sha }}
+          CONNECT_E2E_PRODUCER_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_E2E_PRODUCER_ARTIFACT_ID: ${{ inputs.artifact_id }}
+        run: |
+          set -euo pipefail
+          git -C connect-harness diff --exit-code
+          printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
+          changed=$(git -C connect-harness diff --name-only)
+          [[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
+          npm --prefix connect-harness/web ci
+          connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
+          make -C connect-harness e2e-connected-dispatch-check
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+            RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
+            CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
+            make -C connect-harness e2e-connected E2E_SCENARIO=top-disk
+
+      - name: Bind and validate sanitized evidence
        shell: bash
        env:
          BUILD_RUN_ID: ${{ inputs.build_run_id }}
          ARTIFACT_ID: ${{ inputs.artifact_id }}
          ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_SHA: ${{ inputs.connect_sha }}
+          SOURCE_SHA: ${{ inputs.source_sha }}
        run: |
          set -euo pipefail
-          jq \
-            --arg workflowRunId "$GITHUB_RUN_ID" \
-            --arg buildRunId "$BUILD_RUN_ID" \
-            --arg artifactId "$ARTIFACT_ID" \
-            --arg artifactDigest "$ARTIFACT_DIGEST" \
-            '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest}' \
-            top-disk-runtime-evidence.json >top-disk-runtime-evidence.bound.json
-          mv top-disk-runtime-evidence.bound.json top-disk-runtime-evidence.json
+          jq -e --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" --arg artifact "$ARTIFACT_ID" '
+            .status == "PASS" and .officialArtifactId == $artifact and .producerSource == $source and .producerSha256 == $binary
+            and .result.provenance.sourceCommit == $source
+            and .result.provenance.executableSha256 == $binary
+          ' top-disk-service-job-evidence.json >/dev/null
+          jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
+            --arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
+            --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" \
+            --arg expiry "$(cat artifact-expires-at)" '
+            . + {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry,connectSha:$connect,sourceSha:$source,binarySha256:$binary}
+          ' top-disk-service-job-evidence.json > top-disk-service-job-evidence.bound.json
+          mv top-disk-service-job-evidence.bound.json top-disk-service-job-evidence.json

-      - name: Upload acceptance evidence
+      - name: Upload only verified sanitized evidence
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
-          name: connect-top-disk-evidence-${{ github.run_id }}
-          path: |
-            top-disk-runtime-evidence.json
-            top-disk.zip
+          name: connect-top-disk-service-job-evidence-${{ github.run_id }}
+          path: top-disk-service-job-evidence.json
          retention-days: 14
          if-no-files-found: error
@@ -76,7 +76,7 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

@@ -107,7 +107,7 @@ jobs:
          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]

-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
          jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
          jq -e --arg name "$expected_job" '
            [.[].jobs[] | select(.name == $name)] as $matches
@@ -76,7 +76,7 @@ jobs:
      - name: Set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
-          node-version: 22.22.2
+          node-version: 25.8.1
          cache: npm
          cache-dependency-path: connect-harness/web/package-lock.json

@@ -107,7 +107,7 @@ jobs:
          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]

-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
          jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
          jq -e --arg name "$expected_job" '
            [.[].jobs[] | select(.name == $name)] as $matches
@@ -128,7 +128,7 @@ jobs:
    # manual workflow_dispatch runs stay quiet so a debugging run never files a
    # spurious alert.
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -38,6 +38,7 @@ on:
      - "Cargo.lock"
      - "Cargo.toml"
      - ".config/nextest.toml"
+      - ".config/e2e-distributed-selection.txt"
      - ".github/workflows/e2e-distributed.yml"
      - "crates/audit/**"
      - "crates/common/**"
@@ -200,7 +201,7 @@ jobs:
    name: Alert on scheduled failure
    needs: [distributed]
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -230,7 +230,7 @@ jobs:
    # manual workflow_dispatch runs stay quiet so a debugging run never files a
    # spurious alert.
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -453,7 +453,7 @@ jobs:
    # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
    # dispatch failures are already watched by a human.
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -51,8 +51,53 @@ env:
  UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d

 jobs:
+  build:
+    name: Build upgrade candidate
+    runs-on: sm-standard-2
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          cache-shared-key: e2e-upgrade-server
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          install-build-packaging-tools: "false"
+          install-test-tools: "false"
+
+      - name: Download pinned previous release
+        run: |
+          mkdir -p target/debug
+          archive="target/debug/$UPGRADE_SOURCE_ASSET"
+          curl --fail --location --retry 3 --output "$archive" \
+            "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
+          echo "$UPGRADE_SOURCE_SHA256  $archive" | sha256sum --check --strict
+
+      - name: Build current RustFS binary
+        run: python3 scripts/e2e_binary.py build
+
+      - name: Upload verified upgrade candidate
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: upgrade-candidate-${{ github.run_id }}
+          path: |
+            target/debug/rustfs
+            target/debug/rustfs.e2e.json
+            target/debug/${{ env.UPGRADE_SOURCE_ASSET }}
+          if-no-files-found: error
+          retention-days: 3
+          compression-level: 0
+          overwrite: true
+
  upgrade:
    name: ${{ matrix.name }}
+    needs: build
    strategy:
      fail-fast: false
      matrix:
@@ -63,31 +108,24 @@ jobs:
          # the CI required-check names. UPGRADE_SOURCE_VERSION above is the
          # single source of truth for which release they actually run against.
          - name: Direct upgrade from the previous release
-            cache_key: e2e-direct-upgrade
            test: direct_upgrade_from_rc2_preserves_object_contracts
            artifact: direct-upgrade
          - name: Mixed-version rolling upgrade from the previous release
-            cache_key: e2e-mixed-version-upgrade
            test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
            artifact: mixed-version-upgrade
          - name: Bucket configuration survives the upgrade
-            cache_key: e2e-bucket-config-upgrade
            test: direct_upgrade_from_previous_release_preserves_bucket_configuration
            artifact: bucket-config-upgrade
          - name: Rollback reads current bucket metadata
-            cache_key: e2e-bucket-config-rollback
            test: rollback_to_previous_release_reads_current_bucket_metadata
            artifact: bucket-config-rollback
          - name: ODM configuration recovery after rc.5 rollback
-            cache_key: e2e-odm-config-rollback
            test: rc5_rollback_requires_restoring_odm_configuration
            artifact: odm-config-rollback
          - name: Multipart layouts survive the rc.5 upgrade
-            cache_key: e2e-multipart-layout-upgrade
            test: direct_upgrade_from_rc5_preserves_multipart_layouts
            artifact: multipart-layout-upgrade
          - name: rc.5 multipart replication baseline
-            cache_key: e2e-multipart-layout-baseline
            test: rc5_baseline_replicates_multipart_layouts
            artifact: multipart-layout-baseline
    runs-on: sm-standard-2
@@ -103,19 +141,28 @@ jobs:
      - name: Setup Rust environment
        uses: ./.github/actions/setup
        with:
-          cache-shared-key: ${{ matrix.cache_key }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          cache-shared-key: e2e-upgrade-tests
+          # One main-branch consumer warms the shared test-harness dependencies.
+          cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
          install-build-packaging-tools: "false"
+          install-test-tools: "false"

-      - name: Download pinned previous release
+      - name: Download verified upgrade candidate
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: upgrade-candidate-${{ github.run_id }}
+          path: target/debug
+
+      - name: Restore candidate executable bit
+        run: chmod +x target/debug/rustfs
+
+      - name: Verify and unpack pinned previous release
        env:
          SOURCE_DIR: ${{ runner.temp }}/rustfs-upgrade-source
        run: |
          set -euo pipefail
          mkdir -p "$SOURCE_DIR"
-          archive="$SOURCE_DIR/$UPGRADE_SOURCE_ASSET"
-          curl --fail --location --retry 3 --output "$archive" \
-            "https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
+          archive="target/debug/$UPGRADE_SOURCE_ASSET"
          echo "$UPGRADE_SOURCE_SHA256  $archive" | sha256sum --check --strict
          unzip -q "$archive" -d "$SOURCE_DIR"
          chmod +x "$SOURCE_DIR/rustfs"
@@ -123,14 +170,12 @@ jobs:
          echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
          echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"

-      - name: Build current RustFS binary
-        run: |
-          python3 scripts/e2e_binary.py build
-
      - name: Run upgrade compatibility test
        env:
          RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
+          NO_PROXY: 127.0.0.1,localhost,::1
        run: |
+          export no_proxy="$NO_PROXY"
          python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
            "upgrade_compatibility_test::${{ matrix.test }}" \
            -- --ignored --exact --nocapture
@@ -11,12 +11,44 @@ concurrency:
  cancel-in-progress: false
 jobs:
  collect:
-    runs-on: sm-standard-2
+    runs-on: smoke-testing
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
        with:
          persist-credentials: false
+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
      - name: Collect and publish verified chain health
        env:
          GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -240,7 +240,7 @@ jobs:
    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
    # ci-8); PR and manual dispatch failures are already watched by a human.
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -145,7 +145,9 @@ jobs:
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: helm-package
-          path: helm/rustfs/*.tgz
+          path: |
+            helm/rustfs/*.tgz
+            helm/rustfs/README.md
          retention-days: 1

  publish-helm-package:
@@ -60,9 +60,9 @@ permissions:
 jobs:
  minio-interop:
    name: MinIO interop (EC + SSE read parity)
-    # Skip on forks: needs the repo's runners and is not a contributor gate.
+    # Keep this scheduled interoperability lane scoped to the upstream repo.
    if: github.repository == 'rustfs/rustfs'
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 40
    env:
      # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
@@ -131,7 +131,7 @@ jobs:
    name: Alert on scheduled failure
    needs: [minio-interop]
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -258,7 +258,7 @@ jobs:
    # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
    # dispatch failures are already watched by a human.
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -498,7 +498,7 @@ jobs:
    if: >-
      always() && github.event_name == 'schedule' &&
      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -51,7 +51,7 @@ concurrency:
 jobs:
  oidc-keycloak-live:
    name: OIDC Keycloak live gate
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 60
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
@@ -72,6 +72,11 @@ jobs:
      - name: Build RustFS
        run: cargo build --locked -p rustfs --bin rustfs

+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+
      - name: Install pinned request signer
        run: |
          python3 -m pip install --user --upgrade pip "awscurl==0.44"
@@ -95,7 +100,7 @@ jobs:
    if: >-
      always() && github.event_name == 'schedule' &&
      (needs.oidc-keycloak-live.result == 'failure' || needs.oidc-keycloak-live.result == 'cancelled')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -69,10 +69,9 @@ env:
 jobs:
  minio-source:
    name: MinIO source (read-through, list-through, backfill)
-    # Skip on forks: needs this repository's runners and is not a contributor
-    # gate.
+    # Keep this scheduled interoperability lane scoped to the upstream repo.
    if: github.repository == 'rustfs/rustfs'
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 90
    env:
      NO_PROXY: 127.0.0.1,localhost
@@ -108,11 +107,15 @@ jobs:
        run: |
          set -euo pipefail
          mkdir -p artifacts/odm-interop/minio
+          docker build -t rustfs-odm-interop-minio \
+            -f crates/rio-v2/tests/minio_fixture_lab/Dockerfile \
+            crates/rio-v2/tests/minio_fixture_lab
          docker run -d --name rustfs-odm-interop-minio \
+            --entrypoint /usr/local/bin/minio \
            -e "MINIO_ROOT_USER=${MINIO_ROOT_USER}" \
            -e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
            -p 9100:9000 \
-            minio/minio:RELEASE.2025-09-07T16-13-09Z server /data
+            rustfs-odm-interop-minio server /data
          for _ in $(seq 1 120); do
            curl -fsS http://127.0.0.1:9100/minio/health/live >/dev/null 2>&1 && break
            sleep 1
@@ -300,7 +303,7 @@ jobs:
    if: >-
      always() && github.event_name == 'schedule' &&
      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -77,7 +77,7 @@ jobs:
      (github.event.workflow_run.conclusion == 'success' &&
       github.event.workflow_run.event == 'push' &&
       github.event.workflow_run.head_branch != 'main')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
    timeout-minutes: 10
    outputs:
      version: ${{ steps.resolve.outputs.version }}
@@ -123,7 +123,7 @@ jobs:
          semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
          prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
          if [[ -n "$TAG" ]]; then
-            if [[ "$TAG" =~ ^${semver_core}-${prerelease_id}-preview\.(0|[1-9][0-9]*)$ ]]; then
+            if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
              BUILD_TYPE=preview
            elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
              BUILD_TYPE=prerelease
@@ -229,7 +229,7 @@ jobs:
  package:
    name: Package (${{ matrix.arch }}-${{ matrix.libc }})
    needs: resolve
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
    timeout-minutes: 30
    strategy:
      fail-fast: false
@@ -504,7 +504,7 @@ jobs:
          retention-days: 30

      - name: Upload packages to Cloudflare R2
-        if: env.R2_ACCESS_KEY_ID != ''
+        if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
        env:
          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -366,7 +366,7 @@ jobs:
    if: >-
      always() && github.event_name == 'schedule' &&
      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -66,7 +66,7 @@ jobs:
    # scheduled runs file a tracking issue, manual dispatch stays quiet so
    # debugging never produces a spurious alert.
    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -42,7 +42,7 @@ concurrency:
 jobs:
  prepare:
    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
-    runs-on: sm-standard-2
+    runs-on: smoke-testing
    timeout-minutes: 10
    outputs:
      manifest: ${{ steps.candidate.outputs.manifest }}
@@ -188,7 +188,7 @@ jobs:
  complete-chain:
    needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
    if: ${{ always() && needs.prepare.result != 'skipped' }}
-    runs-on: sm-standard-2
+    runs-on: smoke-testing
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -7,6 +7,11 @@ on:
        description: Verified candidate and chain attempt from the chain driver
        type: string
        required: true
+      pool_timeout_minutes:
+        description: Wall-clock budget (minutes) for the pool run step
+        type: string
+        required: false
+        default: '240'
  workflow_dispatch:
    inputs:
      rustfs_version:
@@ -39,6 +44,10 @@ on:
        description: 'Run the pool decommission step (3-pool topology only)'
        type: boolean
        default: true
+      pool_timeout_minutes:
+        description: 'Wall-clock budget (minutes) for the whole pool run: fill + rebalance wait + decommission'
+        required: false
+        default: '240'
      cleanup_before:
        description: 'Reset the nodes before the test (DESTROYS existing data/config)'
        type: boolean
@@ -82,7 +91,8 @@ jobs:
  pool-expansion-test:
    name: Pool expansion / decommission test
    runs-on: smoke-testing
-    timeout-minutes: 60
+    # Leave setup and finalizer headroom beyond the default 240-minute test.
+    timeout-minutes: 360
    if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
    env:
      RUSTFS_POOL_ADMIN_ENDPOINT: ${{ secrets.RUSTFS_POOL_ADMIN_ENDPOINT || vars.RUSTFS_POOL_ADMIN_ENDPOINT || 'http://rustfs-node1:9000' }}
@@ -251,7 +261,12 @@ jobs:

      - name: Run pool expansion & decommission test
        id: pool_test
-        timeout-minutes: 45
+        # The suite budgets up to 24h each for rebalance and decommission
+        # (REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh);
+        # the previous 45m step cap killed four consecutive runs mid-rebalance
+        # with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
+        # in ~21m) plus one decommission pass; override per dispatch.
+        timeout-minutes: ${{ inputs.pool_timeout_minutes || 240 }}
        # Case failures keep the run green: the report and the backlog
        # issue manager carry the product signal.
        continue-on-error: true
@@ -568,6 +583,25 @@ jobs:
            echo "pool test script reported a failure (step FAIL verdict or die() abort)" >&2
            failed=1
          fi
+          # A runner-level kill (step timeout) truncates the suite without a
+          # FAIL verdict or an [ERROR] line, so require every expected
+          # [POOL-STEP] marker to be present with PASS - missing markers mean
+          # the run was truncated, not green. This also closes the gap for
+          # standalone dispatches, where the chain driver's evidence
+          # validation does not run.
+          last_step=6
+          if [ "${{ inputs.pools || '3' }}" = "3" ]; then
+            last_step=8
+            if [ "${{ inputs.run_decommission != 'false' }}" = "true" ]; then
+              last_step=9
+            fi
+          fi
+          for n in $(seq 1 "${last_step}"); do
+            grep -q "^\[POOL-STEP\] ${n} .* PASS" "${POOL_ARTIFACT_DIR}/pool-test.log" 2>/dev/null || {
+              echo "pool test log is missing [POOL-STEP] ${n} PASS (expected through step ${last_step}); the run was truncated" >&2
+              failed=1
+            }
+          done
          [ "${failed}" -eq 0 ] || exit 1

      - name: Upload functional report to dashboard
@@ -106,7 +106,7 @@ jobs:
        if: ${{ inputs.chain_manifest != '' }}
        env:
          CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
-        run: python3 scripts/functional_chain_evidence.py consume
+        run: python3 rustfs-repo/scripts/functional_chain_evidence.py consume

      - name: Checkout auto-testing scripts
        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -389,7 +389,7 @@ jobs:
          CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }}
          CHAIN_REPORT_OUTCOME: ${{ steps.report.outcome }}
        run: >-
-          python3 scripts/functional_chain_evidence.py record --suite security
+          python3 rustfs-repo/scripts/functional_chain_evidence.py record --suite security
          --report "${SECURITY_ARTIFACTS_DIR}/suite-report.md"
          --output "${RUNNER_TEMP}/chain-security-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/security.json"

@@ -58,7 +58,7 @@ jobs:
    # Mirrors the consumer wiring, minus the schedule-event guard (this
    # workflow is dispatch-only by design).
    if: always() && contains(needs.*.result, 'failure')
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -177,7 +177,7 @@ jobs:
    if: >-
      always() && github.event_name == 'schedule' &&
      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
    timeout-minutes: 10
    permissions:
      contents: read
@@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.

 ### Fixed
+- **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117.
 - **Vault backends accepted key names that escape the key prefix**: only the Local backend refused a key identifier containing a path separator, so on Vault KV2 `POST /rustfs/admin/v3/kms/keys` with a name such as `bad/name` succeeded and created a nested KV2 path that the key listing then reported as a directory rather than a key, and an identifier containing `..` addressed a record outside the configured key prefix once the HTTP client normalised the URL. Both Vault backends now apply the Local backend's containment rule at the point where the identifier becomes a path or a Transit key name: an empty identifier, one containing `/`, `\\` or NUL, or the dot segments `.` and `..` is refused with `400` (`InvalidKey`) before any request reaches Vault. The AWS backend is unaffected, since it addresses keys by ARN and alias. A Vault deployment that created such a key on an earlier build must re-create it under a plain name; objects encrypted under the old name stay readable only until that key is refused, so migrate them first. Refs rustfs/backlog#2474.
 - **Fresh multi-pool bootstrap with distinct format creators**: a new deployment whose pools have their first endpoint on different nodes (for example two single-node pools) could never publish its initial `pool.bin`: each node held fresh-bootstrap proof only for the pool it formatted, the deployment-wide proof collapsed to none, and every node died with `pool metadata recovery required: no durable bootstrap identity or pool.bin replica is available` after the startup retry budget. The first pool's creator now mints the pending cluster identity on its own pool, every other creator copies that nonce-bound identity onto the pool it formatted first-hand, and the elected writer publishes `pool.bin` once every pool replica carries the same pending identity. Corrupt or disagreeing replicas, pools that merely have a format, expansion pools joining an initialized deployment, and restarts without first-hand proof still fail closed. Non-elected nodes that start before `pool.bin` exists, and the elected writer while it waits for the other creators, no longer latch their pool-metadata write gate for the life of the process. Refs rustfs/backlog#2338, rustfs/backlog#2375.
 - **Lock RPC timeout storms** (#7363): the remote lock client no longer evicts and re-dials the shared internode HTTP/2 channel on every request deadline. A timeout evicts only when the peer has not completed any lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (`RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS`, default 5 s), and a timed-out request is left running instead of being reset (bounded per peer by `RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT`, default 256), so a slow lock endpoint can no longer drive the `RST_STREAM`/`GOAWAY too_many_resets`/reconnect loop. A lock granted after its caller timed out is released immediately, and unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New `rustfs_remote_lock_*` metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer. Operator guide at `docs/operations/lock-rpc-storm-protection.md`.
@@ -685,9 +685,9 @@ dependencies = [

 [[package]]
 name = "async-compression"
-version = "0.4.48"
+version = "0.4.50"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fb61aea1a7def73ee7c350a184f0e70b32c182344e2e75bf70c9b621b83417fd"
+checksum = "ee19bd99b43e3691acbad4e840420a4881cea6c0b66a208125a824f8fd53f5a1"
 dependencies = [
 "compression-codecs",
 "compression-core",
@@ -771,17 +771,6 @@ dependencies = [
 "url",
 ]

-[[package]]
-name = "async-recursion"
-version = "1.1.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 2.0.119",
-]
-
 [[package]]
 name = "async-rs"
 version = "0.8.12"
@@ -987,9 +976,9 @@ dependencies = [

 [[package]]
 name = "aws-sdk-kms"
-version = "1.121.0"
+version = "1.122.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9febf0461c7441895de457dcff49dd7fa8f013793367f6b81d422c64e939e1cc"
+checksum = "ae92186214a60f1316f6e25df2b5bef8f67b5b08b45f8e3eb6e1278b9ddd7b29"
 dependencies = [
 "arc-swap",
 "aws-credential-types",
@@ -1012,9 +1001,9 @@ dependencies = [

 [[package]]
 name = "aws-sdk-s3"
-version = "1.149.0"
+version = "1.151.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d48257fb0eb1907a02a965ff7d1b910770a6de2c59a2d03461678df989c5cdcd"
+checksum = "c82d4b41fe41bfca3c9e1de55645d9a525868ef5ce5e14cee9499ccaf1fc7225"
 dependencies = [
 "arc-swap",
 "aws-credential-types",
@@ -1048,9 +1037,9 @@ dependencies = [

 [[package]]
 name = "aws-sdk-sso"
-version = "1.112.0"
+version = "1.113.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d3d59c47556b283bb2d20fc0b416df7d9d9a3d6f5fa0e77867167a3f2efc23b9"
+checksum = "87184f36ade012f46cdc773c7be10031b4545a1aea70a9a22a044e8f36f1e5c5"
 dependencies = [
 "arc-swap",
 "aws-credential-types",
@@ -1073,9 +1062,9 @@ dependencies = [

 [[package]]
 name = "aws-sdk-ssooidc"
-version = "1.114.0"
+version = "1.115.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "eceb9f31b46ba6da21f666e456395624824a2a086caed940b13d78832fa3ee02"
+checksum = "84edc7495d56e006dff0031cbc9782784902456f7024651c2da902a3d2336262"
 dependencies = [
 "arc-swap",
 "aws-credential-types",
@@ -1098,9 +1087,9 @@ dependencies = [

 [[package]]
 name = "aws-sdk-sts"
-version = "1.117.0"
+version = "1.118.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9291df9ea767ab5e77d08a53f4f22daa917909a34ed7a22c1a4b82d193a423ad"
+checksum = "df7572569d018ca72923073494eac3c4671a0daea90b9e3923c2d27a3a02a2c1"
 dependencies = [
 "arc-swap",
 "aws-credential-types",
@@ -1193,9 +1182,9 @@ dependencies = [

 [[package]]
 name = "aws-smithy-http"
-version = "0.64.0"
+version = "0.64.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "37843d9add67c3aff5856f409c6dc315d3cdff60f9c0cb5b670dab1e9920306d"
+checksum = "639b4d8f8555f24a9be649811c3eb0b4d4616f4d61daf0c32e28873bc1ea9af1"
 dependencies = [
 "aws-smithy-eventstream",
 "aws-smithy-runtime-api",
@@ -1215,9 +1204,9 @@ dependencies = [

 [[package]]
 name = "aws-smithy-http-client"
-version = "1.4.2"
+version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7bd25384a4e437aa8d8f339afad4b69e786b936a7cb10db668a7aaf66717b1a8"
+checksum = "51c89cc3f1f281d659a67a519a1b5c6d445b5ce09fa7e5aee40c2c2707e9509d"
 dependencies = [
 "aws-smithy-async",
 "aws-smithy-protocol-test",
@@ -1297,9 +1286,9 @@ dependencies = [

 [[package]]
 name = "aws-smithy-runtime"
-version = "1.15.0"
+version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "591e0024cdf2a8de8711860f7e1d5cfbca26b800ce13793a9aa2f4f5918e6d95"
+checksum = "d6e302ac1d88b99652489df31abdec6ac42a2ab2ac3982ad0ac49f64dfaf28ba"
 dependencies = [
 "aws-smithy-async",
 "aws-smithy-http",
@@ -1321,9 +1310,9 @@ dependencies = [

 [[package]]
 name = "aws-smithy-runtime-api"
-version = "1.17.0"
+version = "1.19.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3fbf162725183ec0df77a9dc82ce22d9ba0b6ea7f376e8ec4f6b695727c70698"
+checksum = "c0730c16f91124c6a2abb4932c77e299288b3dd9f967ea2e9ec48cc6731e87a4"
 dependencies = [
 "aws-smithy-async",
 "aws-smithy-runtime-api-macros",
@@ -1360,9 +1349,9 @@ dependencies = [

 [[package]]
 name = "aws-smithy-types"
-version = "1.8.0"
+version = "1.8.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "16ecf7989e381d0cbb7ca06104140d940bf38d8eb9004bd3a12d23df03e3d363"
+checksum = "69bb407740a197147da48238ecc94498493c9e85445732360cec180296ca45f1"
 dependencies = [
 "base64-simd",
 "bytes",
@@ -1544,9 +1533,9 @@ dependencies = [

 [[package]]
 name = "bigdecimal"
-version = "0.4.10"
+version = "0.4.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695"
+checksum = "8fa3f3d8cbf4dffcfe4991de61d012bef509a409ecbe9dd41049bfe32b4d4653"
 dependencies = [
 "autocfg",
 "libm",
@@ -1746,7 +1735,7 @@ checksum = "f8b851b75c23ca7873623d612fe49bd1989aeb03d08fb9432187eb253d3d4c6b"
 dependencies = [
 "alloc-no-stdlib 3.0.0",
 "alloc-stdlib 0.3.0",
- "brotli-decompressor 6.0.0",
+ "brotli-decompressor 6.0.1",
 ]

 [[package]]
@@ -1761,9 +1750,9 @@ dependencies = [

 [[package]]
 name = "brotli-decompressor"
-version = "6.0.0"
+version = "6.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "46087801e40b785410c352bee60e9b6226312709cf52d5fff92776cdf5cf6490"
+checksum = "941cd9bd4ddab83cb46fa5a2d428f1c857b24ac78cb876cf7beb710840934bd7"
 dependencies = [
 "alloc-no-stdlib 3.0.0",
 "alloc-stdlib 0.3.0",
@@ -1971,9 +1960,9 @@ dependencies = [

 [[package]]
 name = "cc"
-version = "1.4.7"
+version = "1.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "54413ede23c2daf518f35156dfde027feb2374004d63bd497f983c8db9c0e313"
+checksum = "f360145194ee8e21db5ee7f3fcd4fe52210864c75c985dae33218202c8bbe040"
 dependencies = [
 "find-msvc-tools",
 "jobserver",
@@ -2209,11 +2198,11 @@ dependencies = [

 [[package]]
 name = "compression-codecs"
-version = "0.4.43"
+version = "0.4.45"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bef16c47ba2797aa6a909cc37d39911f3a6743811fe7408ac0b0cc0276b656e9"
+checksum = "98fc98460ba0ad5317075d3632b8dfc45d0be8c4a49347c2a38272019717614a"
 dependencies = [
- "brotli 8.0.4",
+ "brotli 9.0.0",
 "bzip2",
 "compression-core",
 "flate2",
@@ -4217,13 +4206,12 @@ checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"

 [[package]]
 name = "encoding_rs"
-version = "0.8.41"
+version = "0.8.42"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7b5ef0006ac9ab233c38522f5ae99cae3625151de8f706cacee1cba4b8e2832a"
+checksum = "8e985e0451871ad22fb8d2b6b076e2028a502a0d3950998c2c5c0a4f9b5d9679"
 dependencies = [
 "cfg-if",
 "core_detect",
- "multiversion",
 "multiversion_no_op",
 "rustversion",
 "scopeguard",
@@ -4406,9 +4394,9 @@ dependencies = [

 [[package]]
 name = "find-msvc-tools"
-version = "0.1.13"
+version = "0.1.14"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ef25905e51abafe4dcea6c15fec58c57b601cdbd0ee53d22ea1d3016c587d39b"
+checksum = "aedcfb3409746eddb02b9e19ebda1c3394f759a152e48ee875a0844d1b955484"

 [[package]]
 name = "findshlibs"
@@ -4756,9 +4744,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-auth"
-version = "1.16.0"
+version = "1.17.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ff461519b1a948200f163574be072753bcfb462a323f0eb426629d89872dd685"
+checksum = "ac9870b8ea484519aa8601e14886f2efc148ea490034e43d04bef1d246170cc9"
 dependencies = [
 "async-trait",
 "aws-lc-rs",
@@ -4785,9 +4773,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-gax"
-version = "1.14.0"
+version = "1.15.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c5615cff28ee59cfe52fbb4c11b8b1e77f650296e2ea4f4c2b7757ac6b19e752"
+checksum = "fc64ee071c0d22cb68662b454f1bea5424506cafeaddc51260fbc585bac1d29d"
 dependencies = [
 "bytes",
 "futures",
@@ -4805,9 +4793,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-gax-internal"
-version = "0.7.18"
+version = "0.7.20"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d2766757d877a7a8ac23da9884cb0e3f10ed9b75a0ce59801ce6b19bf9d5819e"
+checksum = "e873f7e6e39030c07b6d9697bf3b999a269175c8b433e058f56ae10d129f12f7"
 dependencies = [
 "bytes",
 "futures",
@@ -4844,9 +4832,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-iam-v1"
-version = "1.12.0"
+version = "1.13.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5f962b40234b1531e6ef73f7558871c96e117231e962086c98804323fb8d2c82"
+checksum = "48f0895fa7b7bd70629ae58daaa962677e93a72c251b2f297bc058a035940821"
 dependencies = [
 "async-trait",
 "bytes",
@@ -4862,9 +4850,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-longrunning"
-version = "1.13.0"
+version = "1.14.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1c0363c5389ffda2b55cd8a86eef4b19a3481a48467c91dc5d77f626a9572766"
+checksum = "e8ea252eee22a4e2187025ea6789c682a0d791a9748ea99de2abec3026a32cb4"
 dependencies = [
 "async-trait",
 "bytes",
@@ -4880,9 +4868,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-lro"
-version = "1.10.0"
+version = "1.11.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "47af3deef75c14a2983c430898d960c765bddbcc9f9188ca0563108e9227cfe7"
+checksum = "dfb938b40ede9ef70c24f49c2687d84f289c9e9104ddfece739985e4b4844b8f"
 dependencies = [
 "google-cloud-gax",
 "google-cloud-gax-internal",
@@ -4909,9 +4897,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-storage"
-version = "1.18.0"
+version = "1.20.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "973399251b245c63f1d02d0768772833dcf1372ee358f593fb9159de8fe9c7d4"
+checksum = "570eb8d0784118a5f4ae1219fd76ff3b14e90b7b0f877478c02b1374e9bbcb8c"
 dependencies = [
 "async-trait",
 "base64 0.23.1",
@@ -4962,11 +4950,11 @@ dependencies = [

 [[package]]
 name = "google-cloud-wkt"
-version = "1.7.0"
+version = "1.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7fccf98cfd5481a5f5a285181ab0c62123d7d47cd2bb7299448440649349e4e7"
+checksum = "7f3037907e542cf8b8580aa4cf8ad68568d7ac8e7274519a7d0a121867d434da"
 dependencies = [
- "base64 0.22.1",
+ "base64 0.23.1",
 "bytes",
 "serde",
 "serde_json",
@@ -5096,9 +5084,9 @@ dependencies = [

 [[package]]
 name = "headers"
-version = "0.4.1"
+version = "0.4.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b3314d5adb5d94bcdf56771f2e50dbbc80bb4bdf88967526706205ac9eff24eb"
+checksum = "fc2700e3fe555c3310aa7286cac6167449f2c87e0eb58769c9208a1c58a1d106"
 dependencies = [
 "base64 0.22.1",
 "bytes",
@@ -5283,9 +5271,9 @@ dependencies = [

 [[package]]
 name = "hotpath"
-version = "0.26.0"
+version = "0.28.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f37234f85a8d1bda6b84d486245dd0847c5b33cca10d5f3691375e66704de0f8"
+checksum = "d8345e171844f80e5ae4cbed8bc8276e094f6ff2293a6f8e547c200183976674"
 dependencies = [
 "arc-swap",
 "async-channel",
@@ -5297,6 +5285,7 @@ dependencies = [
 "futures-core",
 "futures-sink",
 "hdrhistogram",
+ "hotpath-drain",
 "hotpath-macros",
 "http 1.5.0",
 "libc",
@@ -5310,15 +5299,22 @@ dependencies = [
 "rustc-demangle",
 "serde",
 "serde_json",
+ "time",
 "tiny_http",
 "tokio",
 ]

 [[package]]
-name = "hotpath-macros"
-version = "0.26.0"
+name = "hotpath-drain"
+version = "0.0.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2ac39b5517f540cab511bdbd5a4dc8cfc62858d6cd26a095463c1d2acfb3f290"
+checksum = "e149b660b611650bdf3307941ef3e9b81be3c30ed1dfdbe7f670319674f023a6"
+
+[[package]]
+name = "hotpath-macros"
+version = "0.28.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bf73e28fe4a7595c322e4265c1608cb10c215cbbce11e49cfcc258766ba686d6"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -5459,16 +5455,17 @@ dependencies = [

 [[package]]
 name = "hyper-util"
-version = "0.1.20"
+version = "0.1.21"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
+checksum = "ddc03d96684f9226b8a787cdb71488417b53ab5ea8fdb1dac946cb9431cc8bff"
 dependencies = [
- "base64 0.22.1",
+ "base64 0.23.1",
 "bytes",
 "futures-channel",
 "futures-util",
 "http 1.5.0",
 "http-body",
+ "httparse",
 "hyper",
 "ipnet",
 "libc",
@@ -5905,9 +5902,9 @@ dependencies = [

 [[package]]
 name = "js-sys"
-version = "0.3.105"
+version = "0.3.106"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ce57d20d1ea864ce2ac172ab472d409214f4fd359f0b2a2775abdf522e2af99e"
+checksum = "7883d941dae510fb2d978fc3fe018c71c9e2892fd38854de3e8b92c2e5ad9cc5"
 dependencies = [
 "cfg-if",
 "futures-util",
@@ -6037,9 +6034,9 @@ dependencies = [

 [[package]]
 name = "lazy_static"
-version = "1.5.0"
+version = "1.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
+checksum = "20870f649af7073d53e38067b2a84312175d56ea15217e1b15bc83506ec50afb"
 dependencies = [
 "spin 0.9.9",
 ]
@@ -6202,9 +6199,9 @@ dependencies = [

 [[package]]
 name = "libunftp"
-version = "0.23.0"
+version = "0.23.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c9f8ff371890db2cf65a0758dba9a79f9cd965de369f6dbdc6581a22780af45e"
+checksum = "3dc54288ecbf8beefa5cab3a9921ab03bf1f18a3a0872cbcdb5313fc0baebaeb"
 dependencies = [
 "async-trait",
 "bitflags 2.13.2",
@@ -6213,7 +6210,7 @@ dependencies = [
 "dashmap",
 "derive_more",
 "futures-util",
- "getrandom 0.3.4",
+ "getrandom 0.4.3",
 "lazy_static",
 "moka",
 "nix 0.30.1",
@@ -6699,27 +6696,6 @@ version = "0.10.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084"

-[[package]]
-name = "multiversion"
-version = "0.9.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b4ca4bea16ffc3f443cf7d866912118196bfef4c6a1556ca00f9f9b00bb43f7c"
-dependencies = [
- "multiversion-macros",
-]
-
-[[package]]
-name = "multiversion-macros"
-version = "0.9.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0d416831a7317ef4b08bee00b69cbbb9c8763da7959a7026244d6266869f9c83"
-dependencies = [
- "proc-macro2",
- "quote",
- "rustversion",
- "syn 3.0.6",
-]
-
 [[package]]
 name = "multiversion_no_op"
 version = "1.0.0"
@@ -7226,6 +7202,18 @@ dependencies = [
 "ruzstd",
 ]

+[[package]]
+name = "object"
+version = "0.40.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dd229a0361b9d0d4396176e02d65897f487eebeab7caa6d443855ee152ca0b9c"
+dependencies = [
+ "crc32fast",
+ "hashbrown 0.17.1",
+ "indexmap 2.14.2",
+ "memchr",
+]
+
 [[package]]
 name = "object_store"
 version = "0.13.2"
@@ -7322,9 +7310,9 @@ checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"

 [[package]]
 name = "opentelemetry"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0142c63252a9e054e68a4c61a5778f7b14f576274d593f8ce883d191a099682"
+checksum = "6cdb0b1b267eb9db3331b434ed9ddab10d50e280a9adf9d13e5233e2002b61b5"
 dependencies = [
 "futures-core",
 "futures-sink",
@@ -7336,9 +7324,9 @@ dependencies = [

 [[package]]
 name = "opentelemetry-appender-tracing"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2c0080f0dc1d7c786f467cd85a4e395fcab11ee852004f39a29a18ab7c25d837"
+checksum = "e2b304e8d8f11326ba366a99ca5c7aaf7cd397b67268a332437175aeb42a40db"
 dependencies = [
 "opentelemetry",
 "tracing",
@@ -7349,9 +7337,9 @@ dependencies = [

 [[package]]
 name = "opentelemetry-http"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5683015d09e2df236ef005b17f6f196f0d5f6313c4fa43a7b6a53b52776e4331"
+checksum = "ee2c3625b8aa04209f7e01e513bc8044da9687fa38a9eacf59628ca5f3b87300"
 dependencies = [
 "async-trait",
 "bytes",
@@ -7362,12 +7350,13 @@ dependencies = [

 [[package]]
 name = "opentelemetry-otlp"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9966929966d17620d7c316c643ba62631826e10021409357772d5eea84f62c35"
+checksum = "699a67345e21962a231955b9059a157e428b219fa5df8efe11f08346fb23a344"
 dependencies = [
 "flate2",
 "http 1.5.0",
+ "httpdate",
 "opentelemetry",
 "opentelemetry-http",
 "opentelemetry-proto",
@@ -7379,9 +7368,9 @@ dependencies = [

 [[package]]
 name = "opentelemetry-proto"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "56d658ba1faf63f7b9c492cfbe6e0ec365440a16132d3270c1065f7b33f1b638"
+checksum = "25da1ac11a0aeccf38d7f77ee0348715adaf8340f65ad46c94a02c6b20e2f65d"
 dependencies = [
 "opentelemetry",
 "opentelemetry_sdk",
@@ -7390,15 +7379,15 @@ dependencies = [

 [[package]]
 name = "opentelemetry-semantic-conventions"
-version = "0.32.1"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c913ac17a6c451661ee255f4625d143e51647ae78ebd969b75e41c4442f4fe47"
+checksum = "d288b1e1bc3590052fbb8f0f635ef248dd9022cf1fbaeb6582b7ad6081a17b87"

 [[package]]
 name = "opentelemetry-stdout"
-version = "0.32.0"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a1b1c6a247d79091f0062a5f4bd058589525cf987a8d4c169440d9c1be72f0ad"
+checksum = "9f71803f481d3f5ccbe477edb42a78f473613712b2327835911a9ea0ab069824"
 dependencies = [
 "chrono",
 "opentelemetry",
@@ -7407,9 +7396,9 @@ dependencies = [

 [[package]]
 name = "opentelemetry_sdk"
-version = "0.32.1"
+version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9b59f80e1ac4d5ff7a2db8fb6c80badb7f0f3f858211fba08dd9aaec750894f9"
+checksum = "cb39533d9d1c912123efd7d41d7e0c29d16917b60ce15b4c8d87cb1af7f67520"
 dependencies = [
 "futures-channel",
 "futures-executor",
@@ -7904,9 +7893,9 @@ checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"

 [[package]]
 name = "pin-utils"
-version = "0.1.0"
+version = "0.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
+checksum = "13bee6c73da26345c729282832b60b0363cf3dd9f4bfd81d8551b7a1c889a113"

 [[package]]
 name = "piper"
@@ -8438,9 +8427,9 @@ dependencies = [

 [[package]]
 name = "pulldown-cmark-to-cmark"
-version = "22.0.1"
+version = "22.0.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ab1ad36992cead65f02aa399a373a42730922f1525d988172634fdefdecb8a60"
+checksum = "84bbb29c624230c4bd1047bbdb2aa47e41c860e9665ce62ba9504eebe91bf867"
 dependencies = [
 "pulldown-cmark",
 ]
@@ -8675,9 +8664,9 @@ dependencies = [

 [[package]]
 name = "quinn-proto"
-version = "0.11.18"
+version = "0.11.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
+checksum = "0e750cca55fe4f0439a15d0bb529da9651e79993e8e72c61a899a36d462befbe"
 dependencies = [
 "aws-lc-rs",
 "bytes",
@@ -8698,9 +8687,9 @@ dependencies = [

 [[package]]
 name = "quinn-udp"
-version = "0.5.15"
+version = "0.5.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
+checksum = "af66907df18639dcf4db56ca65490cabc4b27a97dbadd96f2926cca73298f016"
 dependencies = [
 "cfg_aliases",
 "libc",
@@ -8943,9 +8932,9 @@ dependencies = [

 [[package]]
 name = "redis"
-version = "1.7.0"
+version = "1.7.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2acbc41a996f7652b2ddd9dfd98cc4ff602cfd742ae35382f07f608405ab50ed"
+checksum = "ed5b98ae99461e02895c3a96a25dbe4d852a3d5e5b815d31d5d9e4b5344d99e9"
 dependencies = [
 "arc-swap",
 "arcstr",
@@ -9381,9 +9370,9 @@ dependencies = [

 [[package]]
 name = "russh-sftp"
-version = "3.0.0"
+version = "3.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "093197e526668d92bba562e2bbbe98d1af9831bf080b619c736316ca1fa35101"
+checksum = "9bb17f72aba4314656b359ffba72c68dcbd70ed9e97497898b75498fe7198042"
 dependencies = [
 "bitflags 2.13.2",
 "bytes",
@@ -9518,6 +9507,7 @@ dependencies = [
 "metrics-util",
 "mime_guess",
 "moka",
+ "object 0.40.0",
 "opentelemetry",
 "opentelemetry-proto",
 "opentelemetry_sdk",
@@ -9597,6 +9587,7 @@ dependencies = [
 "snap",
 "socket2",
 "subtle",
+ "symbolic-demangle",
 "sysinfo",
 "temp-env",
 "tempfile",
@@ -9697,6 +9688,18 @@ dependencies = [
 "serde_json",
 ]

+[[package]]
+name = "rustfs-connect-symbol-catalog"
+version = "1.0.1"
+dependencies = [
+ "hex-simd",
+ "object 0.40.0",
+ "serde",
+ "serde_json",
+ "sha2 0.11.0",
+ "symbolic-demangle",
+]
+
 [[package]]
 name = "rustfs-credentials"
 version = "1.0.1"
@@ -9748,7 +9751,6 @@ version = "1.0.1"
 dependencies = [
 "arc-swap",
 "async-channel",
- "async-recursion",
 "async-trait",
 "aws-credential-types",
 "aws-sdk-s3",
@@ -9949,6 +9951,7 @@ dependencies = [
 "thiserror 2.0.21",
 "tokio",
 "tokio-util",
+ "tonic",
 "tracing",
 "uuid",
 "walkdir",
@@ -10712,7 +10715,6 @@ dependencies = [
 name = "rustfs-s3select-query"
 version = "1.0.1"
 dependencies = [
- "async-recursion",
 "async-trait",
 "bytes",
 "datafusion",
@@ -11120,9 +11122,9 @@ dependencies = [

 [[package]]
 name = "rustls-connector"
-version = "0.23.8"
+version = "0.23.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1babecfcc65b139b812e74bcc7f9ec7b4e00db659fd42d99567b7e77f0c714c6"
+checksum = "524994fd6957827c775abc941f41a146ce351bbd0a9b8cf632ee37f43a13c205"
 dependencies = [
 "futures-io",
 "futures-rustls",
@@ -11157,9 +11159,9 @@ dependencies = [

 [[package]]
 name = "rustls-platform-verifier"
-version = "0.7.0"
+version = "0.7.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
+checksum = "1167586491e2b18b8bfbb293e8180ec17c201c4f076d7cb3070ca964e7598f98"
 dependencies = [
 "core-foundation 0.10.1",
 "core-foundation-sys",
@@ -11178,9 +11180,9 @@ dependencies = [

 [[package]]
 name = "rustls-platform-verifier-android"
-version = "0.1.1"
+version = "0.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
+checksum = "eec689c0bc40ff2458a5977b6619cb718087084a18e02a131c599b62d05e1a5f"

 [[package]]
 name = "rustls-webpki"
@@ -11641,9 +11643,9 @@ dependencies = [

 [[package]]
 name = "serde_with"
-version = "3.23.0"
+version = "3.24.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "935177bb8c0cd8ca1a4e6d1a2ac8988bea69cab4f9d3a31311e012ad27868ea4"
+checksum = "df9adc193c780ef8f159aee8b61e2d5801aaa555e6eb0947fe45530ec506296f"
 dependencies = [
 "base64 0.23.1",
 "bs58",
@@ -11662,9 +11664,9 @@ dependencies = [

 [[package]]
 name = "serde_with_macros"
-version = "3.23.0"
+version = "3.24.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1d607aa01a3cb0ad757d6fd216136910db3c97b102fe686585689615a02dbcdc"
+checksum = "3e17bbc68e28663bbbb90df47e058aa7eda4fb445b89fe70457bb94fbccf6e49"
 dependencies = [
 "darling 0.24.1",
 "proc-macro2",
@@ -11899,9 +11901,9 @@ dependencies = [

 [[package]]
 name = "siphasher"
-version = "1.0.3"
+version = "1.0.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
+checksum = "33f4fe9184a62d842c9ef383018f3306d8ba224fd9d836f56d7288308847c256"

 [[package]]
 name = "sketches-ddsketch"
@@ -11951,9 +11953,9 @@ dependencies = [

 [[package]]
 name = "smallvec"
-version = "1.16.1"
+version = "1.16.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ba467056f1b547ed52077911161fc86985becbc60e8e1857c8a144dab0def891"
+checksum = "f9395f0f0eee849a9b707b2f06bb92a6a422090e2123bb2ef8e87a0e61892a8e"
 dependencies = [
 "serde",
 ]
@@ -12233,9 +12235,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"

 [[package]]
 name = "suppaftp"
-version = "12.0.1"
+version = "12.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ba93b140f4d07f1b3f3fb4d7264b2d422d5c8b4e925eb6a2646e1342d5d2070a"
+checksum = "b161d71217da0cc91bd09c8f98ac5ad2a5e5b2735c6dfe1c6cb08b4471612f35"
 dependencies = [
 "async-trait",
 "chrono",
@@ -12726,9 +12728,9 @@ dependencies = [

 [[package]]
 name = "tokio-rustls"
-version = "0.26.5"
+version = "0.26.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0c85f2c3ef0b1cd58b36682f4b17aaa995f0e5db534d85692b4903abce21f67"
+checksum = "c9cc2678c2cdd569ef8215e2afd7954ada2ae20b4fdd2c5fe6139a3b02d105db"
 dependencies = [
 "rustls",
 "tokio",
@@ -12748,9 +12750,9 @@ dependencies = [

 [[package]]
 name = "tokio-test"
-version = "0.4.5"
+version = "0.4.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3f6d24790a10a7af737693a3e8f1d03faef7e6ca0cc99aae5066f533766de545"
+checksum = "12bc5db7778a8e08ee255090861e0e7aac55e784300077e7ed293c7be36168b2"
 dependencies = [
 "futures-core",
 "tokio",
@@ -13038,9 +13040,9 @@ dependencies = [

 [[package]]
 name = "tracing-opentelemetry"
-version = "0.33.0"
+version = "0.34.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "adbc64cba7137545b8044cb1fe9814f7aacf3c6b5f9b45be8bb5db538befdb26"
+checksum = "0a904802a1b902f43638b677ff2a650847e3b4404101b6c586d648e8c1e3e8fe"
 dependencies = [
 "js-sys",
 "opentelemetry",
@@ -13401,9 +13403,9 @@ dependencies = [

 [[package]]
 name = "wasm-bindgen"
-version = "0.2.128"
+version = "0.2.129"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aecb87a33d3b0c5e3b7aa46336eaf486cffafbd281b195e4c8b80d50df2351bf"
+checksum = "9bb54f33acc68fd454578d9820b0bde1a1a3d17aa17bb7b6595806d02886d409"
 dependencies = [
 "cfg-if",
 "once_cell",
@@ -13414,19 +13416,20 @@ dependencies = [

 [[package]]
 name = "wasm-bindgen-futures"
-version = "0.4.78"
+version = "0.4.79"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6ef4c5d3d2cdf5c54f4231181768f5510842e350db025faf1f7163b1030ed928"
+checksum = "3cbab34de2d982e9b48e18d216d04c4a6f641066ff19ffb699980f591ee3610e"
 dependencies = [
 "js-sys",
+ "tokio",
 "wasm-bindgen",
 ]

 [[package]]
 name = "wasm-bindgen-macro"
-version = "0.2.128"
+version = "0.2.129"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a690d511e3c1a8b3a55e33511e3c2c00c78415cd23650f32b808627f5696b9ed"
+checksum = "2e29d0c35b16e224a7eeb5cd2d25e3e1968fbd65604117b44d3b789d00ee8535"
 dependencies = [
 "quote",
 "wasm-bindgen-macro-support",
@@ -13434,9 +13437,9 @@ dependencies = [

 [[package]]
 name = "wasm-bindgen-macro-support"
-version = "0.2.128"
+version = "0.2.129"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "411e4887f0071ef2d2164a9d5fdf2d20efbef78fccd3a78b0c10a1dc5295e48a"
+checksum = "6f501a8bc3719dba86ef8ae4728879c08001bea749eb1333ac5b91e040e2a6b7"
 dependencies = [
 "bumpalo",
 "proc-macro2",
@@ -13447,9 +13450,9 @@ dependencies = [

 [[package]]
 name = "wasm-bindgen-shared"
-version = "0.2.128"
+version = "0.2.129"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "81941cd78d0c92026c33e5e01312845a4cb1e9af3407f9134b100dd03144103e"
+checksum = "23f0c9c52aa7cd7d77769a4cfe2a9adb1b331f489a41d912ce14513d5ab995c6"
 dependencies = [
 "unicode-ident",
 ]
@@ -13469,9 +13472,9 @@ dependencies = [

 [[package]]
 name = "web-sys"
-version = "0.3.105"
+version = "0.3.106"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9fbddc4a036f00ec4f18c83445bd3115cb306a91da554919a099d9222fe4a7f8"
+checksum = "88261b9deccee56594c11a3460c462c41f58d148598fe70ad77070126a68aba4"
 dependencies = [
 "js-sys",
 "wasm-bindgen",
@@ -13977,9 +13980,9 @@ dependencies = [

 [[package]]
 name = "xxhash-rust"
-version = "0.8.18"
+version = "0.8.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6"
+checksum = "550a2b930b62486a393c52d5c3b84bff264b28aa437ed64694d31e93b1757af7"

 [[package]]
 name = "yansi"
@@ -14010,9 +14013,9 @@ dependencies = [

 [[package]]
 name = "yoke-derive"
-version = "0.8.3"
+version = "0.8.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "33811428bee40dbceb6d545e95754741d17a6aef9a4849f0fd62e2ba4f412a78"
+checksum = "ec8ebde2db3681e8c9980cc27822030e68752690ddfa9473e739aeb4dbde6d71"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -14022,18 +14025,18 @@ dependencies = [

 [[package]]
 name = "zerocopy"
-version = "0.8.58"
+version = "0.8.59"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c17e8fafad82b542ff3717217ecdc736231b59e387768c9630123b4ce4d2db44"
+checksum = "6df92bf3d9227be3d53173901ddbffac2babc27ae50f397776ffd6dc33f800cb"
 dependencies = [
 "zerocopy-derive",
 ]

 [[package]]
 name = "zerocopy-derive"
-version = "0.8.58"
+version = "0.8.59"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "595f56e044df4f46a0c9a626f65c3d99eb8488f7e8a8baa12dd76326d9710bf2"
+checksum = "ac4f328cf2f05d084e496c3e9c3f33ed0a183656a16e1fcec4d464d8373aec82"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -20,6 +20,7 @@ members = [
    "crates/common", # Shared utilities and data structures
    "crates/data-usage", # Shared data usage models and algorithms
    "crates/config", # Configuration management
+    "crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
    "crates/credentials", # Credential management system
    "crates/crypto", # Cryptography and security features
    "crates/ecstore", # Erasure coding storage implementation
@@ -145,8 +146,7 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
 async-channel = "2.5.0"
 async_zip = { default-features = false, version = "0.0.19" }
 mysql_async = { default-features = false, version = "0.37.1" }
-async-compression = { version = "0.4.48" }
-async-recursion = "1.1.1"
+async-compression = { version = "0.4.50" }
 async-trait = "0.1.92"
 async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
@@ -159,7 +159,7 @@ pulsar = { default-features = false, version = "6.9.0" }
 lapin = { default-features = false, version = "4.12.0" }
 hyper = { version = "1.11.1" }
 hyper-rustls = { default-features = false, version = "0.27.10" }
-hyper-util = { version = "0.1.20" }
+hyper-util = { version = "0.1.21" }
 http = "1.5.0"
 http-body = "1.1.0"
 http-body-util = "0.1.5"
@@ -169,9 +169,9 @@ reqwest = "0.13.5"
 rustfs-kafka-async = { version = "1.3.1" }
 socket2 = { version = "0.6.5" }
 tokio = { version = "1.53.1" }
-tokio-rustls = { default-features = false, version = "0.26.5" }
+tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
-tokio-test = "0.4.5"
+tokio-test = "0.4.6"
 tokio-util = { version = "0.7.19" }
 tonic = { version = "0.14.6" }
 tonic-prost = { version = "0.14.6" }
@@ -244,13 +244,13 @@ atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.12.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-kms = { default-features = false, version = "1.121.0" }
-aws-sdk-s3 = { default-features = false, version = "1.149.0" }
-aws-sdk-sts = { default-features = false, version = "1.117.0" }
+aws-sdk-kms = { default-features = false, version = "1.122.0" }
+aws-sdk-s3 = { default-features = false, version = "1.151.0" }
+aws-sdk-sts = { default-features = false, version = "1.118.0" }
 aws-smithy-async = { version = "1.3.0" }
-aws-smithy-http-client = { default-features = false, version = "1.4.2" }
-aws-smithy-runtime-api = { version = "1.17.0" }
-aws-smithy-types = { version = "1.8.0" }
+aws-smithy-http-client = { default-features = false, version = "1.5.0" }
+aws-smithy-runtime-api = { version = "1.19.0" }
+aws-smithy-types = { version = "1.8.1" }
 base64-simd = "0.8.0"
 brotli = "9.0.0"
 clap = { version = "4.6.7" }
@@ -268,8 +268,8 @@ enumset = "1.1.14"
 faster-hex = "1.0.0"
 flate2 = "1.1.10"
 glob = "0.3.4"
-google-cloud-storage = "1.18.0"
-google-cloud-auth = "1.16.0"
+google-cloud-storage = "1.20.0"
+google-cloud-auth = "1.17.0"
 hashbrown = { version = "0.17.1" }
 # Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
 # every authenticator app expects). Already in the graph transitively.
@@ -278,7 +278,7 @@ hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 hostname = "0.4.2"
 ipnetwork = { version = "0.21.1" }
-lazy_static = "1.5.0"
+lazy_static = "1.5.1"
 libc = "0.2.189"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
@@ -292,6 +292,7 @@ moka = { version = "0.12.16" }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.13.0"
+object = { version = "=0.40.0", default-features = false }
 parking_lot = "0.12.5"
 path-absolutize = "4.0.1"
 percent-encoding = "2.3.2"
@@ -308,7 +309,7 @@ rustfs-erasure-codec = { version = "9.0.0" }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.34.0" }
-redis = { version = "1.7.0" }
+redis = { version = "1.7.1" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
 rust-embed = { version = "8.12.0" }
@@ -317,8 +318,8 @@ s3s = { version = "0.17.0", features = ["minio"] }
 s3s-sigv4 = "0.17.0"
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
-siphasher = "1.0.3"
-smallvec = { version = "1.16.1" }
+siphasher = "1.0.4"
+smallvec = { version = "1.16.2" }
 compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.3.0" }
@@ -332,7 +333,7 @@ tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
 tracing-error = "0.2.1"
-tracing-opentelemetry = { version = "0.33" }
+tracing-opentelemetry = { version = "0.34" }
 tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
@@ -344,7 +345,7 @@ walkdir = "2.5.0"
 winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
-xxhash-rust = { version = "0.8.18" }
+xxhash-rust = { version = "0.8.19" }
 zip = "8.6.0"
 zstd = "0.14.0"

@@ -352,22 +353,22 @@ zstd = "0.14.0"
 metrics = "0.24.6"
 metrics-util = "0.20"
 dial9 = { version = "0.5.2", default-features = false }
-opentelemetry = { version = "0.32.0" }
-opentelemetry-appender-tracing = { version = "0.32.0" }
-opentelemetry-otlp = { version = "0.32.0" }
-opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.32.1" }
-opentelemetry-semantic-conventions = { version = "0.32.1" }
-opentelemetry-stdout = { version = "0.32.0" }
+opentelemetry = { version = "0.33.0" }
+opentelemetry-appender-tracing = { version = "0.33.0" }
+opentelemetry-otlp = { version = "0.33.0" }
+opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
+opentelemetry_sdk = { version = "0.33.0" }
+opentelemetry-semantic-conventions = { version = "0.33.0" }
+opentelemetry-stdout = { version = "0.33.0" }
 pyroscope = { version = "2.1.1" }

 # FTP and SFTP
-libunftp = { version = "0.23.0" }
+libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.0.1" }
+suppaftp = { version = "12.1.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.63.3" }
-russh-sftp = "3.0.0"
+russh-sftp = "3.0.1"

 # WebDAV
 dav-server = "0.11.0"
@@ -375,7 +376,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.26.0", default-features = false }
+hotpath = { version = "0.28.0", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }

@@ -106,7 +106,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh

 RUN addgroup -g 10001 -S rustfs && \
    adduser -u 10001 -G rustfs -S rustfs -D && \
-    mkdir -p /data /logs && \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
+    ln -s /data/.rustfs/events /opt/rustfs/events && \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit && \
    chown -R rustfs:rustfs /data /logs && \
    chmod 0750 /data /logs

@@ -44,7 +44,9 @@ RUN set -eux; \
 WORKDIR /app

 RUN set -eux; \
-    mkdir -p /data /logs; \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
+    ln -s /data/.rustfs/events /opt/rustfs/events; \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit; \
    chown -R rustfs:rustfs /data /logs /app; \
    chmod 0750 /data /logs

@@ -110,7 +110,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh

 RUN groupadd -g 10001 rustfs && \
    useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
-    mkdir -p /data /logs && \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
+    ln -s /data/.rustfs/events /opt/rustfs/events && \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit && \
    chown -R rustfs:rustfs /data /logs && \
    chmod 0750 /data /logs

@@ -240,7 +240,9 @@ WORKDIR /app

 # Prepare data/log directories with sane defaults
 RUN set -eux; \
-    mkdir -p /data /logs; \
+    mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
+    ln -s /data/.rustfs/events /opt/rustfs/events; \
+    ln -s /data/.rustfs/audit /opt/rustfs/audit; \
    chown -R rustfs:rustfs /data /logs /app; \
    chmod 0750 /data /logs

@@ -195,7 +195,7 @@ docker run -d --name rustfs -p 9000:9000 \
 Notes:
 - `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
 - For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
+- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout.
 - `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
 - Since `1.0.0-beta.11`, webhook endpoints on private or container networks
  (`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
@@ -163,6 +163,15 @@ pub struct MrfDurableRepairAnchor {
    pub bucket_incarnation_id: Uuid,
 }

+/// A completed deep check proved that a legacy object is present and healthy,
+/// but the format has no independent identity commitment that can discharge a
+/// durable partial-write responsibility. Keep the journal entry and pause
+/// automatic retries until an operator has repaired the source of truth.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct MrfUnverifiedLegacyEvent {
+    pub anchor: MrfDurableRepairAnchor,
+}
+
 impl MrfDurableRepairAnchor {
    /// Build a dischargeable anchor only when the caller supplies the storage
    /// incarnation and the original ingress lease. Legacy replay records lack
@@ -324,6 +333,9 @@ pub enum MrfDurableAdmissionError {
 /// cancel repair.
 pub struct MrfDurableSubmission {
    pub intent: MrfIntent,
+    /// Bucket incarnation observed by the committed-write caller, when
+    /// available. Old journal records and legacy callers remain unbound.
+    pub source_bucket_incarnation_id: Option<Uuid>,
    pub response: oneshot::Sender<Result<(), MrfDurableAdmissionError>>,
 }

@@ -344,6 +356,19 @@ pub async fn persist_partial_write_intent(
    version_id: Option<Uuid>,
    scope: MrfScope,
 ) -> Result<(), MrfDurableAdmissionError> {
+    persist_partial_write_intent_with_incarnation(bucket, object, version_id, scope, None).await
+}
+
+pub async fn persist_partial_write_intent_with_incarnation(
+    bucket: &str,
+    object: &str,
+    version_id: Option<Uuid>,
+    scope: MrfScope,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
+    if source_bucket_incarnation_id.is_some_and(|incarnation| incarnation.is_nil()) {
+        return Err(MrfDurableAdmissionError::InvalidIdentity);
+    }
    let intent = MrfIntent {
        bucket: Arc::from(bucket),
        object: Arc::from(object),
@@ -355,7 +380,7 @@ pub async fn persist_partial_write_intent(
        enqueued_at_ms: unix_now_ms(),
        attempts: 0,
    };
-    persist_durable_intent(intent).await
+    persist_durable_intent(intent, source_bucket_incarnation_id).await
 }

 pub async fn persist_delete_marker_purge_intent(
@@ -368,6 +393,7 @@ pub async fn persist_delete_marker_purge_intent(
    if version_id.is_nil() {
        return Err(MrfDurableAdmissionError::InvalidIdentity);
    }
+    let source_bucket_incarnation_id = delete_marker_purge.bucket_incarnation_id;
    let intent = MrfIntent {
        bucket: Arc::from(bucket),
        object: Arc::from(object),
@@ -379,17 +405,23 @@ pub async fn persist_delete_marker_purge_intent(
        enqueued_at_ms: unix_now_ms(),
        attempts: 0,
    };
-    persist_durable_intent_unconditionally(intent).await
+    persist_durable_intent_unconditionally(intent, Some(source_bucket_incarnation_id)).await
 }

-async fn persist_durable_intent(intent: MrfIntent) -> Result<(), MrfDurableAdmissionError> {
+async fn persist_durable_intent(
+    intent: MrfIntent,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
    if !mrf_delivery_enabled() {
        return Err(MrfDurableAdmissionError::Disabled);
    }
-    persist_durable_intent_unconditionally(intent).await
+    persist_durable_intent_unconditionally(intent, source_bucket_incarnation_id).await
 }

-async fn persist_durable_intent_unconditionally(mut intent: MrfIntent) -> Result<(), MrfDurableAdmissionError> {
+async fn persist_durable_intent_unconditionally(
+    mut intent: MrfIntent,
+    source_bucket_incarnation_id: Option<Uuid>,
+) -> Result<(), MrfDurableAdmissionError> {
    if intent.bucket.is_empty()
        || intent.object.is_empty()
        || intent.bucket.len() > MRF_MAX_IDENTITY_COMPONENT
@@ -404,7 +436,11 @@ async fn persist_durable_intent_unconditionally(mut intent: MrfIntent) -> Result
    }
    let (response, receipt) = oneshot::channel();
    sender
-        .try_send(MrfDurableSubmission { intent, response })
+        .try_send(MrfDurableSubmission {
+            intent,
+            source_bucket_incarnation_id,
+            response,
+        })
        .map_err(|err| match err {
            mpsc::error::TrySendError::Full(_) => MrfDurableAdmissionError::Full,
            mpsc::error::TrySendError::Closed(_) => MrfDurableAdmissionError::Unavailable,
@@ -782,6 +818,8 @@ const MRF_REPAIRED_EVENT_CAP: usize = 4096;
 static MRF_REPAIRED_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfRepairedEvent>>> = OnceLock::new();
 static MRF_VERIFIED_REPAIR_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfVerifiedRepairEvent>>> =
    OnceLock::new();
+static MRF_UNVERIFIED_LEGACY_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfUnverifiedLegacyEvent>>> =
+    OnceLock::new();

 /// Record a legacy notification for compatibility. This is not an
 /// acknowledgement of storage verification or durable repair completion.
@@ -859,6 +897,44 @@ pub fn take_mrf_verified_repair_events_for(bucket: &str) -> Vec<MrfVerifiedRepai
    taken
 }

+/// Record a bounded, exact-identity notice that a legacy durable partial-write
+/// was checked but cannot be proven from its on-disk format. This is not a
+/// repair proof and must never release the durable responsibility.
+pub fn note_mrf_unverified_legacy(event: MrfUnverifiedLegacyEvent) {
+    let registry = MRF_UNVERIFIED_LEGACY_EVENTS.get_or_init(|| std::sync::Mutex::new(std::collections::VecDeque::new()));
+    let Ok(mut events) = registry.lock() else {
+        return;
+    };
+    if events.contains(&event) {
+        return;
+    }
+    if events.len() >= MRF_REPAIRED_EVENT_CAP {
+        events.pop_front();
+    }
+    events.push_back(event);
+}
+
+/// Take held-legacy notices for one bucket. The durable ledger still verifies
+/// the full anchor before changing the journal entry to its held state.
+pub fn take_mrf_unverified_legacy_events_for(bucket: &str) -> Vec<MrfUnverifiedLegacyEvent> {
+    let Some(registry) = MRF_UNVERIFIED_LEGACY_EVENTS.get() else {
+        return Vec::new();
+    };
+    let Ok(mut events) = registry.lock() else {
+        return Vec::new();
+    };
+    let mut taken = Vec::new();
+    events.retain(|event| {
+        if event.anchor.bucket.as_ref() == bucket {
+            taken.push(event.clone());
+            false
+        } else {
+            true
+        }
+    });
+    taken
+}
+
 #[cfg(test)]
 mod tests {
    use super::*;
@@ -1233,6 +1309,53 @@ mod tests {
        assert!(take_mrf_verified_repair_events_for("verified-bucket-a").is_empty());
        assert_eq!(take_mrf_verified_repair_events_for("verified-bucket-b").len(), 1);
    }
+
+    #[test]
+    fn unverified_legacy_events_are_bucket_scoped_and_bounded() {
+        let bucket = Arc::<str>::from("unverified-legacy-event-bucket");
+        let _ = take_mrf_unverified_legacy_events_for(bucket.as_ref());
+        let event = MrfUnverifiedLegacyEvent {
+            anchor: MrfDurableRepairAnchor {
+                kind: MrfKind::PartialWrite,
+                bucket: bucket.clone(),
+                object: Arc::from("unverified-object"),
+                version_id: None,
+                scope: Some(MrfScope {
+                    pool_index: 1,
+                    set_index: 2,
+                }),
+                delete_marker_purge: None,
+                lease: MrfIngressLease::new(912),
+                bucket_incarnation_id: Uuid::new_v4(),
+            },
+        };
+        note_mrf_unverified_legacy(event.clone());
+        note_mrf_unverified_legacy(event.clone());
+        assert_eq!(take_mrf_unverified_legacy_events_for(bucket.as_ref()), vec![event]);
+        assert!(take_mrf_unverified_legacy_events_for(bucket.as_ref()).is_empty());
+
+        let cap_bucket = Arc::<str>::from("unverified-legacy-event-cap-bucket");
+        for index in 0..MRF_REPAIRED_EVENT_CAP + 4 {
+            note_mrf_unverified_legacy(MrfUnverifiedLegacyEvent {
+                anchor: MrfDurableRepairAnchor {
+                    kind: MrfKind::PartialWrite,
+                    bucket: cap_bucket.clone(),
+                    object: Arc::from(format!("object-{index}")),
+                    version_id: None,
+                    scope: Some(MrfScope {
+                        pool_index: 1,
+                        set_index: 2,
+                    }),
+                    delete_marker_purge: None,
+                    lease: MrfIngressLease::new(u64::try_from(index).expect("test index fits")),
+                    bucket_incarnation_id: Uuid::new_v4(),
+                },
+            });
+        }
+        let events = take_mrf_unverified_legacy_events_for(cap_bucket.as_ref());
+        assert_eq!(events.len(), MRF_REPAIRED_EVENT_CAP);
+        assert_eq!(events.first().expect("bounded ring is non-empty").anchor.object.as_ref(), "object-4");
+    }
 }
 #[tokio::test]
 async fn partial_write_durable_admission_waits_for_persistence_and_renews_lease() {
@@ -168,9 +168,10 @@ Scanner cycle budget controls:

 ## Foreground write admission environment variables

-Large direct `PutObject` requests and multipart `UploadPart` requests share one
-per-process permit pool that bounds concurrent body ingest and storage writes.
-Small direct PUTs stay on the legacy path.
+Large direct `PutObject` requests, multipart `UploadPart` requests, and
+server-side multipart `UploadPartCopy` requests share one per-process permit
+pool that bounds concurrent body ingest and storage writes. Small direct PUTs
+stay on the legacy path.

 - `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE`
  - enables the default-on pool; `false` keeps only the soft request counter.
@@ -190,11 +191,11 @@ Small direct PUTs stay on the legacy path.
  - smallest `UploadPart` that takes a permit; `0` gates every part.
  - default is `0`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
-  - how long an `UploadPart` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
-  - default is `10000`. Parts wait before body ingest, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
-  - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. Keep it with margin below the shortest such timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
+  - how long an `UploadPart` or `UploadPartCopy` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
+  - default is `10000`. Parts wait before body ingest or source-object reads, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
+  - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. `UploadPartCopy` waits before taking bucket lifecycle locks or opening source readers, and uses the unknown-size weight because the authoritative copy length is available only after source metadata and range validation. Keep the wait with margin below the shortest client/proxy timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING`
-  - maximum `UploadPart` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
+  - maximum `UploadPart` and `UploadPartCopy` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
  - default is `0`, which derives 16 times the large-write slot count, or the explicit request-count limit (512 at stock settings). Automatic subdivision does not enlarge this queue.
  - each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory). Autotuned receive buffers can remain large on reused connections; queue depth does not imply a fixed per-connection memory cost.
 - `RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE`, `RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT`, `RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
@@ -387,7 +387,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 10
 /// Multipart pressure is often many moderate-sized parts rather than one very
 /// large request. The default gates every multipart part through the same permit
 /// pool as large/unknown-size PutObject while keeping small direct PUTs on the
-/// legacy path.
+/// legacy path. Server-side UploadPartCopy requests use this same multipart
+/// admission path with unknown-size weighting because source metadata and range
+/// validation happen after admission.
 pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str =
    "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
 pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 0;
@@ -415,7 +417,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
 /// wait with a 10 s request deadline. The wait must leave margin under the
 /// shortest of those, not merely fall below an SDK default, so the part
 /// receives S3 `SlowDown`/503 for the client to retry instead of losing its
-/// connection (issue #7385). `0` rejects immediately when the pool is full.
+/// connection (issue #7385). Server-side UploadPartCopy waits in the same queue
+/// before taking bucket lifecycle locks or opening source readers. `0` rejects
+/// immediately when the pool is full.
 pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str =
    "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
 pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_000;
@@ -427,7 +431,7 @@ pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_0
 // the wait past any client timeout.
 const _: () = assert!(DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS * 3 <= 30_000);

-/// Maximum multipart UploadPart requests waiting for a foreground write permit per process.
+/// Maximum multipart UploadPart or UploadPartCopy requests waiting for a foreground write permit per process.
 ///
 /// Parts beyond this queue depth are rejected with S3 `SlowDown`/503 without
 /// waiting, so a genuinely saturated node still fails fast instead of holding
@@ -0,0 +1,22 @@
+[package]
+name = "rustfs-connect-symbol-catalog"
+version.workspace = true
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+publish = false
+
+[dependencies]
+hex-simd.workspace = true
+object = { workspace = true, features = ["read", "std", "elf"] }
+serde = { workspace = true, features = ["derive"] }
+serde_json.workspace = true
+sha2.workspace = true
+# Match pyroscope's pprof-rs frame normalization exactly.
+symbolic-demangle = { version = "=13.9.0", default-features = false, features = ["rust"] }
+
+[dev-dependencies]
+object = { workspace = true, features = ["write", "elf"] }
+
+[lints]
+workspace = true
@@ -0,0 +1,284 @@
+// Copyright 2024 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+//! Release tooling only: derive reviewed CPU symbol candidates from the final ELF.
+
+use object::{Object, ObjectSymbol};
+use serde::Serialize;
+use sha2::{Digest, Sha256};
+use std::collections::BTreeSet;
+use std::fs::{self, OpenOptions};
+use std::io::{self, Read, Write};
+use std::path::Path;
+
+const MAX_BINARY_BYTES: u64 = 1_073_741_824;
+const MAX_CATALOG_BYTES: usize = 32 * 1024 * 1024;
+const MAX_SYMBOLS: usize = 200_000;
+const NORMALIZATION: &str = "pyroscope-2.1.1-symbolic-demangle-13.9.0";
+
+#[derive(Serialize)]
+#[serde(rename_all = "camelCase")]
+struct Catalog<'a> {
+    schema_version: u8,
+    normalization: &'static str,
+    source_commit: &'a str,
+    executable_sha256: String,
+    os_family: &'static str,
+    architecture: &'static str,
+    build_features: Vec<String>,
+    symbols: BTreeSet<String>,
+}
+
+fn invalid(message: &'static str) -> io::Error {
+    io::Error::new(io::ErrorKind::InvalidData, message)
+}
+
+fn symbol_name(raw: &[u8]) -> Option<String> {
+    // This is the same lossy UTF-8 conversion and demangler used by pprof-rs.
+    let name = symbolic_demangle::demangle(&String::from_utf8_lossy(raw)).into_owned();
+    if name.is_empty()
+        || name.len() > 4096
+        || matches!(name.as_str(), "Unknown" | "<unresolved>")
+        || name.bytes().any(|b| b < 32 || b == 127)
+    {
+        return None;
+    }
+    Some(name)
+}
+
+fn catalog(binary: &[u8], source: &str, target: &str, features: Vec<String>) -> io::Result<Vec<u8>> {
+    if source.len() != 40 || !source.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) {
+        return Err(invalid("source commit must be a lowercase full SHA"));
+    }
+    let expected_arch = match target {
+        "x86_64-unknown-linux-gnu" => object::Architecture::X86_64,
+        "aarch64-unknown-linux-gnu" => object::Architecture::Aarch64,
+        _ => return Err(invalid("only supported Linux GNU CPU profiling targets have catalogs")),
+    };
+    if features.is_empty()
+        || features.len() > 64
+        || !features.iter().any(|f| f == "pyroscope")
+        || features.iter().any(|f| {
+            f.is_empty()
+                || f.len() > 64
+                || !f.as_bytes().first().is_some_and(u8::is_ascii_lowercase)
+                || !f
+                    .bytes()
+                    .all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-' || b == b'_')
+        })
+        || features.windows(2).any(|pair| pair[0] >= pair[1])
+    {
+        return Err(invalid("build features must be sorted, unique and include pyroscope"));
+    }
+    let file = object::File::parse(binary).map_err(|_| invalid("cannot parse executable"))?;
+    if file.format() != object::BinaryFormat::Elf
+        || file.architecture() != expected_arch
+        || !matches!(file.kind(), object::ObjectKind::Executable | object::ObjectKind::Dynamic)
+    {
+        return Err(invalid("the executable does not match the Linux target"));
+    }
+    if file.symbol_table().is_none() {
+        return Err(invalid("the executable is stripped; retain its function symbol table"));
+    }
+    let mut symbols = BTreeSet::new();
+    let mut symbol_bytes = 0;
+    for symbol in file.symbols().chain(file.dynamic_symbols()) {
+        if symbol.kind() != object::SymbolKind::Text || symbol.is_undefined() {
+            continue;
+        }
+        if let Some(name) = symbol.name_bytes().ok().and_then(symbol_name)
+            && !symbols.contains(&name)
+        {
+            symbol_bytes += name.len() + 3;
+            if symbol_bytes > MAX_CATALOG_BYTES {
+                return Err(invalid("symbol catalog exceeds its byte limit"));
+            }
+            symbols.insert(name);
+        }
+        if symbols.len() > MAX_SYMBOLS {
+            return Err(invalid("symbol catalog exceeds its reviewed symbol limit"));
+        }
+    }
+    if symbols.is_empty() {
+        return Err(invalid("the executable has no usable function symbols"));
+    }
+    let catalog = Catalog {
+        schema_version: 1,
+        normalization: NORMALIZATION,
+        source_commit: source,
+        executable_sha256: hex_simd::encode_to_string(Sha256::digest(binary), hex_simd::AsciiCase::Lower),
+        os_family: "LINUX",
+        architecture: if expected_arch == object::Architecture::X86_64 {
+            "x86_64"
+        } else {
+            "aarch64"
+        },
+        build_features: features,
+        symbols,
+    };
+    let json = serde_json::to_vec(&catalog).map_err(io::Error::other)?;
+    if json.len() > MAX_CATALOG_BYTES {
+        return Err(invalid("symbol catalog exceeds its byte limit"));
+    }
+    Ok(json)
+}
+
+fn main() -> io::Result<()> {
+    let args: Vec<_> = std::env::args().skip(1).collect();
+    let [binary_path, source, target, features_json, output] = args.as_slice() else {
+        return Err(invalid(
+            "usage: rustfs-connect-symbol-catalog BINARY SOURCE_SHA TARGET FEATURES_JSON OUTPUT",
+        ));
+    };
+    let input = fs::File::open(binary_path)?;
+    if !input.metadata()?.is_file() || input.metadata()?.len() > MAX_BINARY_BYTES {
+        return Err(invalid("input must be a bounded regular executable"));
+    }
+    let mut binary = Vec::new();
+    input.take(MAX_BINARY_BYTES + 1).read_to_end(&mut binary)?;
+    if u64::try_from(binary.len()).map_err(io::Error::other)? > MAX_BINARY_BYTES {
+        return Err(invalid("executable grew beyond its byte limit"));
+    }
+    let features = serde_json::from_str(features_json).map_err(io::Error::other)?;
+    let json = catalog(&binary, source, target, features)?;
+    // Never overwrite another build's reviewed artifact.
+    let mut file = OpenOptions::new().write(true).create_new(true).open(Path::new(output))?;
+    file.write_all(&json)?;
+    file.sync_all()?;
+    println!("{}", hex_simd::encode_to_string(Sha256::digest(&json), hex_simd::AsciiCase::Lower));
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn executable() -> Vec<u8> {
+        let mut object =
+            object::write::Object::new(object::BinaryFormat::Elf, object::Architecture::X86_64, object::Endianness::Little);
+        let section = object.section_id(object::write::StandardSection::Text);
+        object.append_section_data(section, &[0xc3], 1);
+        object.add_symbol(object::write::Symbol {
+            name: b"_ZN4test4main17h0123456789abcdefE".to_vec(),
+            value: 0,
+            size: 1,
+            kind: object::SymbolKind::Text,
+            scope: object::SymbolScope::Linkage,
+            weak: false,
+            section: object::write::SymbolSection::Section(section),
+            flags: object::SymbolFlags::None,
+        });
+        let mut bytes = object.write().expect("write ELF symbol fixture");
+        // The object writer emits ET_REL; this fixture models the final ET_EXEC header.
+        bytes[16..18].copy_from_slice(&2_u16.to_le_bytes());
+        bytes
+    }
+
+    fn bulk_executable(count: usize, suffix_len: usize) -> Vec<u8> {
+        let mut object =
+            object::write::Object::new(object::BinaryFormat::Elf, object::Architecture::X86_64, object::Endianness::Little);
+        let section = object.section_id(object::write::StandardSection::Text);
+        object.append_section_data(section, &[0xc3], 1);
+        let suffix = "x".repeat(suffix_len);
+        for index in 0..count {
+            object.add_symbol(object::write::Symbol {
+                name: format!("rustfs::profile::function_{index:06}_{suffix}").into_bytes(),
+                value: 0,
+                size: 1,
+                kind: object::SymbolKind::Text,
+                scope: object::SymbolScope::Linkage,
+                weak: false,
+                section: object::write::SymbolSection::Section(section),
+                flags: object::SymbolFlags::None,
+            });
+        }
+        let mut binary = object.write().expect("write bulk ELF fixture");
+        binary[16..18].copy_from_slice(&2_u16.to_le_bytes());
+        binary
+    }
+
+    #[test]
+    fn catalog_binds_the_exact_executable_and_normalizes_function_names() {
+        let binary = executable();
+        let encoded =
+            catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).expect("generate catalog");
+        let value: serde_json::Value = serde_json::from_slice(&encoded).expect("catalog JSON");
+        assert_eq!(value["symbols"], serde_json::json!(["test::main"]));
+        assert_eq!(
+            value["executableSha256"],
+            hex_simd::encode_to_string(Sha256::digest(&binary), hex_simd::AsciiCase::Lower)
+        );
+        assert_eq!(value["normalization"], NORMALIZATION);
+        assert!(catalog(&binary, &"a".repeat(40), "aarch64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
+        assert!(catalog(&binary, "bad", "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
+        assert!(catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into(); 2]).is_err());
+    }
+
+    #[test]
+    fn stripped_executables_and_wrong_abis_are_refused() {
+        let mut binary = executable();
+        // Removing section headers models a fully stripped executable without dynsym.
+        binary[40..48].fill(0);
+        binary[60..64].fill(0);
+        let error = catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()])
+            .expect_err("stripped binary must not yield an empty but trusted catalog");
+        assert!(error.to_string().contains("stripped"));
+        assert!(catalog(&executable(), &"a".repeat(40), "x86_64-unknown-linux-musl", vec!["pyroscope".into()]).is_err());
+    }
+
+    #[test]
+    fn symbol_hash_matches_the_connect_nonce_vector() {
+        let nonce: Vec<u8> = (0..32).collect();
+        let name = symbol_name(b"core::hint::black_box").expect("reviewed function name");
+        let mut digest = Sha256::new();
+        digest.update(b"rustfs-connect-cpu-symbol-v1\0");
+        digest.update(nonce);
+        digest.update(name.as_bytes());
+        assert_eq!(
+            hex_simd::encode_to_string(digest.finalize(), hex_simd::AsciiCase::Lower),
+            "5d5b472a0dd09124bfedf0bd243f88d5e61952b8da15d7463da019137320d77c"
+        );
+    }
+
+    #[test]
+    fn unresolved_and_non_function_material_cannot_become_catalog_entries() {
+        assert!(symbol_name(b"Unknown").is_none());
+        assert!(symbol_name(b"<unresolved>").is_none());
+        assert!(symbol_name(b"secret\nname").is_none());
+        assert!(symbol_name(&vec![b'x'; 4097]).is_none());
+        assert!(catalog(b"not ELF", &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
+    }
+
+    #[test]
+    fn release_sized_catalog_retains_all_function_names() {
+        let binary = bulk_executable(110_000, 160);
+        let encoded = catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()])
+            .expect("retain complete bounded symbol catalog");
+        assert!(encoded.len() > 16 * 1024 * 1024);
+        assert!(encoded.len() <= MAX_CATALOG_BYTES);
+        let value: serde_json::Value = serde_json::from_slice(&encoded).expect("catalog JSON");
+        assert_eq!(value["symbols"].as_array().expect("symbol array").len(), 110_000);
+    }
+
+    #[test]
+    fn catalogs_over_either_resource_limit_are_refused() {
+        let source = "a".repeat(40);
+        let too_many = catalog(
+            &bulk_executable(MAX_SYMBOLS + 1, 0),
+            &source,
+            "x86_64-unknown-linux-gnu",
+            vec!["pyroscope".into()],
+        )
+        .expect_err("too many function names must be refused");
+        assert!(too_many.to_string().contains("reviewed symbol limit"));
+
+        let too_large = catalog(
+            &bulk_executable(10_000, 3_500),
+            &source,
+            "x86_64-unknown-linux-gnu",
+            vec!["pyroscope".into()],
+        )
+        .expect_err("oversized catalog must be refused");
+        assert!(too_large.to_string().contains("byte limit"));
+    }
+}
@@ -384,17 +384,17 @@ pub(crate) fn is_cluster_heal_coordination_unavailable(error: &(dyn std::error::
    message.contains("500 Internal Server Error") && message.contains("cluster heal coordination unavailable")
 }

-/// Wait for the restarted cluster to admit the first root heal request.
+/// Wait for the restarted cluster to admit a root heal request and return its response.
 pub(crate) async fn start_root_heal_when_control_ready(
    heal_url: &str,
    heal_body: &str,
    access_key: &str,
    secret_key: &str,
-) -> ChaosResult<()> {
+) -> ChaosResult<String> {
    let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(45);
    loop {
        match signed_admin_post(heal_url, Some(heal_body), access_key, secret_key).await {
-            Ok(_) => return Ok(()),
+            Ok(body) => return Ok(body),
            Err(error) if is_cluster_heal_coordination_unavailable(error.as_ref()) && tokio::time::Instant::now() < deadline => {
                tokio::time::sleep(std::time::Duration::from_millis(250)).await;
            }
@@ -22,6 +22,8 @@
 //! - Common test constants and utilities

 use aws_sdk_s3::config::{Credentials, Region};
+use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
+use aws_sdk_s3::operation::list_buckets::ListBucketsError;
 use aws_sdk_s3::{Client, Config};
 use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
 use http::header::{CONTENT_TYPE, HOST};
@@ -61,6 +63,41 @@ const TEST_PORT_COUNTER_PATH: &str = "/tmp/rustfs_e2e_next_port";
 const TEST_PORT_LOCK_DIR: &str = "/tmp/rustfs_e2e_port_allocator.lock";
 const TEST_PORT_LOCK_STALE_AFTER: Duration = Duration::from_secs(30);

+fn list_buckets_readiness_error(err: &SdkError<ListBucketsError>) -> String {
+    // SDK Display reports only the category. Do not expose raw response bodies or headers.
+    let service_code = err.as_service_error().and_then(|error| error.code()).filter(|code| {
+        matches!(
+            *code,
+            "ServiceUnavailable"
+                | "ServerNotInitialized"
+                | "InternalError"
+                | "AccessDenied"
+                | "InvalidAccessKeyId"
+                | "SignatureDoesNotMatch"
+        )
+    });
+    let transport = match err {
+        SdkError::DispatchFailure(failure) if failure.is_timeout() => Some("timeout"),
+        SdkError::DispatchFailure(failure) if failure.is_io() => Some("io"),
+        SdkError::DispatchFailure(failure) if failure.is_user() => Some("user"),
+        SdkError::DispatchFailure(failure) if failure.is_other() => Some("other"),
+        _ => None,
+    };
+    let mut source: Option<&(dyn std::error::Error + 'static)> = Some(err);
+    let mut io_error = None;
+    while let Some(error) = source {
+        if let Some(error) = error.downcast_ref::<std::io::Error>() {
+            io_error = Some((error.kind(), error.raw_os_error()));
+            break;
+        }
+        source = error.source();
+    }
+    format!(
+        "{err}; http_status={:?}; service_code={service_code:?}; transport={transport:?}; io_error={io_error:?}",
+        err.raw_response().map(|response| response.status().as_u16())
+    )
+}
+
 fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
    let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
    Some(log_dir.join(format!("{temp_name}.log")))
@@ -1575,8 +1612,9 @@ impl RustFSTestClusterEnvironment {
    ///
    /// Verifies service availability by calling the S3 `list_buckets` API against the requested node,
    /// retries up to 120 times with a 1-second interval between attempts.
-    async fn wait_for_node_service_ready(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    async fn wait_for_node_service_ready(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
        let client = self.create_s3_client(node_idx)?;
+        let mut last_error = None;

        for attempt in 0..120 {
            match client.list_buckets().send().await {
@@ -1584,13 +1622,28 @@ impl RustFSTestClusterEnvironment {
                    info!("Cluster node {} service ready after {} attempts", node_idx, attempt + 1);
                    return Ok(());
                }
-                Err(_) => {
+                Err(err) => {
+                    last_error = Some(err);
                    sleep(Duration::from_secs(1)).await;
                }
            }
        }

-        Err(format!("Cluster node {} service failed to become ready", node_idx).into())
+        let last_error = last_error.as_ref().map(list_buckets_readiness_error);
+        // Observe liveness only after the existing readiness budget is exhausted, before cleanup.
+        let process_status = match self.nodes[node_idx].process.as_mut() {
+            Some(process) => match process.try_wait() {
+                Ok(Some(status)) => format!("exited ({status})"),
+                Ok(None) => "running".to_string(),
+                Err(error) => format!("unavailable (kind={:?}, os_code={:?})", error.kind(), error.raw_os_error()),
+            },
+            None => "not started".to_string(),
+        };
+        Err(format!(
+            "Cluster node {node_idx} service failed to become ready; last ListBuckets error={last_error:?}; process_status={process_status}; capture_log_path={:?}",
+            self.node_capture_log_paths[node_idx]
+        )
+        .into())
    }

    /// Create an S3 client configured to communicate with a specific cluster node.
@@ -2101,6 +2154,34 @@ mod tests {
        verify_awscurl_path(&executable).expect("an available client with a working help command should pass");
    }

+    #[test]
+    fn readiness_error_reports_transport_without_sensitive_source_text() {
+        use aws_sdk_s3::error::ConnectorError;
+
+        let sensitive = "request credentials must not appear in diagnostics";
+        for (connector, category) in [
+            (
+                ConnectorError::io(std::io::Error::new(ErrorKind::ConnectionReset, sensitive).into()),
+                "io",
+            ),
+            (ConnectorError::timeout(sensitive.into()), "timeout"),
+            (ConnectorError::user(sensitive.into()), "user"),
+            (ConnectorError::other(sensitive.into(), None), "other"),
+        ] {
+            let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(connector));
+            assert!(diagnostic.contains(&format!("transport=Some(\"{category}\")")), "{diagnostic}");
+            assert!(diagnostic.contains("http_status=None"), "{diagnostic}");
+            assert!(!diagnostic.contains(sensitive), "{diagnostic}");
+            if category == "io" {
+                assert!(diagnostic.contains("io_error=Some((ConnectionReset, None))"), "{diagnostic}");
+            }
+        }
+        let os_error = std::io::Error::from_raw_os_error(13);
+        let expected = format!("io_error=Some(({:?}, Some(13)))", os_error.kind());
+        let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(ConnectorError::io(os_error.into())));
+        assert!(diagnostic.contains(&expected), "{diagnostic}");
+    }
+
    #[test]
    fn capture_log_path_uses_temp_directory_basename() {
        assert_eq!(
@@ -282,7 +282,7 @@ async fn sdk_list_bucket_and_list_bucket_versions_permissions_are_independent()
 }

 #[tokio::test]
-async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers() -> TestResult {
+async fn explicit_force_delete_permission_removes_prefix_versions_and_delete_markers() -> TestResult {
    init_logging();
    let mut env = RustFSTestEnvironment::new().await?;
    env.start_rustfs_server(vec![]).await?;
@@ -300,7 +300,14 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
    put(&root, bucket, "folder-sibling/keep.txt").await?;
    let keep = versions(&root, bucket, "keep.txt").await?;
    let sibling = versions(&root, bucket, "folder-sibling/").await?;
-    let user = policy_user(&env, "consoleAdmin", None).await?;
+    let user = policy_user(
+        &env,
+        "explicit-force-delete",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")}
+        ]})),
+    )
+    .await?;

    force_delete(&user, bucket, "folder/").await?;
    assert!(
@@ -322,6 +329,126 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
    Ok(())
 }

+#[tokio::test]
+async fn force_delete_header_requires_explicit_permission() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(vec![]).await?;
+    let root = env.create_s3_client();
+    let admin = policy_user(&env, "consoleAdmin", None).await?;
+
+    let owner_bucket = "force-owner-bucket";
+    root.create_bucket().bucket(owner_bucket).send().await?;
+    put(&root, owner_bucket, "kept-until-force.txt").await?;
+    force_delete_bucket(&root, owner_bucket, "x-rustfs-force-delete").await?;
+    assert!(
+        head_bucket_missing(&root, owner_bucket).await?,
+        "root force-delete must remove a non-empty bucket"
+    );
+
+    let denied_bucket = "force-admin-denied";
+    root.create_bucket().bucket(denied_bucket).send().await?;
+    put(&root, denied_bucket, "folder/child.txt").await?;
+    put(&root, denied_bucket, "plain.txt").await?;
+    assert_boxed_denied(force_delete(&admin, denied_bucket, "folder/").await);
+    assert!(
+        !versions(&root, denied_bucket, "folder/").await?.is_empty(),
+        "consoleAdmin without s3:ForceDeleteObject must not recursively delete"
+    );
+    assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-rustfs-force-delete").await);
+    assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-minio-force-delete").await);
+    assert!(
+        !head_bucket_missing(&root, denied_bucket).await?,
+        "consoleAdmin without s3:ForceDeleteBucket must not force-delete a non-empty bucket"
+    );
+    let not_empty = admin.delete_bucket().bucket(denied_bucket).send().await;
+    let not_empty = not_empty.expect_err("plain delete of a non-empty bucket must fail");
+    assert_eq!(
+        not_empty.as_service_error().and_then(|error| error.code()),
+        Some("BucketNotEmpty"),
+        "plain delete must keep the emptiness check, got {not_empty:?}"
+    );
+    admin.delete_object().bucket(denied_bucket).key("plain.txt").send().await?;
+    assert!(
+        versions(&root, denied_bucket, "plain.txt").await?.is_empty(),
+        "plain DeleteObject without the force header must stay allowed for consoleAdmin"
+    );
+
+    let object_bucket = "force-explicit-object";
+    root.create_bucket().bucket(object_bucket).send().await?;
+    put(&root, object_bucket, "folder/child.txt").await?;
+    put(&root, object_bucket, "keep.txt").await?;
+    let keep = versions(&root, object_bucket, "keep.txt").await?;
+    let object_user = policy_user(
+        &env,
+        "explicit-object-force",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{object_bucket}/*")}
+        ]})),
+    )
+    .await?;
+    force_delete(&object_user, object_bucket, "folder/").await?;
+    assert!(versions(&root, object_bucket, "folder/").await?.is_empty());
+    assert_eq!(versions(&root, object_bucket, "keep.txt").await?, keep);
+
+    let allowed_bucket = "force-explicit-bucket";
+    root.create_bucket().bucket(allowed_bucket).send().await?;
+    put(&root, allowed_bucket, "inside.txt").await?;
+    let bucket_user = policy_user(
+        &env,
+        "explicit-bucket-force",
+        Some(json!({"Version":"2012-10-17","Statement":[
+            {"Effect":"Allow","Action":["s3:DeleteBucket","s3:ForceDeleteBucket"],"Resource":[
+                format!("arn:aws:s3:::{allowed_bucket}"), format!("arn:aws:s3:::{allowed_bucket}/*")
+            ]}
+        ]})),
+    )
+    .await?;
+    force_delete_bucket(&bucket_user, allowed_bucket, "x-minio-force-delete").await?;
+    assert!(
+        head_bucket_missing(&root, allowed_bucket).await?,
+        "an explicit s3:ForceDeleteBucket grant must remove a non-empty bucket"
+    );
+    Ok(())
+}
+
+async fn force_delete_bucket(
+    client: &Client,
+    bucket: &str,
+    header: &str,
+) -> Result<(), Box<SdkError<aws_sdk_s3::operation::delete_bucket::DeleteBucketError>>> {
+    let header: &'static str = match header {
+        "x-rustfs-force-delete" => "x-rustfs-force-delete",
+        "x-minio-force-delete" => "x-minio-force-delete",
+        other => panic!("unexpected force-delete header {other}"),
+    };
+    client
+        .delete_bucket()
+        .bucket(bucket)
+        .customize()
+        .mutate_request(move |request| {
+            request.headers_mut().insert(header, "true");
+        })
+        .send()
+        .await
+        .map(|_| ())
+        .map_err(Box::new)
+}
+
+async fn head_bucket_missing(client: &Client, bucket: &str) -> TestResult<bool> {
+    match client.head_bucket().bucket(bucket).send().await {
+        Ok(_) => Ok(false),
+        Err(error) => {
+            let code = error.as_service_error().and_then(|service| service.code());
+            if matches!(code, Some("NotFound" | "NoSuchBucket")) {
+                Ok(true)
+            } else {
+                Err(error.into())
+            }
+        }
+    }
+}
+
 #[tokio::test]
 async fn force_delete_authorizes_only_its_path_scope_without_list_permissions() -> TestResult {
    init_logging();
@@ -339,7 +466,7 @@ async fn force_delete_authorizes_only_its_path_scope_without_list_permissions()
        &env,
        "delete-only",
        Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":[
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":[
                format!("arn:aws:s3:::{bucket}/selected.txt"), format!("arn:aws:s3:::{bucket}/selected.txt/*")
            ]},
            {"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/selected.txt-sibling")}
@@ -376,7 +503,7 @@ async fn force_directory_delete_cannot_remove_an_unauthorized_colliding_parent()
        &env,
        "parent-denier",
        Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
            {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/collision.txt"),
             "Condition":{"StringEquals":{"s3:VersionId":protected_parent_version}}}
        ]})),
@@ -409,7 +536,7 @@ async fn force_unversioned_directory_requires_only_delete_object() -> TestResult
        &env,
        "unversioned-deleter",
        Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
            {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/*")}
        ]})),
    )
@@ -438,7 +565,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
        &env,
        "child-denier",
        Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
            {"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/folder/z-denied.txt")}
        ]})),
    )
@@ -468,7 +595,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
        &env,
        "replica-deleter",
        Some(json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
            {"Effect":"Allow","Action":"s3:ReplicateDelete","Resource":format!("arn:aws:s3:::{bucket}/*")}
        ]})),
    )
@@ -500,7 +627,7 @@ async fn force_delete_denied_historical_version_preserves_versions_and_markers()
    put(&root, bucket, "folder/a-allowed.txt").await?;
    let policy = |version: &str| {
        json!({"Version":"2012-10-17","Statement":[
-            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
            {"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/folder/*"),
             "Condition":{"StringEquals":{"s3:VersionId":version}}}
        ]})
@@ -628,7 +755,7 @@ async fn force_delete_checks_every_version_page_before_mutation() -> TestResult
        .try_collect::<Vec<_>>()
        .await?;
    put(&root, bucket, "folder/z-denied.txt").await?;
-    let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
+    let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
    let user = policy_user(
        &env,
        "paged-deleter",
@@ -34,6 +34,9 @@ use crate::common::{
 };
 use crate::replication_extension_test::LOOPBACK_REPLICATION_TARGET_ENV;
 use aws_sdk_s3::Client;
+use aws_sdk_s3::config::retry::RetryConfig;
+use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
+use aws_sdk_s3::operation::{get_object::GetObjectError, put_object::PutObjectError};
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
 use http::{Method, StatusCode};
@@ -41,7 +44,7 @@ use sha2::{Digest, Sha256};
 use std::collections::BTreeMap;
 use std::path::{Path, PathBuf};
 use std::time::Duration;
-use tokio::time::{Instant, sleep};
+use tokio::time::{Instant, sleep, timeout_at};
 use uuid::Uuid;

 pub(crate) type TestResult<T = ()> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
@@ -400,9 +403,9 @@ pub(crate) async fn put_inventory(
    Ok(inventory)
 }

-/// Retry only transport-level service availability failures while a data
-/// movement operation changes the pool map. Generic InternalError responses
-/// remain fatal because accepting them would hide server defects.
+/// Retry only explicit S3 availability responses while a data movement
+/// operation changes the pool map. Transport failures and InternalError
+/// responses remain fatal because callers already require a ready endpoint.
 pub(crate) async fn put_inventory_retrying(
    client: &Client,
    bucket: &str,
@@ -449,19 +452,48 @@ where
    let deadline = Instant::now() + timeout;
    let mut delay = Duration::from_millis(50);
    loop {
-        let last_error = match probe().await {
-            Ok(true) => return Ok(()),
-            Ok(false) => format!("{label} still false"),
-            Err(error) => error.to_string(),
-        };
        if Instant::now() >= deadline {
-            return Err(format!("{label} did not become true within {timeout:?}: {last_error}").into());
+            return Err(format!("{label} did not become true within {timeout:?}").into());
        }
-        sleep(delay).await;
+        // A pending request must consume the same budget as unsuccessful probes.
+        match timeout_at(deadline, probe()).await {
+            Err(_) => return Err(format!("{label} did not become true within {timeout:?}").into()),
+            Ok(Err(error)) => return Err(error),
+            Ok(Ok(true)) => return Ok(()),
+            Ok(Ok(false)) => {}
+        }
+        tokio::time::sleep_until((Instant::now() + delay).min(deadline)).await;
        delay = (delay * 2).min(Duration::from_secs(1));
    }
 }

+// Boxed SDK errors retain the operation's service code; Display only says
+// "service error" and cannot distinguish convergence from a server defect.
+pub(crate) fn s3_probe_error_code<'a>(error: &'a (dyn std::error::Error + Send + Sync + 'static)) -> Option<&'a str> {
+    error
+        .downcast_ref::<SdkError<GetObjectError>>()
+        .and_then(|error| error.as_service_error())
+        .and_then(ProvideErrorMetadata::code)
+        .or_else(|| {
+            error
+                .downcast_ref::<SdkError<PutObjectError>>()
+                .and_then(|error| error.as_service_error())
+                .and_then(ProvideErrorMetadata::code)
+        })
+}
+
+pub(crate) fn s3_probe_client(client: &Client) -> Client {
+    // The probe owns the retry policy. SDK retries must not hide a response
+    // that the probe would classify as fatal.
+    Client::from_conf(
+        client
+            .config()
+            .to_builder()
+            .retry_config(RetryConfig::standard().with_max_attempts(1))
+            .build(),
+    )
+}
+
 pub(crate) async fn cluster_admin(
    cluster: &RustFSTestClusterEnvironment,
    method: Method,
@@ -598,15 +630,20 @@ pub(crate) async fn wait_for_replicated_bytes(
    expected: &[u8],
    timeout: Duration,
 ) -> TestResult {
+    let client = s3_probe_client(client);
    wait_until(
        timeout,
        || async {
-            match get_object_bytes(client, bucket, key).await {
+            match get_object_bytes(&client, bucket, key).await {
                Ok(got) if got.as_slice() == expected => Ok(true),
-                Ok(_) => Ok(false),
+                Ok(got) => Err(format!(
+                    "replicated object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                    sha256_hex(expected),
+                    sha256_hex(&got)
+                )
+                .into()),
                Err(error) => {
-                    let message = error.to_string();
-                    if message.contains("NoSuchKey") || message.contains("NotFound") {
+                    if matches!(s3_probe_error_code(error.as_ref()), Some("NoSuchKey" | "NotFound")) {
                        Ok(false)
                    } else {
                        Err(error)
@@ -988,6 +1025,7 @@ pub(crate) async fn list_pools_json(cluster: &RustFSTestClusterEnvironment) -> T
 }

 pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, timeout: Duration) -> TestResult {
+    let client = s3_probe_client(client);
    wait_until(
        timeout,
        || {
@@ -999,8 +1037,7 @@ pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body:
                match put_object(&client, &bucket, &key, body).await {
                    Ok(()) => Ok(true),
                    Err(error) => {
-                        let message = error.to_string();
-                        if message.contains("SlowDown") || message.contains("ServiceUnavailable") || message.contains("503") {
+                        if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
                            Ok(false)
                        } else {
                            Err(error)
@@ -1021,19 +1058,20 @@ pub(crate) async fn retrying_get_equals(
    expected: &[u8],
    timeout: Duration,
 ) -> TestResult {
+    let client = s3_probe_client(client);
    wait_until(
        timeout,
        || async {
-            match get_object_bytes(client, bucket, key).await {
+            match get_object_bytes(&client, bucket, key).await {
                Ok(got) if got.as_slice() == expected => Ok(true),
-                Ok(_) => Ok(false),
+                Ok(got) => Err(format!(
+                    "object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                    sha256_hex(expected),
+                    sha256_hex(&got)
+                )
+                .into()),
                Err(error) => {
-                    let message = error.to_string();
-                    if message.contains("NoSuchKey")
-                        || message.contains("SlowDown")
-                        || message.contains("ServiceUnavailable")
-                        || message.contains("503")
-                    {
+                    if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
                        Ok(false)
                    } else {
                        Err(error)
@@ -1046,6 +1084,162 @@ pub(crate) async fn retrying_get_equals(
    .await
 }

+#[cfg(test)]
+mod retry_tests {
+    use super::*;
+    use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation, SeedMetadata};
+    use std::cell::Cell;
+
+    fn client(target: &FakeS3Target) -> Client {
+        Client::from_conf(build_test_s3_config(
+            target.endpoint(),
+            FAKE_ACCESS_KEY,
+            FAKE_SECRET_KEY,
+            None,
+            "distributed-retry-test",
+        ))
+    }
+
+    #[tokio::test]
+    async fn wait_until_preserves_first_error() {
+        let attempts = Cell::new(0);
+        let error = wait_until(
+            Duration::from_secs(5),
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(if attempts.get() == 1 {
+                    Err(std::io::Error::new(std::io::ErrorKind::PermissionDenied, "permanent probe failure").into())
+                } else {
+                    Ok(true)
+                })
+            },
+            "permanent failure",
+        )
+        .await
+        .expect_err("a later success must not hide a permanent failure");
+        assert_eq!(attempts.get(), 1);
+        assert_eq!(
+            error
+                .downcast_ref::<std::io::Error>()
+                .expect("preserve the original error")
+                .kind(),
+            std::io::ErrorKind::PermissionDenied
+        );
+    }
+
+    #[tokio::test]
+    async fn wait_until_retries_explicit_pending_state() -> TestResult {
+        let attempts = Cell::new(0);
+        wait_until(
+            Duration::from_secs(5),
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(Ok(attempts.get() == 2))
+            },
+            "eventual readiness",
+        )
+        .await?;
+        assert_eq!(attempts.get(), 2);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn wait_until_bounds_a_pending_probe() {
+        let error = tokio::time::timeout(
+            Duration::from_secs(5),
+            wait_until(Duration::from_millis(10), std::future::pending, "hung probe"),
+        )
+        .await
+        .expect("the probe deadline must finish before the test watchdog")
+        .expect_err("a permanently pending probe must time out");
+        assert!(error.to_string().contains("hung probe did not become true"));
+    }
+
+    #[tokio::test]
+    async fn wait_until_does_not_start_a_probe_after_its_deadline() {
+        let attempts = Cell::new(0);
+        wait_until(
+            Duration::ZERO,
+            || {
+                attempts.set(attempts.get() + 1);
+                std::future::ready(Ok(true))
+            },
+            "expired budget",
+        )
+        .await
+        .expect_err("an expired budget must not admit another probe");
+        assert_eq!(attempts.get(), 0);
+    }
+
+    #[tokio::test]
+    async fn retrying_put_does_not_hide_internal_error() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
+        let error = retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5))
+            .await
+            .expect_err("InternalError must remain fatal even if the next PUT would succeed");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn retrying_put_allows_transient_availability_errors() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        for status in [429, 503] {
+            target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
+            retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5)).await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn object_polls_reject_wrong_object_bytes() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.put_seed_object("retry-probe", "key", "wrong bytes", &SeedMetadata::default());
+        let error = retrying_get_equals(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+            .await
+            .expect_err("a successful response with wrong bytes must fail immediately");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.take_requests().len(), 1);
+        let error = wait_for_replicated_bytes(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+            .await
+            .expect_err("replication may lag, but it must not return corrupt bytes");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn object_polls_distinguish_committed_and_replicated_keys() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("retry-probe");
+        target.put_seed_object("retry-probe", "key", "expected bytes", &SeedMetadata::default());
+        let client = client(&target);
+        for (status, expected_code) in [(404, "NoSuchKey"), (500, "InternalError"), (403, "AccessDenied")] {
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            let error = retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
+                .await
+                .expect_err("an acknowledged object must not disappear or fail before a later successful read");
+            assert_eq!(s3_probe_error_code(error.as_ref()), Some(expected_code));
+            assert_eq!(target.take_requests().len(), 1);
+        }
+        for status in [429, 503] {
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+        target.inject(Operation::GetObject, FaultAction::ResponseStatus(404), 1);
+        wait_for_replicated_bytes(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
+        assert_eq!(target.requests().len(), 2);
+        Ok(())
+    }
+}
+
 #[tokio::test]
 async fn append_single_node_pool_extends_ellipses_volumes() {
    let mut env = RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]))
@@ -26,7 +26,7 @@

 use super::harness::{
    DistCluster, DistLayout, TestResult, assert_object_bytes, cluster_admin_ok, enable_versioning, get_object_bytes, put_object,
-    unique_bucket, wait_until,
+    s3_probe_client, s3_probe_error_code, sha256_hex, unique_bucket, wait_until,
 };
 use crate::common::{
    AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
@@ -111,6 +111,7 @@ async fn create_iam_user(dist: &DistCluster, user: &str, secret: &str, policy_na
 }

 async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, label: &str) -> TestResult {
+    let client = s3_probe_client(client);
    wait_until(
        CREDENTIAL_TIMEOUT,
        || {
@@ -119,8 +120,18 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
            let key = key.to_string();
            let body = body.clone();
            async move {
-                put_object(&client, &bucket, &key, body).await?;
-                Ok(true)
+                match put_object(&client, &bucket, &key, body).await {
+                    Ok(()) => Ok(true),
+                    Err(error)
+                        if matches!(
+                            s3_probe_error_code(error.as_ref()),
+                            Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
+                        ) =>
+                    {
+                        Ok(false)
+                    }
+                    Err(error) => Err(error),
+                }
            }
        },
        label,
@@ -129,6 +140,7 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
 }

 async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8], label: &str) -> TestResult {
+    let client = s3_probe_client(client);
    wait_until(
        CREDENTIAL_TIMEOUT,
        || {
@@ -137,8 +149,24 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
            let key = key.to_string();
            let expected = expected.to_vec();
            async move {
-                let got = get_object_bytes(&client, &bucket, &key).await?;
-                Ok(got == expected)
+                match get_object_bytes(&client, &bucket, &key).await {
+                    Ok(got) if got == expected => Ok(true),
+                    Ok(got) => Err(format!(
+                        "{label}: object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
+                        sha256_hex(&expected),
+                        sha256_hex(&got)
+                    )
+                    .into()),
+                    Err(error)
+                        if matches!(
+                            s3_probe_error_code(error.as_ref()),
+                            Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
+                        ) =>
+                    {
+                        Ok(false)
+                    }
+                    Err(error) => Err(error),
+                }
            }
        },
        label,
@@ -146,6 +174,57 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
    .await
 }

+#[cfg(test)]
+mod retry_tests {
+    use super::*;
+    use crate::common::build_test_s3_config;
+    use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation};
+
+    #[tokio::test]
+    async fn iam_polls_only_retry_auth_propagation_and_availability() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket("upgrade-retry-probe");
+        let client = Client::from_conf(build_test_s3_config(
+            target.endpoint(),
+            FAKE_ACCESS_KEY,
+            FAKE_SECRET_KEY,
+            None,
+            "upgrade-retry-test",
+        ));
+
+        for status in [403, 503] {
+            target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
+            wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT").await?;
+            assert_eq!(target.take_requests().len(), 2);
+
+            target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
+            wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET").await?;
+            assert_eq!(target.take_requests().len(), 2);
+        }
+
+        target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
+        let error = wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT")
+            .await
+            .expect_err("IAM convergence must not hide InternalError on PUT");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.take_requests().len(), 1);
+
+        target.inject(Operation::GetObject, FaultAction::ResponseStatus(500), 1);
+        let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET")
+            .await
+            .expect_err("IAM convergence must not hide InternalError on GET");
+        assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
+        assert_eq!(target.take_requests().len(), 1);
+
+        let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"wrong bytes", "IAM GET")
+            .await
+            .expect_err("IAM convergence must not hide a payload mismatch");
+        assert!(error.to_string().contains("bytes mismatch"));
+        assert_eq!(target.requests().len(), 1);
+        Ok(())
+    }
+}
+
 async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
    let history_bucket = unique_bucket("upg-hist");
    let versioned_bucket = unique_bucket("upg-ver");
@@ -215,6 +215,8 @@ mod tests {
        probe_startup_cas_binary(&binary, &nonce, &artifact).await?;

        let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
+        // Isolate bootstrap CAS and its S3 readback from periodic usage publication.
+        cluster.set_env("RUSTFS_SCANNER_ENABLED", "false");
        let mut logs = Vec::new();
        let mut disks = Vec::new();
        let mut endpoints = Vec::new();
@@ -0,0 +1,153 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging};
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+use tracing::info;
+
+const PARTS_COUNT_BUCKET: &str = "parts-count-test-bucket";
+const MULTIPART_KEY: &str = "multipart-object.bin";
+const SINGLE_PUT_KEY: &str = "single-put-object.bin";
+
+/// s3's minimum size for every multipart part but the last.
+const PART_SIZE: usize = 5 * 1024 * 1024;
+const LAST_PART_SIZE: usize = 1024;
+const EXPECTED_PARTS: i32 = 3;
+
+#[tokio::test]
+async fn get_object_reports_parts_count_for_multipart_upload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Starting GetObject x-amz-mp-parts-count regression test");
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(Vec::new()).await?;
+
+    let client = env.create_s3_client();
+    env.create_test_bucket(PARTS_COUNT_BUCKET).await?;
+
+    // A genuine multipart upload of three parts.
+    let created = client
+        .create_multipart_upload()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .send()
+        .await?;
+    let upload_id = created.upload_id().expect("upload id");
+
+    let mut completed_parts = Vec::new();
+    for part_number in 1..=EXPECTED_PARTS {
+        let size = if part_number == EXPECTED_PARTS {
+            LAST_PART_SIZE
+        } else {
+            PART_SIZE
+        };
+        let uploaded = client
+            .upload_part()
+            .bucket(PARTS_COUNT_BUCKET)
+            .key(MULTIPART_KEY)
+            .upload_id(upload_id)
+            .part_number(part_number)
+            .body(ByteStream::from(vec![b'a' + (part_number as u8); size]))
+            .send()
+            .await?;
+        completed_parts.push(
+            CompletedPart::builder()
+                .part_number(part_number)
+                .e_tag(uploaded.e_tag().expect("part etag"))
+                .build(),
+        );
+    }
+
+    client
+        .complete_multipart_upload()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .upload_id(upload_id)
+        .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed_parts)).build())
+        .send()
+        .await?;
+
+    let total_len = (PART_SIZE * 2 + LAST_PART_SIZE) as i64;
+
+    // The header the fix is about.
+    let first_part = client
+        .get_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .part_number(1)
+        .send()
+        .await?;
+    assert_eq!(
+        first_part.parts_count(),
+        Some(EXPECTED_PARTS),
+        "GetObject with partNumber must report the object's part count"
+    );
+    assert_eq!(
+        first_part.content_length(),
+        Some(PART_SIZE as i64),
+        "the response body must be part 1, not the whole object"
+    );
+    let expected_range = format!("bytes 0-{}/{}", PART_SIZE - 1, total_len);
+    assert_eq!(
+        first_part.content_range(),
+        Some(expected_range.as_str()),
+        "Content-Range must state part 1's extent and the whole object's size"
+    );
+
+    // Without partNumber s3 omits the header, so a plain GET is unchanged.
+    let whole = client
+        .get_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .send()
+        .await?;
+    assert_eq!(whole.parts_count(), None, "a GET that does not name a part is owed no part count");
+    assert_eq!(whole.content_length(), Some(total_len), "a plain GET returns the whole object");
+
+    // A single PutObject is not a multipart upload, so it is owed no count
+    // even when a part is named.
+    client
+        .put_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(SINGLE_PUT_KEY)
+        .body(ByteStream::from_static(b"0123456789abcdef"))
+        .send()
+        .await?;
+    let single = client
+        .get_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(SINGLE_PUT_KEY)
+        .part_number(1)
+        .send()
+        .await?;
+    assert_eq!(single.parts_count(), None, "an object stored by a single PutObject is not multipart");
+
+    client
+        .delete_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .send()
+        .await?;
+    client
+        .delete_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(SINGLE_PUT_KEY)
+        .send()
+        .await?;
+    env.delete_test_bucket(PARTS_COUNT_BUCKET).await?;
+    env.stop_server();
+
+    Ok(())
+}
@@ -1450,10 +1450,18 @@ mod tests {
            cluster.set_env("RUSTFS_HEAL_PAGE_PARALLEL_ENABLE", "false");
        }
        // Keep every node's Heal runtime enabled for normal disk registration.
-        let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG")
-            .unwrap_or_else(|_| "rustfs::heal::task=info,rustfs=error".to_string());
+        // Capture durable handoff and marker cleanup failures during shutdown.
+        let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG").unwrap_or_else(|_| {
+            "rustfs::heal::task=info,rustfs::heal::manager=info,rustfs::main::handle_shutdown=info,rustfs_heal::heal=warn,rustfs=error"
+                .to_string()
+        });
        cluster.set_env("RUST_LOG", server_rust_log);
-        let log_dir = std::env::var("RUSTFS_HEAL_CHAOS_LOG_DIR").unwrap_or_else(|_| format!("{}/logs", cluster.temp_dir));
+        let log_dir = std::env::var("RUSTFS_HEAL_CHAOS_LOG_DIR").unwrap_or_else(|_| {
+            // Keep explicit directories compatible and isolate parallel suite captures.
+            std::env::var("RUSTFS_HEAL_CHAOS_LOG_ROOT")
+                .map(|root| format!("{root}/{interruption_kind}-{}", uuid::Uuid::new_v4()))
+                .unwrap_or_else(|_| format!("{}/logs", cluster.temp_dir))
+        });
        std::fs::create_dir_all(&log_dir)?;
        for node_index in 0..cluster.nodes.len() {
            cluster.set_node_capture_log_path(node_index, format!("{log_dir}/node{node_index}.log"))?;
@@ -1799,7 +1807,8 @@ mod tests {
        let (mut client_token, mut task_status_url) = if background_rejoin_heal_evidence {
            (String::new(), String::new())
        } else {
-            let heal_start_body = signed_admin_post(&heal_url, Some(heal_body), &cluster.access_key, &cluster.secret_key).await?;
+            let heal_start_body =
+                start_root_heal_when_control_ready(&heal_url, heal_body, &cluster.access_key, &cluster.secret_key).await?;
            let heal_start: serde_json::Value = serde_json::from_str(&heal_start_body)
                .map_err(|err| format!("heal start response is not JSON ({err}): {heal_start_body}"))?;
            let client_token = heal_start["clientToken"]
@@ -2140,7 +2149,8 @@ mod tests {
                    && status["summary"].as_str() == Some("failed")
                {
                    let heal_start_body =
-                        signed_admin_post(&heal_url, Some(heal_body), &cluster.access_key, &cluster.secret_key).await?;
+                        start_root_heal_when_control_ready(&heal_url, heal_body, &cluster.access_key, &cluster.secret_key)
+                            .await?;
                    let heal_start: serde_json::Value = serde_json::from_str(&heal_start_body)
                        .map_err(|err| format!("recovery heal start response is not JSON ({err}): {heal_start_body}"))?;
                    client_token = heal_start["clientToken"]
@@ -886,6 +886,8 @@ fn configure_reader_metric_cluster(cluster: &mut RustFSTestClusterEnvironment, c
    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "true");
    cluster.set_env("RUSTFS_OBS_TRACES_EXPORT_ENABLED", "false");
    cluster.set_env("RUSTFS_OBS_LOGS_EXPORT_ENABLED", "false");
+    // Keep startup diagnostics available when OTLP log export is disabled.
+    cluster.set_env("RUSTFS_OBS_LOG_STDOUT_ENABLED", "true");
    cluster.set_env("RUSTFS_OBS_METER_INTERVAL", "1");
    cluster.set_env("RUSTFS_OBS_USE_STDOUT", "false");
    cluster.set_env("RUSTFS_GET_CODEC_STREAMING_ENABLE", "false");
@@ -232,6 +232,10 @@ mod tier_stats_cluster_test;
 #[cfg(test)]
 mod checksum_upload_test;

+// GetObject with partNumber reporting x-amz-mp-parts-count
+#[cfg(test)]
+mod get_object_parts_count_test;
+
 // Group deletion tests
 #[cfg(test)]
 mod group_delete_test;
@@ -272,6 +276,8 @@ mod storage_class_capability_test;
 // S3 dummy-compat bucket API tests
 #[cfg(test)]
 mod bucket_logging_test;
+#[cfg(test)]
+mod website_hosting_test;

 // Multipart control API auth regression tests
 #[cfg(test)]
@@ -311,4 +311,233 @@ mod tests {

        env.stop_server();
    }
+
+    async fn collect_prefix_pages(
+        client: &Client,
+        bucket: &str,
+        prefix: &str,
+        delimiter: Option<&str>,
+        max_keys: i32,
+        start_after: Option<&str>,
+    ) -> (Vec<aws_sdk_s3::types::Object>, Vec<String>) {
+        let mut objects = Vec::new();
+        let mut prefixes = Vec::new();
+        let mut token = None;
+        let mut seen_tokens = std::collections::HashSet::new();
+        for _ in 0..64 {
+            let page = client
+                .list_objects_v2()
+                .bucket(bucket)
+                .prefix(prefix)
+                .set_delimiter(delimiter.map(ToOwned::to_owned))
+                .max_keys(max_keys)
+                .set_start_after(start_after.map(ToOwned::to_owned))
+                .set_continuation_token(token)
+                .send()
+                .await
+                .expect("prefix pagination request should succeed");
+            let count = page.contents().len() + page.common_prefixes().len();
+            assert!(count <= usize::try_from(max_keys).expect("positive page size"));
+            assert_eq!(page.key_count(), Some(i32::try_from(count).expect("small page")));
+            objects.extend_from_slice(page.contents());
+            prefixes.extend(
+                page.common_prefixes()
+                    .iter()
+                    .map(|item| item.prefix().expect("CommonPrefix must contain a prefix").to_owned()),
+            );
+            if page.is_truncated() == Some(false) {
+                assert!(page.next_continuation_token().is_none(), "final page must not have a next token");
+                return (objects, prefixes);
+            }
+            assert_eq!(page.is_truncated(), Some(true));
+            let next = page.next_continuation_token().expect("truncated page requires a token");
+            assert!(!next.is_empty());
+            assert!(seen_tokens.insert(next.to_owned()), "continuation token must advance");
+            token = Some(next.to_owned());
+        }
+        panic!("prefix pagination exceeded its finite page budget");
+    }
+
+    fn object_keys(objects: &[aws_sdk_s3::types::Object]) -> Vec<&str> {
+        objects
+            .iter()
+            .map(|object| object.key().expect("listed object must have a key"))
+            .collect()
+    }
+
+    /// Issue #8175: the prefix object must occur once, including across page boundaries.
+    #[tokio::test]
+    async fn test_list_objects_v2_prefix_marker_pagination() {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
+        env.start_rustfs_server(vec![]).await.expect("start RustFS");
+        let client = create_s3_client(&env);
+        let bucket = "test-prefix-marker-pagination";
+        create_bucket(&client, bucket).await.expect("create bucket");
+        let mut expected = vec!["content/".to_owned()];
+        expected.extend((0..25).map(|index| format!("content/0123456789abcdef0123456789abcdef/{index:03}")));
+        for key in &expected {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(if key == "content/" { b"" } else { b"x" }))
+                .send()
+                .await
+                .expect("create prefix fixture");
+        }
+        for max_keys in [1000, 1, 2] {
+            let (objects, prefixes) = collect_prefix_pages(&client, bucket, "content/", None, max_keys, None).await;
+            assert_eq!(object_keys(&objects), expected, "all keys must appear once at MaxKeys={max_keys}");
+            assert!(prefixes.is_empty());
+        }
+        let (objects, prefixes) = collect_prefix_pages(&client, bucket, "content/", None, 1, Some("content/")).await;
+        assert_eq!(object_keys(&objects), expected[1..]);
+        assert!(prefixes.is_empty());
+
+        // V1 shares the storage listing path but resumes with a key marker.
+        let mut marker = Some("content/".to_owned());
+        let mut v1_keys = Vec::new();
+        let mut finished = false;
+        for _ in 0..32 {
+            let page = client
+                .list_objects()
+                .bucket(bucket)
+                .prefix("content/")
+                .max_keys(1)
+                .set_marker(marker.clone())
+                .send()
+                .await
+                .expect("list V1 continuation page");
+            assert!(page.contents().len() <= 1);
+            let keys = object_keys(page.contents());
+            v1_keys.extend(keys.iter().map(|key| (*key).to_owned()));
+            if page.is_truncated() == Some(false) {
+                finished = true;
+                break;
+            }
+            assert_eq!(page.is_truncated(), Some(true));
+            let next = page
+                .next_marker()
+                .or_else(|| keys.last().copied())
+                .expect("V1 page must advance");
+            assert!(marker.as_deref().is_none_or(|previous| next > previous));
+            marker = Some(next.to_owned());
+        }
+        assert!(finished, "V1 pagination must terminate");
+        assert_eq!(v1_keys, expected[1..]);
+        env.stop_server();
+    }
+
+    /// An exact prefix match does not establish EOF, including for ordinary keys.
+    #[tokio::test]
+    async fn test_list_objects_v2_exact_prefix_pagination_boundaries() {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
+        env.start_rustfs_server(vec![]).await.expect("start RustFS");
+        let client = create_s3_client(&env);
+        let bucket = "test-exact-prefix-boundaries";
+        create_bucket(&client, bucket).await.expect("create bucket");
+        for key in [
+            "a",
+            "ab",
+            "solo/",
+            "marker/",
+            "marker/file",
+            "marker/subdir/",
+            "marker/subdir/file",
+        ] {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(b""))
+                .send()
+                .await
+                .expect("create boundary fixture");
+        }
+        let (objects, prefixes) = collect_prefix_pages(&client, bucket, "a", None, 1, None).await;
+        assert_eq!(object_keys(&objects), vec!["a", "ab"]);
+        assert!(prefixes.is_empty());
+        let first = client
+            .list_objects()
+            .bucket(bucket)
+            .prefix("a")
+            .max_keys(1)
+            .send()
+            .await
+            .expect("list V1 exact-prefix first page");
+        assert_eq!(object_keys(first.contents()), vec!["a"]);
+        assert_eq!(first.is_truncated(), Some(true));
+        let last = client
+            .list_objects()
+            .bucket(bucket)
+            .prefix("a")
+            .max_keys(1)
+            .marker(first.next_marker().unwrap_or("a"))
+            .send()
+            .await
+            .expect("list V1 exact-prefix final page");
+        assert_eq!(object_keys(last.contents()), vec!["ab"]);
+        assert_eq!(last.is_truncated(), Some(false));
+
+        let solo = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .prefix("solo/")
+            .max_keys(1)
+            .send()
+            .await
+            .expect("list isolated marker");
+        assert_eq!(object_keys(solo.contents()), vec!["solo/"]);
+        assert_eq!(solo.is_truncated(), Some(false));
+        assert!(solo.next_continuation_token().is_none());
+        let (objects, prefixes) = collect_prefix_pages(&client, bucket, "marker/", Some("/"), 1, None).await;
+        assert_eq!(object_keys(&objects), vec!["marker/", "marker/file"]);
+        assert_eq!(prefixes, vec!["marker/subdir/"]);
+        env.stop_server();
+    }
+
+    /// A directory marker must retain its own metadata when a same-named object exists.
+    #[tokio::test]
+    async fn test_list_objects_v2_prefix_marker_preserves_metadata() {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
+        env.start_rustfs_server(vec![]).await.expect("start RustFS");
+        let client = create_s3_client(&env);
+        let bucket = "test-prefix-marker-metadata";
+        create_bucket(&client, bucket).await.expect("create bucket");
+        let plain = client
+            .put_object()
+            .bucket(bucket)
+            .key("content")
+            .body(ByteStream::from_static(b"plain object body"))
+            .send()
+            .await
+            .expect("put plain object");
+        let directory = client
+            .put_object()
+            .bucket(bucket)
+            .key("content/")
+            .body(ByteStream::from_static(b""))
+            .send()
+            .await
+            .expect("put directory marker");
+        assert_ne!(plain.e_tag(), directory.e_tag(), "fixture ETags must distinguish the objects");
+        for max_keys in [1, 1000] {
+            let (objects, prefixes) = collect_prefix_pages(&client, bucket, "content/", None, max_keys, None).await;
+            assert_eq!(object_keys(&objects), vec!["content/"]);
+            assert!(prefixes.is_empty());
+            assert_eq!(objects[0].size(), Some(0));
+            assert_eq!(objects[0].e_tag(), directory.e_tag());
+            let (objects, prefixes) = collect_prefix_pages(&client, bucket, "content", None, max_keys, None).await;
+            assert_eq!(object_keys(&objects), vec!["content", "content/"]);
+            assert!(prefixes.is_empty());
+            assert_eq!(objects[0].size(), Some(17));
+            assert_eq!(objects[0].e_tag(), plain.e_tag());
+            assert_eq!(objects[1].size(), Some(0));
+            assert_eq!(objects[1].e_tag(), directory.e_tag());
+        }
+        env.stop_server();
+    }
 }
@@ -31,8 +31,19 @@ mod tests {
    use aws_sdk_s3::Client;
    use aws_sdk_s3::primitives::ByteStream;
    use std::collections::HashSet;
+    use std::future::Future;
+    use std::time::Duration;
+    use tokio::time::{Instant, error::Elapsed, timeout_at};
    use tracing::info;

+    const PAGINATION_TIMEOUT: Duration = Duration::from_secs(120);
+
+    // Fixture population can require thousands of durable PUTs. Bound the
+    // listing traversal itself with one deadline shared by every page.
+    async fn listing_request_before<T>(deadline: Instant, request: impl Future<Output = T>) -> Result<T, Elapsed> {
+        timeout_at(deadline, request).await
+    }
+
    /// Helper function to create an S3 client for testing
    fn create_s3_client(env: &RustFSTestEnvironment) -> Client {
        env.create_s3_client()
@@ -57,6 +68,54 @@ mod tests {
        }
    }

+    #[tokio::test]
+    async fn test_list_objects_v2_deadline_rejects_unresponsive_peer() {
+        use tokio::io::AsyncReadExt;
+        use tokio::net::TcpListener;
+        use tokio::sync::oneshot;
+        use tokio::time::timeout;
+
+        let listener = TcpListener::bind("127.0.0.1:0")
+            .await
+            .expect("bind unresponsive listing peer");
+        let endpoint = format!("http://{}", listener.local_addr().expect("listing peer address"));
+        let (received_tx, received_rx) = oneshot::channel();
+        let server = tokio::spawn(async move {
+            let (mut stream, _) = listener.accept().await.expect("accept listing request");
+            let mut bytes = [0; 4096];
+            assert!(stream.read(&mut bytes).await.expect("read listing request") > 0);
+            received_tx.send(()).expect("signal received listing request");
+            std::future::pending::<()>().await;
+            drop(stream);
+        });
+        let client = Client::from_conf(
+            crate::common::build_test_s3_config(&endpoint, "test-access", "test-secret", None, "pagination-deadline")
+                .to_builder()
+                .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+                .build(),
+        );
+        let request = client.list_objects_v2().bucket("deadline-test").send();
+        tokio::pin!(request);
+        tokio::select! {
+            response = &mut request => panic!("unresponsive listing peer unexpectedly returned: {response:?}"),
+            received = timeout(Duration::from_secs(10), received_rx) => {
+                received.expect("listing request must reach peer").expect("listing peer must remain alive");
+            }
+        }
+
+        // Expire the same absolute deadline after the real request is observed;
+        // this tests a stalled response without relying on transport timing.
+        let result = timeout(Duration::from_secs(1), listing_request_before(Instant::now(), request))
+            .await
+            .expect("expired pagination deadline must stop the in-flight request");
+        server.abort();
+        let _ = server.await;
+        assert!(
+            result.is_err(),
+            "an unresponsive ListObjectsV2 request must exceed the pagination deadline"
+        );
+    }
+
    /// Test for Issue #2775: continuation forwarding must not
    /// skip a child directory when the prefix component repeats in the key.
    #[tokio::test]
@@ -514,12 +573,12 @@ mod tests {
                .expect("Failed to put object");
        }

-        let output = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .max_keys(1001)
-            .send()
+        eprintln!("Seeded {object_count} objects in {bucket}; starting ListObjectsV2 pagination");
+        let deadline = Instant::now() + PAGINATION_TIMEOUT;
+
+        let output = listing_request_before(deadline, client.list_objects_v2().bucket(bucket).max_keys(1001).send())
            .await
+            .expect("ListObjectsV2 pagination exceeded its deadline after fixture population")
            .expect("Failed to list objects");

        assert_eq!(output.contents().len(), 1000);
@@ -535,14 +594,18 @@ mod tests {
            .expect("NextContinuationToken should be present when capped response is truncated")
            .to_string();

-        let output = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .max_keys(1001)
-            .continuation_token(next_token)
-            .send()
-            .await
-            .expect("Failed to list objects with continuation token");
+        let output = listing_request_before(
+            deadline,
+            client
+                .list_objects_v2()
+                .bucket(bucket)
+                .max_keys(1001)
+                .continuation_token(next_token)
+                .send(),
+        )
+        .await
+        .expect("ListObjectsV2 continuation exceeded the shared pagination deadline")
+        .expect("Failed to list objects with continuation token");

        assert_eq!(output.contents().len(), 2);
        assert!(!output.is_truncated().unwrap_or(false));
@@ -765,6 +828,9 @@ mod tests {
            }
        }

+        eprintln!("Seeded {} objects in {bucket}; starting ListObjectsV2 pagination", all_keys.len());
+        let deadline = Instant::now() + PAGINATION_TIMEOUT;
+
        // Paginate with delimiter="/" and max_keys=50
        // Visible per page: up to 50 CommonPrefixes
        let mut listed_keys = Vec::new();
@@ -780,7 +846,10 @@ mod tests {
                request = request.continuation_token(token);
            }

-            let output = request.send().await.expect("Failed to list objects");
+            let output = listing_request_before(deadline, request.send())
+                .await
+                .expect("ListObjectsV2 delimiter traversal exceeded the shared pagination deadline")
+                .expect("Failed to list objects");
            last_page_is_truncated = output.is_truncated().unwrap_or(false);

            for obj in output.contents() {
@@ -960,6 +1029,8 @@ mod tests {
    /// tests the non-delimiter case.
    #[tokio::test]
    async fn test_list_objects_v2_maxkeys_above_limit_with_delimiter() {
+        use futures::{StreamExt, stream};
+
        init_logging();
        info!("Starting test: ListObjectsV2 MaxKeys above limit with delimiter");

@@ -974,29 +1045,44 @@ mod tests {
        // 12 dirs × 100 files = 1200 raw keys
        let dir_count = 12;
        let files_per_dir = 100;
-        for d in 0..dir_count {
-            for f in 0..files_per_dir {
-                let key = format!("dir-{:02}/file{:03}.txt", d, f);
-                client
-                    .put_object()
-                    .bucket(bucket)
-                    .key(&key)
-                    .body(ByteStream::from_static(b"x"))
-                    .send()
-                    .await
-                    .expect("Failed to put object");
-            }
-        }
+        const FIXTURE_CONCURRENCY: usize = 8;
+        let fixture_started = std::time::Instant::now();
+        eprintln!(
+            "Preparing {} objects with at most {FIXTURE_CONCURRENCY} concurrent PUTs",
+            dir_count * files_per_dir
+        );
+        // Object creation order is irrelevant to this listing contract. Bound
+        // fixture I/O and await every PUT before issuing the LIST request.
+        stream::iter((0..dir_count).flat_map(|d| (0..files_per_dir).map(move |f| (d, f))))
+            .for_each_concurrent(FIXTURE_CONCURRENCY, |(d, f)| {
+                let client = &client;
+                async move {
+                    let key = format!("dir-{d:02}/file{f:03}.txt");
+                    client
+                        .put_object()
+                        .bucket(bucket)
+                        .key(&key)
+                        .body(ByteStream::from_static(b"x"))
+                        .send()
+                        .await
+                        .unwrap_or_else(|err| panic!("Failed to prepare fixture object {key}: {err:?}"));
+                }
+            })
+            .await;
+        eprintln!(
+            "Seeded {} objects in {bucket} in {:?}; starting ListObjectsV2 pagination",
+            dir_count * files_per_dir,
+            fixture_started.elapsed()
+        );
+        let deadline = Instant::now() + PAGINATION_TIMEOUT;

        // With delimiter: 12 CommonPrefixes visible, all fit within capped 1000
-        let output = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .delimiter("/")
-            .max_keys(2000)
-            .send()
-            .await
-            .expect("Failed to list objects");
+        let list_started = std::time::Instant::now();
+        let output =
+            listing_request_before(deadline, client.list_objects_v2().bucket(bucket).delimiter("/").max_keys(2000).send())
+                .await
+                .expect("ListObjectsV2 delimiter listing exceeded its deadline after fixture population")
+                .expect("Failed to list objects");

        assert_eq!(
            output.common_prefixes().len(),
@@ -1005,6 +1091,15 @@ mod tests {
            dir_count,
            output.common_prefixes().len()
        );
+        let prefixes = output
+            .common_prefixes()
+            .iter()
+            .map(|entry| entry.prefix().expect("each CommonPrefix must carry its prefix").to_owned())
+            .collect::<Vec<_>>();
+        let expected_prefixes = (0..dir_count).map(|d| format!("dir-{d:02}/")).collect::<Vec<_>>();
+        assert_eq!(prefixes, expected_prefixes);
+        assert!(output.contents().is_empty(), "all fixture objects must collapse into prefixes");
+        assert_eq!(output.key_count(), Some(i32::try_from(dir_count).expect("fixture count fits i32")));
        assert_eq!(output.max_keys(), Some(1000));
        // 12 visible < 1000 capped MaxKeys → not truncated
        assert!(
@@ -1013,6 +1108,11 @@ mod tests {
            output.common_prefixes().len()
        );

+        assert!(
+            output.next_continuation_token().is_none(),
+            "a complete page must not carry a continuation token"
+        );
+        eprintln!("ListObjectsV2 assertions completed in {:?}", list_started.elapsed());
        info!("MaxKeys above limit with delimiter test passed");

        env.stop_server();
@@ -462,7 +462,10 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
        }
        let write_ready = survivors >= 3;
        let read_quorum = survivors >= 2;
-        let expected_status = if write_ready { 200 } else { 503 };
+        // `/health/ready` keeps a node that can still serve reads in the Service.
+        let node_ready = read_quorum;
+        let node_status = if node_ready { 200 } else { 503 };
+        let cluster_status = if write_ready { 200 } else { 503 };
        for (idx, client) in clients.iter().enumerate().take(survivors) {
            let url = &cluster.nodes[idx].url;
            let deadline = Instant::now() + Duration::from_secs(30);
@@ -472,27 +475,27 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
                let response = http.get(format!("{url}/health/ready")).send().await?;
                let status = response.status().as_u16();
                let payload: serde_json::Value = response.json().await?;
-                if status == expected_status
-                    && payload["ready"] == write_ready
-                    && payload["details"]["storage"]["ready"] == write_ready
+                if status == node_status
+                    && payload["ready"] == node_ready
+                    && payload["details"]["storage"]["ready"] == node_ready
                    && payload["details"]["storage"]["readQuorum"] == read_quorum
                    && payload["details"]["storage"]["writeQuorum"] == write_ready
                    && payload["details"]["poolMetadata"]["ready"] == true
                    && payload["details"]["iam"]["ready"] == true
-                    && payload["details"]["lock"]["ready"] == write_ready
+                    && payload["details"]["lock"]["ready"] == read_quorum
                {
                    break payload;
                }
                assert!(Instant::now() < deadline, "node {idx}, survivors={survivors}: HTTP {status}, {payload}");
                tokio::time::sleep(Duration::from_millis(200)).await;
            };
-            assert_eq!(payload["details"]["storage"]["readinessScope"], "write_quorum_and_pool_metadata");
+            assert_eq!(payload["details"]["storage"]["readinessScope"], "read_quorum");
            assert_eq!(payload["details"]["storage"]["source"], "local_runtime");
            assert_eq!(
                payload["details"]["storage"]["status"],
-                if write_ready { "connected" } else { "disconnected" }
+                if node_ready { "connected" } else { "disconnected" }
            );
-            if !write_ready {
+            if !node_ready {
                assert!(
                    payload["degradedReasons"]
                        .as_array()
@@ -504,10 +507,10 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi

            for path in ["/health/ready", "/minio/health/ready"] {
                let head = http.head(format!("{url}{path}")).send().await?;
-                assert_eq!(head.status().as_u16(), expected_status, "HEAD {path}, survivors={survivors}");
+                assert_eq!(head.status().as_u16(), node_status, "HEAD {path}, survivors={survivors}");
                assert!(head.bytes().await?.is_empty());
                let response = http.get(format!("{url}{path}")).send().await?;
-                assert_eq!(response.status().as_u16(), expected_status);
+                assert_eq!(response.status().as_u16(), node_status);
                let body: serde_json::Value = response.json().await?;
                assert_eq!(body["details"]["storage"], payload["details"]["storage"]);
                assert_eq!(body["details"]["poolMetadata"], payload["details"]["poolMetadata"]);
@@ -526,7 +529,7 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
                    let response = http.get(format!("{url}{path}")).send().await?;
                    let status = response.status().as_u16();
                    let body: serde_json::Value = response.json().await?;
-                    if status == expected_status
+                    if status == cluster_status
                        && body["details"]["storage"]["ready"] == storage_ready
                        && body["details"]["lock"]["ready"] == write_ready
                    {
@@ -559,11 +562,12 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
            let get = client.get_object().bucket(BUCKET).key(seed_key).send().await;
            let get_status = match get {
                Ok(object) => {
+                    assert!(read_quorum, "GET must fail once read quorum is lost");
                    assert_eq!(object.body.collect().await?.into_bytes().as_ref(), seed_body);
                    200
                }
                Err(error) => {
-                    assert!(!write_ready, "GET must succeed on a ready cluster: {error:?}");
+                    assert!(!read_quorum, "GET must succeed while read quorum holds: {error:?}");
                    let status = error
                        .raw_response()
                        .expect("GET should have an HTTP response")
@@ -576,11 +580,12 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
            let list = client.list_objects_v2().bucket(BUCKET).send().await;
            let list_status = match list {
                Ok(result) => {
+                    assert!(read_quorum, "listing must fail once read quorum is lost");
                    assert!(result.contents().iter().any(|object| object.key() == Some(seed_key)));
                    200
                }
                Err(error) => {
-                    assert!(!write_ready, "listing must succeed on a ready cluster: {error:?}");
+                    assert!(!read_quorum, "listing must succeed while read quorum holds: {error:?}");
                    let status = error
                        .raw_response()
                        .expect("LIST should have an HTTP response")
@@ -1056,7 +1056,17 @@ async fn test_hermetic_transition_restore_failure_expiry_and_retry() -> TestResu
    cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;

    let mut hot = RustFSTestEnvironment::new().await?;
-    start_tier_source(&mut hot, &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_DEBUG_DAY_SECS", "5")]).await?;
+    // Revisit the restored key every cycle; the default 16-cycle sampling is
+    // independent of the accelerated lifecycle clock used by this fixture.
+    start_tier_source(
+        &mut hot,
+        &[
+            ("RUSTFS_SCANNER_CYCLE", "1"),
+            ("RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES", "1"),
+            ("RUSTFS_ILM_DEBUG_DAY_SECS", "5"),
+        ],
+    )
+    .await?;
    let hot_client = hot.create_s3_client();
    add_rustfs_tier(&hot, &cold).await?;

@@ -7194,13 +7194,30 @@ async fn test_site_replication_state_edit_fresh_and_stale_real_dual_node() -> Re
    assert!(source_info.sites.iter().all(|peer| !peer.replicate_ilm_expiry));
    assert!(target_info.sites.iter().all(|peer| !peer.replicate_ilm_expiry));

+    let source_deployment_id = source_info
+        .sites
+        .iter()
+        .find(|peer| peer.endpoint == source_env.url)
+        .map(|peer| peer.deployment_id.clone())
+        .ok_or("source site missing from its own replication info")?;
+    let target_deployment_id = target_info
+        .sites
+        .iter()
+        .find(|peer| peer.endpoint == target_env.url)
+        .map(|peer| peer.deployment_id.clone())
+        .ok_or("target site missing from its own replication info")?;
    let target_status =
        wait_for_site_replication_status(&target_env, "peer-state=true", |status| status.peer_states.len() == 2).await?;
    let current_updated_at = target_status
        .peer_states
-        .values()
-        .find_map(|state| state.updated_at)
+        .get(&target_deployment_id)
+        .and_then(|state| state.updated_at)
        .ok_or("missing target site replication updated_at")?;
+    let source_updated_at = target_status
+        .peer_states
+        .get(&source_deployment_id)
+        .and_then(|state| state.updated_at)
+        .ok_or("missing source site replication updated_at")?;

    let mut stale_peers = BTreeMap::new();
    for peer in target_info.sites {
@@ -7244,16 +7261,23 @@ async fn test_site_replication_state_edit_fresh_and_stale_real_dual_node() -> Re
    .await?;
    assert!(target_after_fresh.sites.iter().all(|peer| peer.replicate_ilm_expiry));

+    // State edits apply only to the addressed site; status reports each site's own state.
    let target_status_after_fresh = wait_for_site_replication_status(&target_env, "peer-state=true", |status| {
        status.peer_states.len() == 2
-            && status.peer_states.values().all(|state| {
-                state.updated_at == Some(fresh_updated_at) && state.peers.values().all(|peer| peer.replicate_ilm_expiry)
+            && status.peer_states.get(&target_deployment_id).is_some_and(|state| {
+                state.updated_at == Some(fresh_updated_at)
+                    && state.peers.len() == 2
+                    && state.peers.values().all(|peer| peer.replicate_ilm_expiry)
            })
    })
    .await?;
-    assert!(target_status_after_fresh.peer_states.values().all(|state| {
-        state.updated_at == Some(fresh_updated_at) && state.peers.values().all(|peer| peer.replicate_ilm_expiry)
-    }));
+    let source_state_after_fresh = target_status_after_fresh
+        .peer_states
+        .get(&source_deployment_id)
+        .ok_or("missing source site replication state after target edit")?;
+    assert_eq!(source_state_after_fresh.updated_at, Some(source_updated_at));
+    assert_eq!(source_state_after_fresh.peers.len(), 2);
+    assert!(source_state_after_fresh.peers.values().all(|peer| !peer.replicate_ilm_expiry));

    let source_after_fresh = site_replication_info(&source_env).await?;
    assert!(source_after_fresh.sites.iter().all(|peer| !peer.replicate_ilm_expiry));
@@ -51,6 +51,10 @@ const PLAIN_BUCKET: &str = "upgrade-plain-data";
 const VERSIONED_BUCKET: &str = "upgrade-versioned-data";
 const MIXED_BUCKET: &str = "upgrade-mixed-version-data";
 const MIXED_NODE_COUNT: usize = 4;
+const UPGRADE_READINESS_BODY: &[u8] = b"upgrade write readiness";
+// Written by the "previous-seed" readiness probe through node 0 before any
+// current binary starts; every later phase must still read it.
+const PREVIOUS_RELEASE_SEED_KEY: &str = ".upgrade-readiness/previous-seed/node-0";
 const MULTIPART_WORKERS: usize = 16;
 const MULTIPART_UPLOADS_PER_WORKER: usize = 16;
 // Peers keep a restarted node's drive in Suspect/Returning for roughly
@@ -67,6 +71,7 @@ const CONFIG_REPLICA_BUCKET: &str = "upgrade-config-replica";
 const ROLLBACK_BUCKET: &str = "rollback-config-data";
 const ROLLBACK_REPLICA_BUCKET: &str = "rollback-config-replica";
 const BUCKET_QUOTA_BYTES: u64 = 64 * 1024 * 1024;
+const QUOTA_USAGE_READINESS_TIMEOUT: Duration = Duration::from_secs(90);
 const LIFECYCLE_RULE_ID: &str = "upgrade-expire-logs";
 const LIFECYCLE_PREFIX: &str = "logs/";
 const LIFECYCLE_DAYS: i32 = 30;
@@ -395,6 +400,7 @@ async fn exercise_mixed_cluster(
    previous_node: usize,
 ) -> TestResult {
    let clients = cluster.create_all_clients()?;
+    wait_for_upgrade_write_readiness(&clients, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
    let current_client = &clients[current_node];
    let previous_client = &clients[previous_node];

@@ -423,6 +429,11 @@ async fn exercise_mixed_cluster(
    let multipart_keys = write_multipart_load(&clients, phase).await?;
    let expected_count = multipart_keys.len() + 2;
    for (label, client) in [("current", current_client), ("previous", previous_client)] {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "{phase}: the {label} node must retain the previous-release seed"
+        );
        wait_for_phase_listing(
            client,
            phase,
@@ -441,6 +452,231 @@ async fn exercise_mixed_cluster(
    Ok(())
 }

+fn upgrade_probe_client(client: &Client) -> Client {
+    Client::from_conf(
+        client
+            .config()
+            .to_builder()
+            .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
+            .build(),
+    )
+}
+
+async fn wait_for_upgrade_write_readiness(clients: &[Client], phase: &str, budget: Duration) -> TestResult {
+    // ListBuckets can succeed before peers recover a restarted disk. Cluster
+    // health also accepts Returning disks whose write health is still FAULTY.
+    // Probe every writer outside the asserted phase prefix; compatibility
+    // writes still execute once and retain their original assertions.
+    let deadline = Instant::now() + budget;
+    for (node, client) in clients.iter().enumerate() {
+        let client = upgrade_probe_client(client);
+        let key = format!(".upgrade-readiness/{phase}/node-{node}");
+        let mut last_response = "no response".to_string();
+        loop {
+            if Instant::now() >= deadline {
+                return Err(
+                    format!("{phase}: node {node} write readiness deadline exceeded; last response: {last_response}").into(),
+                );
+            }
+            let response = tokio::time::timeout_at(
+                deadline,
+                client
+                    .put_object()
+                    .bucket(MIXED_BUCKET)
+                    .key(&key)
+                    .body(ByteStream::from_static(UPGRADE_READINESS_BODY))
+                    .send(),
+            )
+            .await;
+            match response {
+                Ok(Ok(_)) => break,
+                Ok(Err(error)) => {
+                    if error.raw_response().map(|response| response.status().as_u16()) != Some(503)
+                        || error.as_service_error().and_then(ProvideErrorMetadata::code) != Some("ServiceUnavailable")
+                    {
+                        return Err(format!("{phase}: node {node} write readiness failed: {error:?}").into());
+                    }
+                    last_response = format!("{error:?}");
+                }
+                Err(_) => {
+                    return Err(format!(
+                        "{phase}: node {node} write readiness deadline exceeded during PutObject; last response: {last_response}"
+                    )
+                    .into());
+                }
+            }
+            tokio::time::sleep_until(deadline.min(Instant::now() + Duration::from_millis(500))).await;
+        }
+    }
+    Ok(())
+}
+
+async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnvironment, previous_binary: &Path) -> TestResult {
+    let clients: Vec<_> = cluster.create_all_clients()?.iter().map(upgrade_probe_client).collect();
+    // rc.5 can latch a non-elected node's write fence while its first startup
+    // waits for pool metadata (#7473). First prove the elected writer can
+    // persist data, then restart each old process once with three peers still
+    // readable. This preparation ends before any current binary is started.
+    wait_for_upgrade_write_readiness(&clients[..1], "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
+    for node in [1, 2, 3, 0] {
+        cluster.stop_node(node)?;
+        cluster.start_node_from_binary(node, previous_binary).await?;
+        wait_for_upgrade_write_readiness(
+            std::slice::from_ref(&clients[node]),
+            &format!("previous-restart-{node}"),
+            LISTING_CONVERGENCE_TIMEOUT,
+        )
+        .await?;
+    }
+
+    wait_for_upgrade_write_readiness(&clients, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
+    for (reader, client) in clients.iter().enumerate() {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "previous-release node {reader} must retain the seed across its rolling restart"
+        );
+        for writer in 0..clients.len() {
+            let key = format!(".upgrade-readiness/previous-baseline/node-{writer}");
+            assert_eq!(
+                read_object(client, MIXED_BUCKET, &key, None).await?.1,
+                UPGRADE_READINESS_BODY,
+                "previous-release node {reader} must read baseline data from writer {writer}"
+            );
+        }
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod upgrade_write_readiness_tests {
+    use super::*;
+    use crate::fake_s3_target::FaultAction;
+
+    #[tokio::test]
+    async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket(MIXED_BUCKET);
+        let client = fake_source_client(&target);
+        client.head_bucket().bucket(MIXED_BUCKET).send().await?;
+
+        let phase = "one-previous-node";
+        let first_key = format!(".upgrade-readiness/{phase}/node-0");
+        let second_key = format!(".upgrade-readiness/{phase}/node-1");
+        target.inject_for_key(
+            FakeTargetOperation::PutObject,
+            &first_key,
+            FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
+            2,
+        );
+        target.inject_for_key(
+            FakeTargetOperation::PutObject,
+            &second_key,
+            FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
+            1,
+        );
+        // Metadata readiness does not prove that a data write can succeed.
+        let premature = client
+            .put_object()
+            .bucket(MIXED_BUCKET)
+            .key(&first_key)
+            .body(ByteStream::from_static(b"upgrade write readiness"))
+            .send()
+            .await
+            .expect_err("metadata readiness does not prove write readiness");
+        assert_eq!(premature.raw_response().map(|response| response.status().as_u16()), Some(503));
+
+        wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], phase, Duration::from_secs(5)).await?;
+        assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &first_key), 3);
+        assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &second_key), 2);
+        assert!(target.has_object(MIXED_BUCKET, &first_key));
+        assert!(target.has_object(MIXED_BUCKET, &second_key));
+        assert!(
+            client
+                .list_objects_v2()
+                .bucket(MIXED_BUCKET)
+                .prefix(format!("{phase}/"))
+                .send()
+                .await?
+                .contents()
+                .is_empty()
+        );
+        target.shutdown().await;
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn rejects_permanent_errors_without_sdk_retries() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket(MIXED_BUCKET);
+        let client = Client::from_conf(
+            fake_source_client(&target)
+                .config()
+                .to_builder()
+                .retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(3))
+                .build(),
+        );
+        for status in [
+            StatusCode::INTERNAL_SERVER_ERROR,
+            StatusCode::FORBIDDEN,
+            StatusCode::NOT_FOUND,
+        ] {
+            let phase = format!("permanent-{}", status.as_u16());
+            let key = format!(".upgrade-readiness/{phase}/node-0");
+            target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Status(status), 1);
+            let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), &phase, Duration::from_secs(5))
+                .await
+                .expect_err("a permanent error must not be retried into success");
+            assert!(error.to_string().contains("node 0 write readiness failed"), "{error}");
+            assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &key), 1);
+            assert!(!target.has_object(MIXED_BUCKET, &key));
+        }
+        target.shutdown().await;
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn transient_errors_stop_at_the_deadline() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket(MIXED_BUCKET);
+        let client = fake_source_client(&target);
+        client.head_bucket().bucket(MIXED_BUCKET).send().await?;
+        let phase = "deadline";
+        let key = format!(".upgrade-readiness/{phase}/node-0");
+        target.inject_for_key(
+            FakeTargetOperation::PutObject,
+            &key,
+            FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
+            10,
+        );
+        let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
+            .await
+            .expect_err("persistent unavailability must exhaust the shared deadline");
+        // Transport scheduling consumes the same budget; do not require a
+        // response to reach the fake target before the deadline on a busy host.
+        assert!(error.to_string().contains("deadline exceeded"), "{error}");
+        target.shutdown().await;
+        Ok(())
+    }
+
+    #[tokio::test]
+    async fn in_flight_requests_are_bounded_by_the_deadline() -> TestResult {
+        let target = FakeS3Target::start().await?;
+        target.create_bucket(MIXED_BUCKET);
+        let client = fake_source_client(&target);
+        client.head_bucket().bucket(MIXED_BUCKET).send().await?;
+        let phase = "stalled";
+        let key = format!(".upgrade-readiness/{phase}/node-0");
+        target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Stall(Duration::from_secs(30)), 1);
+        let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
+            .await
+            .expect_err("a stalled request must not outlive the readiness deadline");
+        assert!(error.to_string().contains("deadline exceeded during PutObject"), "{error}");
+        target.shutdown().await;
+        Ok(())
+    }
+}
+
 /// Pins the published old writer's limitation and the supported recovery
 /// procedure. This is not a promise that mixed-version ODM is supported.
 /// Replace the loss assertion when ODM gains independent persistence;
@@ -690,6 +926,7 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
    configure_cluster_logs(&mut cluster)?;
    cluster.start_with_binary(&previous_binary).await?;
    cluster.create_test_bucket(MIXED_BUCKET).await?;
+    prepare_previous_release_baseline(&mut cluster, &previous_binary).await?;

    cluster.stop_node(0)?;
    cluster.start_node_from_binary(0, &current_binary).await?;
@@ -705,6 +942,11 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
    cluster.start_node_from_binary(3, &current_binary).await?;

    for (node_idx, client) in cluster.create_all_clients()?.iter().enumerate() {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "current node {node_idx} must retain the previous-release seed"
+        );
        for phase in ["one-current-node", "one-previous-node"] {
            wait_for_phase_listing(
                client,
@@ -732,7 +974,8 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
                "assertions": [
                    "current node reads objects written through previous-release client",
                    "previous-release node reads objects written through current client",
-                    "all nodes list every mixed-version object after homogeneous-current convergence"
+                    "all nodes list every mixed-version object after homogeneous-current convergence",
+                    "the previous-release seed survives the baseline restart and every upgrade phase"
                ],
            }),
        )?;
@@ -837,12 +1080,23 @@ async fn put_object_through_quota_warmup(client: &Client, bucket: &str, key: &st

 async fn get_bucket_quota(env: &RustFSTestEnvironment, bucket: &str) -> Result<Option<u64>, BoxError> {
    let path = format!("/rustfs/admin/v3/quota/{bucket}");
-    let (status, response) = admin_request(&env.url, Method::GET, &path, None, &env.access_key, &env.secret_key).await?;
-    if status != StatusCode::OK {
-        return Err(format!("reading the quota of {bucket} failed: {status} {response}").into());
+    let deadline = Instant::now() + QUOTA_USAGE_READINESS_TIMEOUT;
+    loop {
+        let (status, response) = tokio::time::timeout_at(
+            deadline,
+            admin_request(&env.url, Method::GET, &path, None, &env.access_key, &env.secret_key),
+        )
+        .await
+        .map_err(|_| format!("reading the quota of {bucket} exceeded the usage-readiness timeout"))??;
+        if status == StatusCode::OK {
+            let quota: serde_json::Value = serde_json::from_str(&response)?;
+            return Ok(quota.get("quota").and_then(serde_json::Value::as_u64));
+        }
+        if status != StatusCode::SERVICE_UNAVAILABLE || Instant::now() >= deadline {
+            return Err(format!("reading the quota of {bucket} failed: {status} {response}").into());
+        }
+        sleep(Duration::from_millis(500)).await;
    }
-    let quota: serde_json::Value = serde_json::from_str(&response)?;
-    Ok(quota.get("quota").and_then(serde_json::Value::as_u64))
 }

 /// `GET /rustfs/admin/v3/list-remote-targets?bucket=...`.
@@ -0,0 +1,271 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging};
+use aws_sdk_s3::presigning::PresigningConfig;
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{
+    BucketVersioningStatus, Condition, ErrorDocument, IndexDocument, Redirect, RedirectAllRequestsTo, RoutingRule,
+    VersioningConfiguration, WebsiteConfiguration,
+};
+use reqwest::header::HOST;
+use std::time::Duration;
+
+type TestResult<T = ()> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
+
+#[tokio::test]
+async fn website_host_serves_pages_without_changing_s3_reads() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_WEBSITE_DOMAINS", "website.test")])
+        .await?;
+    let s3 = env.create_s3_client();
+    let bucket = "website-hosting";
+    s3.create_bucket().bucket(bucket).send().await?;
+    for (key, body) in [
+        ("index.html", "home"),
+        ("404.html", "error page"),
+        ("docs", "ordinary object"),
+        ("section/index.html", "section page"),
+        ("private.txt", "private"),
+    ] {
+        s3.put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from(body.as_bytes().to_vec()))
+            .send()
+            .await?;
+    }
+    s3.put_object()
+        .bucket(bucket)
+        .key("move")
+        .website_redirect_location("/new-page")
+        .body(ByteStream::from_static(b"ordinary redirect object"))
+        .send()
+        .await?;
+    s3.put_bucket_website()
+        .bucket(bucket)
+        .website_configuration(
+            WebsiteConfiguration::builder()
+                .index_document(IndexDocument::builder().suffix("index.html").build()?)
+                .error_document(ErrorDocument::builder().key("404.html").build()?)
+                .build(),
+        )
+        .send()
+        .await?;
+    let policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [
+            {"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Deny","Principal":"*","Action":"s3:GetObject","Resource":format!("arn:aws:s3:::{bucket}/private.txt")}
+        ]
+    })
+    .to_string();
+    s3.put_bucket_policy().bucket(bucket).policy(policy).send().await?;
+
+    let http = reqwest::Client::builder()
+        .no_proxy()
+        .redirect(reqwest::redirect::Policy::none())
+        .build()?;
+    let website = |path: &str| format!("{}{path}", env.url);
+    let host = format!("{bucket}.website.test");
+    let base_domain = http
+        .get(website("/website-hosting/index.html"))
+        .header(HOST, "website.test")
+        .send()
+        .await?;
+    assert_eq!(base_domain.status(), 404);
+    assert!(!base_domain.text().await?.contains("<Error>"));
+    let home = http.get(website("/")).header(HOST, &host).send().await?;
+    assert_eq!(home.status(), 200);
+    assert_eq!(home.text().await?, "home");
+    let head = http.head(website("/")).header(HOST, &host).send().await?;
+    assert_eq!(head.status(), 200);
+    assert_eq!(head.text().await?, "");
+    let missing = http.get(website("/missing")).header(HOST, &host).send().await?;
+    assert_eq!(missing.status(), 404);
+    assert_eq!(missing.text().await?, "error page");
+    let denied = http.get(website("/private.txt")).header(HOST, &host).send().await?;
+    assert_eq!(denied.status(), 403);
+    assert_eq!(denied.text().await?, "error page");
+    let collision = http.get(website("/docs")).header(HOST, &host).send().await?;
+    assert_eq!(collision.status(), 200);
+    assert_eq!(collision.text().await?, "ordinary object");
+    let directory = http.get(website("/section")).header(HOST, &host).send().await?;
+    assert_eq!(directory.status(), 302);
+    assert_eq!(directory.headers()[reqwest::header::LOCATION], "/section/");
+    let section = http.get(website("/section/")).header(HOST, &host).send().await?;
+    assert_eq!(section.status(), 200);
+    assert_eq!(section.text().await?, "section page");
+    let redirect = http.get(website("/move")).header(HOST, &host).send().await?;
+    assert_eq!(redirect.status(), 301);
+    assert_eq!(redirect.headers()[reqwest::header::LOCATION], "/new-page");
+    let redirect_with_condition = http
+        .get(website("/move"))
+        .header(HOST, &host)
+        .header(reqwest::header::IF_NONE_MATCH, "*")
+        .send()
+        .await?;
+    assert_eq!(redirect_with_condition.status(), 301);
+    let redirect_with_range = http
+        .get(website("/move"))
+        .header(HOST, &host)
+        .header(reqwest::header::RANGE, "bytes=999999-")
+        .send()
+        .await?;
+    assert_eq!(redirect_with_range.status(), 301);
+    let plain_range_error = http
+        .get(website("/index.html"))
+        .header(HOST, &host)
+        .header(reqwest::header::RANGE, "bytes=999999-")
+        .send()
+        .await?;
+    assert_eq!(plain_range_error.status(), 416);
+    assert!(!plain_range_error.text().await?.contains("<Error>"));
+
+    let ordinary = s3.get_object().bucket(bucket).key("move").send().await?;
+    assert_eq!(ordinary.body.collect().await?.into_bytes(), "ordinary redirect object");
+    let ranged = s3.get_object().bucket(bucket).key("move").range("bytes=0-7").send().await?;
+    assert_eq!(ranged.body.collect().await?.into_bytes(), "ordinary");
+    let head_object = s3.head_object().bucket(bucket).key("move").send().await?;
+    let etag = head_object.e_tag().expect("object ETag");
+    let unchanged = s3.get_object().bucket(bucket).key("move").if_none_match(etag).send().await;
+    assert!(unchanged.is_err(), "ordinary S3 conditional GET should return NotModified");
+    let presigned = s3
+        .get_object()
+        .bucket(bucket)
+        .key("move")
+        .presigned(PresigningConfig::expires_in(Duration::from_secs(300))?)
+        .await?;
+    let presigned_read = http.get(presigned.uri().to_string()).send().await?;
+    assert_eq!(presigned_read.status(), 200);
+    assert_eq!(presigned_read.text().await?, "ordinary redirect object");
+    let listed = s3.list_objects_v2().bucket(bucket).send().await?;
+    assert!(listed.contents().iter().any(|object| object.key() == Some("index.html")));
+    let s3_missing = http.get(format!("{}/{bucket}/missing", env.url)).send().await?;
+    assert!(s3_missing.status().is_client_error());
+    assert!(s3_missing.text().await?.contains("<Error>"));
+    s3.put_bucket_versioning()
+        .bucket(bucket)
+        .versioning_configuration(
+            VersioningConfiguration::builder()
+                .status(BucketVersioningStatus::Enabled)
+                .build(),
+        )
+        .send()
+        .await?;
+    let first = s3
+        .put_object()
+        .bucket(bucket)
+        .key("versioned.txt")
+        .body(ByteStream::from_static(b"first"))
+        .send()
+        .await?;
+    s3.put_object()
+        .bucket(bucket)
+        .key("versioned.txt")
+        .body(ByteStream::from_static(b"second"))
+        .send()
+        .await?;
+    let versioned = s3
+        .get_object()
+        .bucket(bucket)
+        .key("versioned.txt")
+        .version_id(first.version_id().expect("first object version"))
+        .send()
+        .await?;
+    assert_eq!(versioned.body.collect().await?.into_bytes(), "first");
+
+    s3.put_bucket_website()
+        .bucket(bucket)
+        .website_configuration(
+            WebsiteConfiguration::builder()
+                .index_document(IndexDocument::builder().suffix("index.html").build()?)
+                .routing_rules(
+                    RoutingRule::builder()
+                        .condition(Condition::builder().http_error_code_returned_equals("404").build())
+                        .redirect(Redirect::builder().replace_key_with("missing-page").build())
+                        .build(),
+                )
+                .routing_rules(
+                    RoutingRule::builder()
+                        .condition(Condition::builder().key_prefix_equals("docs/").build())
+                        .redirect(Redirect::builder().replace_key_with("docs-page").build())
+                        .build(),
+                )
+                .routing_rules(
+                    RoutingRule::builder()
+                        .condition(Condition::builder().http_error_code_returned_equals("500").build())
+                        .redirect(Redirect::builder().replace_key_with("wrong-status").build())
+                        .build(),
+                )
+                .routing_rules(
+                    RoutingRule::builder()
+                        .condition(Condition::builder().http_error_code_returned_equals("416").build())
+                        .redirect(Redirect::builder().replace_key_with("range-error").build())
+                        .build(),
+                )
+                .build(),
+        )
+        .send()
+        .await?;
+    let rule = http.get(website("/docs/missing")).header(HOST, &host).send().await?;
+    assert_eq!(rule.status(), 301);
+    assert_eq!(
+        rule.headers()[reqwest::header::LOCATION],
+        "http://website-hosting.website.test/missing-page"
+    );
+    let ranged_error = http
+        .get(website("/index.html"))
+        .header(HOST, &host)
+        .header(reqwest::header::RANGE, "bytes=999999-")
+        .send()
+        .await?;
+    assert_eq!(ranged_error.status(), 301);
+    assert_eq!(
+        ranged_error.headers()[reqwest::header::LOCATION],
+        "http://website-hosting.website.test/range-error"
+    );
+
+    s3.put_bucket_website()
+        .bucket(bucket)
+        .website_configuration(
+            WebsiteConfiguration::builder()
+                .redirect_all_requests_to(RedirectAllRequestsTo::builder().host_name("destination.test").build()?)
+                .build(),
+        )
+        .send()
+        .await?;
+    let all = http.get(website("/a%20b?x=1")).header(HOST, &host).send().await?;
+    assert_eq!(all.status(), 301);
+    assert_eq!(all.headers()[reqwest::header::LOCATION], "http://destination.test/a%20b?x=1");
+
+    let dotted = "my.website.bucket";
+    s3.create_bucket().bucket(dotted).send().await?;
+    s3.put_bucket_website()
+        .bucket(dotted)
+        .website_configuration(
+            WebsiteConfiguration::builder()
+                .redirect_all_requests_to(RedirectAllRequestsTo::builder().host_name("destination.test").build()?)
+                .build(),
+        )
+        .send()
+        .await?;
+    let dotted_host = format!("{dotted}.website.test");
+    let dotted_response = http.get(website("/key")).header(HOST, dotted_host).send().await?;
+    assert_eq!(dotted_response.status(), 301);
+    assert_eq!(dotted_response.headers()[reqwest::header::LOCATION], "http://destination.test/key");
+    env.stop_server();
+    Ok(())
+}
@@ -208,7 +208,6 @@ aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a",
 urlencoding = { workspace = true }
 smallvec = { workspace = true, features = ["serde"] }
 shadow-rs = { workspace = true, default-features = false }
-async-recursion.workspace = true
 aws-credential-types = { workspace = true }
 aws-smithy-types = { workspace = true }
 aws-smithy-runtime-api = { workspace = true, features = ["http-1x"] }
@@ -448,7 +448,7 @@ pub mod disk {
 pub mod error {
    pub use crate::error::{
        Error, PoolMetadataError, PoolMetadataFailure, Result, StorageError, classify_system_path_failure_reason,
-        is_err_bucket_not_found, is_err_object_not_found, is_err_version_not_found,
+        is_err_bucket_not_found, is_err_invalid_upload_id, is_err_object_not_found, is_err_version_not_found,
    };
 }

@@ -495,6 +495,193 @@ mod tests {
        assert!(rate > 0.0);
    }

+    // Tokio's paused clock does not advance ratelimit's StdClock. Keep natural
+    // refill below one whole token for years and supply each exact budget here.
+    fn supply_reader_test_tokens(throttle: &BucketThrottle, available: u64) {
+        let rate = u64::try_from(throttle.node_bandwidth_per_sec).expect("test throttle rate should be positive");
+        let burst = throttle.burst();
+        *throttle.limiter.lock().expect("test throttle should not poison") = Ratelimiter::builder(rate)
+            .max_tokens(burst)
+            .initial_available(available)
+            .period(Duration::from_nanos(u64::MAX))
+            .build()
+            .expect("controlled token budget should be valid");
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_monitored_reader_retains_partial_tokens_across_waits() {
+        use crate::bucket::bandwidth::reader::{MonitorReaderOptions, MonitoredReader};
+        use futures_util::task::noop_waker_ref;
+        use std::pin::Pin;
+        use std::task::{Context, Poll};
+        use tokio::io::{AsyncRead, ReadBuf};
+
+        for header_size in [0, 25] {
+            let monitor = Monitor::new(1);
+            let opts = BucketOptions {
+                name: "reader-budget".to_string(),
+                replication_arn: "reader-budget-arn".to_string(),
+            };
+            monitor.set_bandwidth_limit(&opts.name, &opts.replication_arn, 100);
+            let throttle = monitor.throttle(&opts).expect("test throttle should exist");
+            supply_reader_test_tokens(&throttle, 25);
+            let data = [0xAB; 100];
+            let mut reader = MonitoredReader::new(
+                monitor,
+                data.as_slice(),
+                MonitorReaderOptions {
+                    bucket_options: opts,
+                    header_size,
+                },
+            );
+            let mut output = [0; 100];
+            let mut buf = ReadBuf::new(&mut output);
+            let mut cx = Context::from_waker(noop_waker_ref());
+
+            assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut buf).is_pending());
+            assert!(buf.filled().is_empty(), "partial payment must not authorize payload reads");
+            assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+
+            supply_reader_test_tokens(&throttle, 75);
+            tokio::time::advance(Duration::from_secs(1)).await;
+            assert!(
+                matches!(Pin::new(&mut reader).poll_read(&mut cx, &mut buf), Poll::Ready(Ok(()))),
+                "25 paid tokens plus 75 later tokens must fund the original read with header_size={header_size}"
+            );
+            assert_eq!(buf.filled(), &data[..100 - header_size]);
+            assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+        }
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_monitored_reader_preserves_paid_read_across_cancellation() {
+        use crate::bucket::bandwidth::reader::{MonitorReaderOptions, MonitoredReader};
+        use futures_util::task::noop_waker_ref;
+        use std::future::Future;
+        use std::pin::Pin;
+        use std::task::{Context, Poll};
+        use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt, ReadBuf};
+
+        let monitor = Monitor::new(1);
+        let opts = BucketOptions {
+            name: "reader-pending".to_string(),
+            replication_arn: "reader-pending-arn".to_string(),
+        };
+        monitor.set_bandwidth_limit(&opts.name, &opts.replication_arn, 100);
+        let throttle = monitor.throttle(&opts).expect("test throttle should exist");
+        supply_reader_test_tokens(&throttle, 100);
+        let (mut writer, inner) = tokio::io::duplex(100);
+        let mut reader = MonitoredReader::new(
+            monitor,
+            inner,
+            MonitorReaderOptions {
+                bucket_options: opts,
+                header_size: 0,
+            },
+        );
+        let mut cx = Context::from_waker(noop_waker_ref());
+        let mut first_output = [0; 100];
+        let mut first_read = Box::pin(reader.read(&mut first_output));
+        assert!(first_read.as_mut().poll(&mut cx).is_pending());
+        assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+        drop(first_read);
+        writer
+            .write_all(&[0xAB; 100])
+            .await
+            .expect("duplex writer should supply the pending read");
+
+        // A canceled read future keeps its reader alive. Resume in a smaller
+        // buffer without charging the already-paid allowance a second time.
+        let mut short_output = [0; 40];
+        let mut short_buf = ReadBuf::new(&mut short_output);
+        assert!(matches!(Pin::new(&mut reader).poll_read(&mut cx, &mut short_buf), Poll::Ready(Ok(()))));
+        assert_eq!(short_buf.filled(), &[0xAB; 40]);
+        assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+
+        // Completing the short read discards unused allowance, matching the
+        // existing precharge policy. The next read must buy a fresh budget.
+        let mut remaining_output = [0; 100];
+        let mut remaining_buf = ReadBuf::new(&mut remaining_output);
+        assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut remaining_buf).is_pending());
+        supply_reader_test_tokens(&throttle, 100);
+        tokio::time::advance(Duration::from_secs(1)).await;
+        assert!(matches!(
+            Pin::new(&mut reader).poll_read(&mut cx, &mut remaining_buf),
+            Poll::Ready(Ok(()))
+        ));
+        assert_eq!(remaining_buf.filled(), &[0xAB; 60]);
+        assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+
+        drop(writer);
+        let mut eof_output = [0; 100];
+        let mut eof_buf = ReadBuf::new(&mut eof_output);
+        assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut eof_buf).is_pending());
+        supply_reader_test_tokens(&throttle, 100);
+        tokio::time::advance(Duration::from_secs(1)).await;
+        assert!(matches!(Pin::new(&mut reader).poll_read(&mut cx, &mut eof_buf), Poll::Ready(Ok(()))));
+        assert!(eof_buf.filled().is_empty());
+        assert_eq!(throttle.limiter.lock().expect("test throttle").available(), 0);
+        assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut eof_buf).is_pending());
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_monitored_reader_shrinks_unpaid_payload_after_bandwidth_reduction() {
+        use crate::bucket::bandwidth::reader::{MonitorReaderOptions, MonitoredReader};
+        use futures_util::task::noop_waker_ref;
+        use std::pin::Pin;
+        use std::task::{Context, Poll};
+        use tokio::io::{AsyncRead, ReadBuf};
+
+        for header_size in [0, 25] {
+            let monitor = Monitor::new(1);
+            let opts = BucketOptions {
+                name: "reader-reduced-budget".to_string(),
+                replication_arn: "reader-reduced-budget-arn".to_string(),
+            };
+            monitor.set_bandwidth_limit(&opts.name, &opts.replication_arn, 100);
+            let initial_throttle = monitor.throttle(&opts).expect("initial throttle should exist");
+            supply_reader_test_tokens(&initial_throttle, 25);
+            let data = [0xAB; 100];
+            let mut reader = MonitoredReader::new(
+                monitor.clone(),
+                data.as_slice(),
+                MonitorReaderOptions {
+                    bucket_options: opts.clone(),
+                    header_size,
+                },
+            );
+            let mut output = [0; 100];
+            let mut buf = ReadBuf::new(&mut output);
+            let mut cx = Context::from_waker(noop_waker_ref());
+            assert!(Pin::new(&mut reader).poll_read(&mut cx, &mut buf).is_pending());
+            assert!(buf.filled().is_empty());
+            assert_eq!(initial_throttle.limiter.lock().expect("initial throttle").available(), 0);
+
+            monitor.set_bandwidth_limit(&opts.name, &opts.replication_arn, 10);
+            let reduced_throttle = monitor.throttle(&opts).expect("reduced throttle should exist");
+            assert_eq!(reduced_throttle.burst(), 10);
+            supply_reader_test_tokens(&reduced_throttle, 0);
+            tokio::time::advance(Duration::from_secs(1)).await;
+            let result = Pin::new(&mut reader).poll_read(&mut cx, &mut buf);
+            let expected_payload = if header_size == 0 {
+                assert!(
+                    matches!(result, Poll::Ready(Ok(()))),
+                    "already-paid payload must not wait for the old burst's unpaid remainder"
+                );
+                25
+            } else {
+                assert!(result.is_pending(), "the initial 25 tokens paid only the header");
+                assert!(buf.filled().is_empty(), "payload still requires its own reduced budget");
+                supply_reader_test_tokens(&reduced_throttle, 10);
+                tokio::time::advance(Duration::from_secs(1)).await;
+                assert!(matches!(Pin::new(&mut reader).poll_read(&mut cx, &mut buf), Poll::Ready(Ok(()))));
+                10
+            };
+            assert_eq!(buf.filled(), &data[..expected_payload]);
+            assert_eq!(reduced_throttle.limiter.lock().expect("reduced throttle").available(), 0);
+        }
+    }
+
    #[test]
    fn test_consume_refills_continuously() {
        let clock = TestClock::new();
@@ -41,6 +41,8 @@ pub struct MonitoredReader<R> {
    m: Arc<Monitor>,
    opts: MonitorReaderOptions,
    wait_state: std::sync::Mutex<Option<WaitState>>,
+    // Payload limit and unpaid tokens for the read retained across Pending polls.
+    pending_read: Option<(usize, u64)>,
    temp_buf: Vec<u8>,
 }

@@ -63,6 +65,7 @@ impl<R> MonitoredReader<R> {
            m,
            opts,
            wait_state: std::sync::Mutex::new(None),
+            pending_read: None,
            temp_buf: Vec::new(),
        }
    }
@@ -90,17 +93,30 @@ impl<R: AsyncRead + Unpin> AsyncRead for MonitoredReader<R> {

        let throttle = match this.m.throttle(&this.opts.bucket_options) {
            Some(t) => t,
-            None => return Pin::new(&mut this.r).poll_read(cx, buf),
+            None => {
+                this.pending_read = None;
+                return Pin::new(&mut this.r).poll_read(cx, buf);
+            }
        };

        let b = throttle.burst();
        debug_assert!(b >= 1, "burst must be at least 1");
-        let (need, tokens) = calc_need_and_tokens(b, buf.remaining(), &mut this.opts.header_size);
-        let (deficit, rate, consumed) = throttle.consume(tokens);
-        let need = need.min(consumed as usize);
+        let (need, tokens) = match this.pending_read {
+            Some(reservation) => reservation,
+            None => calc_need_and_tokens(b, buf.remaining(), &mut this.opts.header_size),
+        };
+        // A lower limit may discard unpaid payload, but must retain paid
+        // payload and any header tokens still owed by this pending read.
+        let excess_payload = need.saturating_sub(usize::try_from(b).unwrap_or(usize::MAX));
+        let reduction = tokens.min(u64::try_from(excess_payload).unwrap_or(u64::MAX));
+        let need = need.saturating_sub(usize::try_from(reduction).unwrap_or(usize::MAX));
+        let tokens = tokens - reduction;
+        let (deficit, rate, _) = throttle.consume(tokens);
+        this.pending_read = Some((need, deficit));

        if deficit > 0 && rate > 0.0 {
-            let duration = std::time::Duration::from_secs_f64(deficit as f64 / rate);
+            // A changed limit may reduce the burst while this read is pending.
+            let duration = std::time::Duration::from_secs_f64(deficit.min(b) as f64 / rate);
            debug!(
                tokens = tokens,
                deficit = deficit,
@@ -115,14 +131,17 @@ impl<R: AsyncRead + Unpin> AsyncRead for MonitoredReader<R> {
                        warn!("MonitoredReader wait_state mutex poisoned, recovering");
                        e.into_inner()
                    }) = Some(WaitState { sleep });
-                    return Poll::Pending;
                }
-                Poll::Ready(()) => {}
+                Poll::Ready(()) => cx.waker().wake_by_ref(),
            }
+            return Poll::Pending;
        }

        let filled_before = buf.filled().len();
        let result = poll_limited_read(&mut this.r, cx, buf, need, &mut this.temp_buf);
+        if result.is_ready() {
+            this.pending_read = None;
+        }
        if let Poll::Ready(Ok(())) = result {
            let read_bytes = buf.filled().len().saturating_sub(filled_before) as u64;
            if read_bytes > 0 {
@@ -409,6 +409,11 @@ pub struct BucketTargetSys {
    /// lock (never held across an await) so the replication worker can record
    /// a verdict from inside its synchronous PUT-response audit.
    version_identity_map: Arc<std::sync::RwLock<HashMap<String, VersionIdentityCapability>>>,
+    /// Target ARNs whose remote rejected a PUT / CreateMultipartUpload
+    /// `versionId` query with `InvalidArgument`; later requests to the same
+    /// ARN omit the query instead of failing again. Reset alongside
+    /// `version_identity_map` — same lifetime, same reasoning.
+    put_version_id_query_rejected: Arc<std::sync::RwLock<HashSet<String>>>,
    pub targets_map: Arc<RwLock<HashMap<String, Vec<BucketTarget>>>>,
    /// Buckets whose persisted `bucket-targets.json` exists but cannot be
    /// decoded (rustfs/backlog#2282). Written under the bucket's update mutex
@@ -458,6 +463,7 @@ impl BucketTargetSys {
            arn_remotes_map: Arc::new(RwLock::new(HashMap::new())),
            ssec_passthrough_map: Arc::new(RwLock::new(HashMap::new())),
            version_identity_map: Arc::new(std::sync::RwLock::new(HashMap::new())),
+            put_version_id_query_rejected: Arc::new(std::sync::RwLock::new(HashSet::new())),
            targets_map: Arc::new(RwLock::new(HashMap::new())),
            unreadable_targets: Arc::new(RwLock::new(HashSet::new())),
            h_mutex: Arc::new(RwLock::new(HashMap::new())),
@@ -782,10 +788,36 @@ impl BucketTargetSys {
                health_map.remove(&target.arn);
                ssec_map.remove(&target.arn);
                self.forget_version_identity_capability(&target.arn);
+                self.forget_put_version_id_query_rejected(&target.arn);
            }
        }
    }

+    /// Whether this target ARN is already known to reject a `versionId`
+    /// query on PUT / CreateMultipartUpload.
+    pub fn put_version_id_query_rejected(&self, arn: &str) -> bool {
+        self.put_version_id_query_rejected
+            .read()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .contains(arn)
+    }
+
+    /// Record that this target ARN rejected a `versionId` query on PUT /
+    /// CreateMultipartUpload, so later requests to it omit the query.
+    pub fn record_put_version_id_query_rejected(&self, arn: &str) {
+        self.put_version_id_query_rejected
+            .write()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .insert(arn.to_string());
+    }
+
+    fn forget_put_version_id_query_rejected(&self, arn: &str) {
+        self.put_version_id_query_rejected
+            .write()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .remove(arn);
+    }
+
    /// Cached version-identity verdict for a target ARN; `Unknown` until a
    /// replication write or a replication-check VersionFidelity probe judged
    /// it since the target was built.
@@ -1252,6 +1284,7 @@ impl BucketTargetSys {
                    .is_none_or(|edited| !same_replication_service(edited, &target))
                {
                    self.forget_version_identity_capability(&target.arn);
+                    self.forget_put_version_id_query_rejected(&target.arn);
                }
                self.update_bandwidth_limit(bucket, &target.arn, 0);
            }
@@ -1449,8 +1482,13 @@ pub fn resolve_delete_api_version_id(version_id: Option<String>, opts: &RemoveOb
 /// "null" (the delete path already maps it via `target_delete_version_id`),
 /// and an empty id means the source object carries no version: send no query
 /// so an unversioned target stays valid.
-fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
-    if source_version_id.is_empty() {
+///
+/// A standard S3 target (AWS S3 included) instead rejects this query on PUT /
+/// CreateMultipartUpload outright with `InvalidArgument: This operation does
+/// not accept a version-id`. Once a target ARN has been observed to reject
+/// it (`arn_rejected_query`), it is never sent to that ARN again.
+fn resolve_put_api_version_id(source_version_id: &str, arn_rejected_query: bool) -> Option<&str> {
+    if arn_rejected_query || source_version_id.is_empty() {
        None
    } else if Uuid::parse_str(source_version_id).is_ok_and(|uuid| uuid.is_nil()) {
        Some(rustfs_filemeta::NULL_VERSION_ID)
@@ -1459,6 +1497,13 @@ fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
    }
 }

+/// True when a PUT / CreateMultipartUpload service error is the standard S3
+/// rejection of a `versionId` query on those operations (AWS S3 and other
+/// strict S3-compatible targets; MinIO/RustFS peers accept the query).
+fn is_put_version_id_query_rejected(code: Option<&str>, message: Option<&str>) -> bool {
+    code == Some("InvalidArgument") && message.is_some_and(|m| m.to_ascii_lowercase().contains("version-id"))
+}
+
 /// Resolve the S3 `versionId` for a proxied read against a remote target.
 /// RustFS represents the null version internally as the nil UUID while the S3
 /// API addresses it as the literal "null" (same mapping as
@@ -2251,7 +2296,9 @@ impl TargetClient {
        if !version_id.is_empty() {
            insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
        }
-        let api_version_id = resolve_put_api_version_id(&version_id).map(ToOwned::to_owned);
+        let api_version_id =
+            resolve_put_api_version_id(&version_id, BucketTargetSys::get().put_version_id_query_rejected(&self.arn))
+                .map(ToOwned::to_owned);

        // A PUT carrying Object Lock parameters must also carry Content-MD5 or
        // an x-amz-checksum-* header on AWS-compatible targets (rustfs#7082).
@@ -2333,6 +2380,9 @@ impl TargetClient {
                SdkError::ServiceError(service_err) => {
                    let err = service_err.into_err();
                    let meta = err.meta();
+                    if is_put_version_id_query_rejected(meta.code(), meta.message()) {
+                        BucketTargetSys::get().record_put_version_id_query_rejected(&self.arn);
+                    }
                    let error = match (meta.code(), meta.message()) {
                        (Some(code), Some(message)) => format!("put_object failed: {code}: {message}"),
                        (Some(code), None) => format!("put_object failed: {code}"),
@@ -2369,7 +2419,9 @@ impl TargetClient {
        let version_id = opts.internal.source_version_id.clone();
        // The remote version of a multipart replication is decided at initiate
        // time; CompleteMultipartUpload does not read a versionId.
-        let api_version_id = resolve_put_api_version_id(&version_id).map(ToOwned::to_owned);
+        let api_version_id =
+            resolve_put_api_version_id(&version_id, BucketTargetSys::get().put_version_id_query_rejected(&self.arn))
+                .map(ToOwned::to_owned);

        match self
            .client
@@ -2395,7 +2447,15 @@ impl TargetClient {
            .await
        {
            Ok(res) => Ok(res.upload_id.unwrap_or_default()),
-            Err(e) => Err(e.into()),
+            Err(e) => {
+                if let SdkError::ServiceError(service_err) = &e {
+                    let meta = service_err.err().meta();
+                    if is_put_version_id_query_rejected(meta.code(), meta.message()) {
+                        BucketTargetSys::get().record_put_version_id_query_rejected(&self.arn);
+                    }
+                }
+                Err(e.into())
+            }
        }
    }

@@ -3597,6 +3657,87 @@ mod tests {
        assert_eq!(sys.version_identity_capability(arn), VersionIdentityCapability::Unknown);
    }

+    #[test]
+    fn put_version_id_query_rejection_is_per_arn_and_forgotten_with_the_target() {
+        let sys = BucketTargetSys::default();
+        let arn = "arn:rustfs:replication:us-east-1:bucket:versionid";
+        assert!(!sys.put_version_id_query_rejected(arn));
+        sys.record_put_version_id_query_rejected(arn);
+        assert!(sys.put_version_id_query_rejected(arn));
+        assert!(!sys.put_version_id_query_rejected("other"));
+        // A rebuilt target may point at a different service.
+        sys.forget_put_version_id_query_rejected(arn);
+        assert!(!sys.put_version_id_query_rejected(arn));
+    }
+
+    #[tokio::test]
+    async fn update_all_targets_forgets_put_version_id_rejection_when_endpoint_changes() {
+        // Editing a target's ARN to point at a different endpoint must not
+        // carry over a stale "this remote rejects versionId" verdict from the
+        // service it used to point at (rustfs review follow-up on the
+        // versionId-on-PUT fix): otherwise a peer that legitimately needs the
+        // query silently loses version-id fidelity until the process restarts.
+        let sys = BucketTargetSys::default();
+        let arn = "arn:rustfs:replication:us-east-1:bucket:versionid";
+        let target = |endpoint: &str| BucketTarget {
+            arn: arn.to_string(),
+            endpoint: endpoint.to_string(),
+            target_bucket: "target-bucket".to_string(),
+            region: "us-east-1".to_string(),
+            credentials: Some(Credentials {
+                access_key: "access".to_string(),
+                secret_key: "secret".to_string(),
+                session_token: None,
+                expiration: None,
+            }),
+            ..Default::default()
+        };
+
+        sys.update_all_targets(
+            "bucket",
+            Some(&BucketTargets {
+                targets: vec![target("aws-s3.example:9000")],
+            }),
+        )
+        .await;
+        sys.record_put_version_id_query_rejected(arn);
+        assert!(sys.put_version_id_query_rejected(arn));
+
+        // Same ARN, different endpoint: the update path (not `delete`) is the
+        // only thing that runs here, so the reset must come from
+        // `update_all_targets_locked` itself.
+        sys.update_all_targets(
+            "bucket",
+            Some(&BucketTargets {
+                targets: vec![target("minio-peer.example:9000")],
+            }),
+        )
+        .await;
+
+        assert!(
+            !sys.put_version_id_query_rejected(arn),
+            "editing the target to a different endpoint must forget the stale rejection verdict"
+        );
+    }
+
+    #[test]
+    fn resolve_put_api_version_id_omits_query_once_rejected() {
+        let version_id = Uuid::new_v4().to_string();
+        assert_eq!(resolve_put_api_version_id(&version_id, false), Some(version_id.as_str()));
+        assert_eq!(resolve_put_api_version_id(&version_id, true), None);
+    }
+
+    #[test]
+    fn is_put_version_id_query_rejected_matches_aws_invalid_argument() {
+        assert!(is_put_version_id_query_rejected(
+            Some("InvalidArgument"),
+            Some("This operation does not accept a version-id.")
+        ));
+        assert!(!is_put_version_id_query_rejected(Some("AccessDenied"), Some("access denied")));
+        assert!(!is_put_version_id_query_rejected(Some("InvalidArgument"), Some("bucket name is invalid")));
+        assert!(!is_put_version_id_query_rejected(None, None));
+    }
+
    #[test]
    fn endpoint_health_key_preserves_explicit_port() {
        let url = Url::parse("https://remote.example:9443").expect("url should parse");
@@ -64,25 +64,40 @@ impl BucketDurabilityConfig {
    }
 }

-/// Default durability tier seeded into a newly created bucket's metadata
-/// (rustfs/backlog#1811). `relaxed` aligns new buckets with MinIO's default
-/// posture: object data is still fdatasynced, while xl.meta and directory-entry
-/// fsyncs follow the relaxed durability gate.
+/// Default durability policy for newly created buckets. An unset value must
+/// inherit the process-wide mode so a default single-node deployment remains
+/// strict. Relaxed and none remain explicit operator choices.
 pub const ENV_NEW_BUCKET_DURABILITY_MODE: &str = "RUSTFS_NEW_BUCKET_DURABILITY_MODE";
-pub const DEFAULT_NEW_BUCKET_DURABILITY_MODE: &str = BUCKET_DURABILITY_MODE_RELAXED;
+pub const DEFAULT_NEW_BUCKET_DURABILITY_MODE: &str = "inherit";
+
+const EVENT_NEW_BUCKET_DURABILITY_MODE: &str = "new_bucket_durability_mode";
+const LOG_COMPONENT_ECSTORE: &str = "ecstore";
+const LOG_SUBSYSTEM_BUCKET_DURABILITY: &str = "bucket_durability";

 /// The `durability.json` bytes to seed into a freshly created bucket's metadata.
 /// Empty means "no override" (the bucket then follows the global
-/// `RUSTFS_DURABILITY_MODE`); otherwise the serialized chosen tier. Operators
-/// can set `inherit` to disable the new-bucket override. Invalid values also
-/// fail closed to inherit the global mode instead of seeding a surprising tier.
-pub fn new_bucket_durability_config_json() -> Vec<u8> {
+/// `RUSTFS_DURABILITY_MODE`); otherwise the serialized chosen tier. Invalid
+/// values also fail closed to inherit the global mode instead of seeding a
+/// surprising tier.
+pub fn new_bucket_durability_config_json(bucket: &str) -> Vec<u8> {
    let raw = std::env::var(ENV_NEW_BUCKET_DURABILITY_MODE).unwrap_or_else(|_| DEFAULT_NEW_BUCKET_DURABILITY_MODE.to_string());
    let mode = raw.trim();
    if mode.eq_ignore_ascii_case("inherit") || mode.is_empty() || !BucketDurabilityConfig::is_valid_mode(mode) {
        return Vec::new();
    }
-    serde_json::to_vec(&BucketDurabilityConfig::new(mode)).expect("BucketDurabilityConfig serialization cannot fail")
+    let config = BucketDurabilityConfig::new(mode);
+    if let Some(mode) = config.normalized_mode().filter(|mode| mode != BUCKET_DURABILITY_MODE_STRICT) {
+        tracing::warn!(
+            event = EVENT_NEW_BUCKET_DURABILITY_MODE,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_BUCKET_DURABILITY,
+            state = "non_strict_new_bucket_override",
+            bucket = %bucket,
+            mode = %mode,
+            "New bucket durability is explicitly configured below strict"
+        );
+    }
+    serde_json::to_vec(&config).expect("BucketDurabilityConfig serialization cannot fail")
 }

 #[cfg(test)]
@@ -90,7 +105,7 @@ mod tests {
    use super::*;

    fn new_bucket_seeded_mode() -> Option<String> {
-        let json = new_bucket_durability_config_json();
+        let json = new_bucket_durability_config_json("test-bucket");
        if json.is_empty() {
            return None;
        }
@@ -132,9 +147,9 @@ mod tests {
    }

    #[test]
-    fn new_bucket_default_seeds_relaxed_when_unset() {
+    fn new_bucket_default_inherits_when_unset() {
        temp_env::with_var_unset(ENV_NEW_BUCKET_DURABILITY_MODE, || {
-            assert_eq!(new_bucket_seeded_mode().as_deref(), Some(BUCKET_DURABILITY_MODE_RELAXED));
+            assert_eq!(new_bucket_seeded_mode(), None);
        });
    }

@@ -5606,6 +5606,11 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(

    //debug!("lc_event.action: {:?}", lc_event.action);
    debug!("expiry_on_non_transitioned_objects opts: {:?}", opts);
+    let usage_accounting = if lc_event.action == IlmAction::DeleteAction && !versioned && !version_suspended && !oi.is_dir {
+        Some(crate::data_usage::begin_expiry_usage_accounting(&oi.bucket).await)
+    } else {
+        None
+    };
    let mut dobj = match api
        .delete_object_with_tier_delete_journal(&oi.bucket, &encode_dir_object(&oi.name), opts)
        .await
@@ -5625,6 +5630,14 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
            return false;
        }
    };
+    if let Some(accounting) = usage_accounting
+        && !dobj.name.is_empty()
+        && !dobj.delete_marker
+        && !dobj.is_dir
+        && let Ok(size) = crate::data_usage::quota_object_size(&dobj)
+    {
+        accounting.commit(size).await;
+    }
    schedule_lifecycle_replication_delete_if_needed(oi, &dobj).await;

    // The object (or all its versions, for delete_all) was expired; evict any
@@ -12891,6 +12904,130 @@ mod tests {
        );
    }

+    #[tokio::test]
+    #[serial]
+    async fn ordinary_expiry_accounts_the_committed_receipt_once() {
+        use crate::bucket::quota::{QuotaOperation, checker::QuotaChecker};
+        use crate::data_usage::{apply_bucket_usage_memory_overlay, replace_bucket_usage_memory_from_info};
+        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
+
+        let (_disk_paths, ecstore) = setup_test_env().await;
+        for size in [0usize, 42] {
+            let bytes = u64::try_from(size).expect("fixture size fits");
+            let bucket = format!("expiry-accounting-{}", Uuid::new_v4().simple());
+            create_test_bucket(&ecstore, &bucket).await;
+            let mut reader = PutObjReader::from_vec(vec![0; size]);
+            let mut queued = ecstore
+                .put_object(&bucket, "expire", &mut reader, &ObjectOptions::default())
+                .await
+                .expect("create expiring object");
+            let mut reader = PutObjReader::from_vec(vec![0; size]);
+            ecstore
+                .put_object(&bucket, "keep", &mut reader, &ObjectOptions::default())
+                .await
+                .expect("create retained object");
+            let mut baseline = DataUsageInfo {
+                last_update: Some(std::time::SystemTime::now()),
+                usage_snapshot_complete: true,
+                buckets_count: 1,
+                ..Default::default()
+            };
+            baseline.buckets_usage.insert(
+                bucket.clone(),
+                BucketUsageInfo {
+                    objects_count: 2,
+                    versions_count: 2,
+                    size: bytes * 2,
+                    ..Default::default()
+                },
+            );
+            baseline.bucket_sizes.insert(bucket.clone(), bytes * 2);
+            baseline.calculate_totals();
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            // Legacy quota admission consumes the same conservative overlay.
+            // Durable reservations have their own storage-commit accounting.
+            metadata_sys::update(
+                &bucket,
+                rustfs_config::QUOTA_CONFIG_FILE,
+                format!(r#"{{"quota":{},"quota_type":"Hard"}}"#, bytes * 2).into_bytes(),
+            )
+            .await
+            .expect("configure the legacy quota");
+            let checker =
+                QuotaChecker::new(Arc::new(tokio::sync::RwLock::new(metadata_sys::BucketMetadataSys::new(ecstore.clone()))));
+            let attempted_size = bytes.max(1);
+            let quota = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                .await
+                .expect("check quota before expiration");
+            assert!(!quota.allowed);
+            assert!(!quota.uses_durable_reservations);
+            assert_eq!(quota.current_usage, Some(bytes * 2));
+            // Queued metadata is not the accounting receipt from the storage commit.
+            queued.size = 1;
+            let event = lifecycle::Event {
+                action: IlmAction::DeleteAction,
+                ..Default::default()
+            };
+            let incarnation = ecstore
+                .bucket_incarnation_id_from_disk(&bucket)
+                .await
+                .expect("bucket incarnation");
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
+                assert_eq!(
+                    super::apply_expiry_on_non_transitioned_objects(
+                        ecstore.clone(),
+                        &queued,
+                        &event,
+                        &LcEventSrc::Scanner,
+                        attempt_incarnation,
+                    )
+                    .await,
+                    expected
+                );
+                let mut response = baseline.clone();
+                apply_bucket_usage_memory_overlay(&mut response).await;
+                let remaining = if attempt_incarnation == incarnation { 1 } else { 2 };
+                assert_eq!(response.buckets_usage[&bucket].objects_count, remaining);
+                assert_eq!(response.buckets_usage[&bucket].size, remaining * bytes);
+                let quota = checker
+                    .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                    .await
+                    .expect("check quota before scanner confirmation");
+                assert!(!quota.allowed, "expiry must not release unconfirmed quota credit");
+                assert_eq!(quota.current_usage, Some(bytes * 2));
+            }
+            let mut confirmed = baseline.clone();
+            confirmed.last_update = Some(std::time::SystemTime::now());
+            confirmed.scanner_epoch = Some(7);
+            confirmed.scanner_cycle = Some(10);
+            let usage = confirmed.buckets_usage.get_mut(&bucket).expect("bucket is in the snapshot");
+            usage.size = bytes;
+            usage.objects_count = 1;
+            usage.versions_count = 0; // The scanner does not count unversioned objects as versions.
+            confirmed.bucket_sizes.insert(bucket.clone(), bytes);
+            confirmed.calculate_totals();
+            replace_bucket_usage_memory_from_info(&confirmed).await;
+            let quota = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                .await
+                .expect("check quota after scanner confirmation");
+            assert_eq!(quota.allowed, bytes > 0);
+            assert_eq!(quota.current_usage, Some(bytes));
+            let too_large = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, bytes + 1)
+                .await
+                .expect("check the first byte above the quota limit");
+            assert!(!too_large.allowed);
+            assert!(
+                ecstore
+                    .get_object_info(&bucket, "keep", &ObjectOptions::default())
+                    .await
+                    .is_ok()
+            );
+        }
+    }
+
    #[tokio::test]
    #[serial]
    async fn queued_delete_all_rechecks_a_same_id_rule_moved_into_the_future() {
@@ -593,7 +593,7 @@ impl BucketMetadata {
    /// durability posture.
    pub fn new_with_default_durability(name: &str) -> Self {
        let mut metadata = Self::new(name);
-        metadata.durability_config_json = super::durability::new_bucket_durability_config_json();
+        metadata.durability_config_json = super::durability::new_bucket_durability_config_json(name);
        metadata
    }

@@ -1732,21 +1732,15 @@ mod test {
    }

    #[test]
-    fn new_bucket_metadata_constructor_seeds_default_durability() {
+    fn new_bucket_metadata_constructor_inherits_default_durability() {
        temp_env::with_var_unset(crate::bucket::durability::ENV_NEW_BUCKET_DURABILITY_MODE, || {
            let metadata = BucketMetadata::new_with_default_durability("new-user-bucket");
-            assert_eq!(
-                metadata.durability_config().and_then(|cfg| cfg.normalized_mode()).as_deref(),
-                Some(crate::bucket::durability::BUCKET_DURABILITY_MODE_RELAXED)
-            );
+            assert!(metadata.durability_config().is_none());

            let encoded = metadata.marshal_msg().expect("marshal metadata");
            let decoded = BucketMetadata::unmarshal(&encoded).expect("unmarshal metadata");
            assert_eq!(decoded.durability_config_json, metadata.durability_config_json);
-            assert_eq!(
-                decoded.durability_config().and_then(|cfg| cfg.normalized_mode()).as_deref(),
-                Some(crate::bucket::durability::BUCKET_DURABILITY_MODE_RELAXED)
-            );
+            assert!(decoded.durability_config().is_none());
        });
    }

@@ -1206,6 +1206,17 @@ mod tests {
        assert_eq!(mixed_timeout, DiskError::Timeout);
    }

+    #[test]
+    fn mixed_missing_and_body_io_errors_remain_an_actionable_quorum_failure() {
+        let failures = [
+            DiskError::FileNotFound,
+            DiskError::Io(std::io::Error::other("remote body stream aborted")),
+        ];
+
+        assert!(!is_benign_not_found_listing_failure(&failures));
+        assert_eq!(classify_listing_quorum_failure(&failures), DiskError::ErasureReadQuorum);
+    }
+
    #[test]
    fn missing_path_error_classification_excludes_actionable_failures() {
        assert!(is_missing_path_error(&DiskError::FileNotFound));
@@ -901,6 +901,7 @@ pub fn build_internode_data_transport_from_env() -> Result<Arc<dyn InternodeData
 mod tests {
    use super::*;
    use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+    use tokio::io::AsyncWriteExt;
    use tokio::sync::{Barrier, Notify};

    async fn wait_for_capability_flight_waiters(entry: &PutFileCapabilityCacheEntry, waiters: usize) {
@@ -1026,6 +1027,45 @@ mod tests {
        assert!(matches!(open_err, Error::MethodNotAllowed));
    }

+    #[tokio::test]
+    async fn walk_dir_restores_explicit_remote_missing_error() {
+        let _ = rustfs_credentials::set_global_rpc_secret("walk-dir-error-test-secret".to_string());
+        let listener = match tokio::net::TcpListener::bind("127.0.0.1:0").await {
+            Ok(listener) => listener,
+            Err(err) if err.kind() == io::ErrorKind::PermissionDenied => return,
+            Err(err) => panic!("test listener should bind: {err}"),
+        };
+        let addr = listener.local_addr().expect("test listener address should be available");
+        let server = tokio::spawn(async move {
+            let (mut socket, _) = listener.accept().await.expect("test server should accept the request");
+            let mut request = [0_u8; 2048];
+            let _ = socket.read(&mut request).await.expect("request should be readable");
+            socket
+                .write_all(
+                    b"HTTP/1.1 500 Internal Server Error\r\nx-rustfs-disk-error: file-not-found\r\ncontent-length: 0\r\nconnection: close\r\n\r\n",
+                )
+                .await
+                .expect("typed disk error response should be writable");
+        });
+
+        let endpoint = format!("http://{addr}");
+        let error = match TcpHttpInternodeDataTransport
+            .open_walk_dir(WalkDirStreamRequest {
+                endpoint: endpoint.clone(),
+                disk: "disk-a".to_string(),
+                body: b"{}".to_vec(),
+                stall_timeout: Some(Duration::from_secs(2)),
+            })
+            .await
+        {
+            Ok(_) => panic!("an explicit remote missing-path response must fail the open"),
+            Err(error) => error,
+        };
+
+        assert!(matches!(error, Error::FileNotFound));
+        server.await.expect("test server task should finish");
+    }
+
    #[test]
    fn tcp_http_capabilities_are_behavior_preserving() {
        let transport = TcpHttpInternodeDataTransport;
@@ -14,13 +14,18 @@

 //! Bounded, authenticated inter-node network probes.

-use std::time::{Duration, Instant};
+use std::future::Future;
+use std::pin::Pin;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::Duration;

 use bytes::Bytes;
 use rustfs_protos::ChannelClass;
 use rustfs_protos::models::{PingBody, PingBodyBuilder};
 use rustfs_protos::proto_gen::node_service::{PingRequest, PingResponse};
 use thiserror::Error;
+use tokio::time::Instant;
 use tokio_util::sync::CancellationToken;
 use tonic::{Code, Request};

@@ -33,6 +38,40 @@ pub const MAX_NETWORK_PROBE_DURATION: Duration = Duration::from_secs(30);
 const PING_PROTOCOL_VERSION: u64 = 1;
 const LATENCY_PAYLOAD: &[u8] = b"network-probe-latency-v1";
 const EXPECTED_RESPONSE_PAYLOAD: &[u8] = b"hello, caller";
+const DIAGNOSTIC_BYTES_PER_SECOND: u64 = 1_048_576;
+const DIAGNOSTIC_CHUNK_BYTES: u64 = 16_384;
+
+#[derive(Debug)]
+struct DiagnosticPacing {
+    started: Instant,
+    deadline: Instant,
+    charged_bytes: AtomicU64,
+}
+
+impl DiagnosticPacing {
+    fn reserve(&self, bytes: u64) -> Result<Instant, NetworkPeerProbeError> {
+        if bytes == 0 {
+            return Err(NetworkPeerProbeError::LimitExceeded);
+        }
+        // Reservation is never refunded: a failed or dropped RPC may have sent some payload.
+        let previous = self
+            .charged_bytes
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |charged| {
+                charged.checked_add(bytes).filter(|total| *total <= MAX_NETWORK_PROBE_BYTES)
+            })
+            .map_err(|_| NetworkPeerProbeError::LimitExceeded)?;
+        let charged = previous.checked_add(bytes).ok_or(NetworkPeerProbeError::LimitExceeded)?;
+        let millis = u128::from(charged)
+            .checked_mul(1_000)
+            .ok_or(NetworkPeerProbeError::LimitExceeded)?
+            .div_ceil(u128::from(DIAGNOSTIC_BYTES_PER_SECOND));
+        self.started
+            .checked_add(Duration::from_millis(
+                u64::try_from(millis).map_err(|_| NetworkPeerProbeError::LimitExceeded)?,
+            ))
+            .ok_or(NetworkPeerProbeError::LimitExceeded)
+    }
+}

 #[derive(Clone, Debug, PartialEq, Eq)]
 pub struct NetworkPeerTarget {
@@ -66,6 +105,7 @@ pub enum NetworkPeerProbeError {
 #[derive(Clone, Debug)]
 pub struct NetworkPeerProbeClient {
    targets: Vec<NetworkPeerTarget>,
+    diagnostic_pacing: Option<Arc<DiagnosticPacing>>,
 }

 impl NetworkPeerProbeClient {
@@ -80,7 +120,25 @@ impl NetworkPeerProbeClient {
                address,
            })
            .collect();
-        Self { targets }
+        Self {
+            targets,
+            diagnostic_pacing: None,
+        }
+    }
+
+    /// Opt in one diagnostic run; clones share its charged payload and absolute deadline.
+    /// Default/admin probes retain their existing unpaced behavior.
+    pub fn with_diagnostic_pacing(mut self, started: Instant, duration: Duration) -> Result<Self, NetworkPeerProbeError> {
+        if self.diagnostic_pacing.is_some() || duration.is_zero() || duration > MAX_NETWORK_PROBE_DURATION {
+            return Err(NetworkPeerProbeError::LimitExceeded);
+        }
+        let deadline = started.checked_add(duration).ok_or(NetworkPeerProbeError::LimitExceeded)?;
+        self.diagnostic_pacing = Some(Arc::new(DiagnosticPacing {
+            started,
+            deadline,
+            charged_bytes: AtomicU64::new(0),
+        }));
+        Ok(self)
    }

    pub fn targets(&self) -> Vec<NetworkPeerTarget> {
@@ -110,7 +168,19 @@ impl NetworkPeerProbeClient {
            .find(|target| target.alias == peer_alias)
            .map(|target| target.address.as_str())
            .ok_or(NetworkPeerProbeError::UnknownPeer)?;
-        let probe = probe_peer(address, traffic_bytes);
+        let probe = probe_peer(address, traffic_bytes, self.diagnostic_pacing.as_deref(), cancel);
+        if let Some(pacing) = &self.diagnostic_pacing {
+            let deadline = Instant::now()
+                .checked_add(max_duration)
+                .ok_or(NetworkPeerProbeError::LimitExceeded)?
+                .min(pacing.deadline);
+            return tokio::select! {
+                biased;
+                () = cancel.cancelled() => Err(NetworkPeerProbeError::Cancelled),
+                () = tokio::time::sleep_until(deadline) => Err(NetworkPeerProbeError::TimedOut),
+                result = probe => result,
+            };
+        }
        tokio::select! {
            () = cancel.cancelled() => Err(NetworkPeerProbeError::Cancelled),
            result = tokio::time::timeout(max_duration, probe) => {
@@ -120,7 +190,12 @@ impl NetworkPeerProbeClient {
    }
 }

-async fn probe_peer(address: &str, traffic_bytes: u64) -> Result<NetworkPeerProbeMeasurement, NetworkPeerProbeError> {
+async fn probe_peer(
+    address: &str,
+    traffic_bytes: u64,
+    pacing: Option<&DiagnosticPacing>,
+    cancel: &CancellationToken,
+) -> Result<NetworkPeerProbeMeasurement, NetworkPeerProbeError> {
    let started = Instant::now();
    let mut control = client(address, ChannelClass::Control).await?;
    let latency_started = Instant::now();
@@ -132,15 +207,18 @@ async fn probe_peer(address: &str, traffic_bytes: u64) -> Result<NetworkPeerProb
    validate_ping_response(&latency_response)?;
    let latency = latency_started.elapsed();

-    let payload_len = usize::try_from(traffic_bytes).map_err(|_| NetworkPeerProbeError::LimitExceeded)?;
-    let payload = vec![0x5a; payload_len];
    let mut bulk = client(address, ChannelClass::Bulk).await?;
-    let response = bulk
-        .ping(Request::new(ping_request(&payload)))
-        .await
-        .map_err(map_status)?
-        .into_inner();
-    validate_ping_response(&response)?;
+    send_payload(traffic_bytes, pacing, cancel, &mut bulk, |bulk, payload| {
+        Box::pin(async move {
+            let response = bulk
+                .ping(Request::new(ping_request(&payload)))
+                .await
+                .map_err(map_status)?
+                .into_inner();
+            validate_ping_response(&response)
+        })
+    })
+    .await?;

    Ok(NetworkPeerProbeMeasurement {
        transferred_bytes: traffic_bytes,
@@ -149,6 +227,59 @@ async fn probe_peer(address: &str, traffic_bytes: u64) -> Result<NetworkPeerProb
    })
 }

+type PayloadSendFuture<'a> = Pin<Box<dyn Future<Output = Result<(), NetworkPeerProbeError>> + Send + 'a>>;
+
+async fn send_payload<T, F>(
+    traffic_bytes: u64,
+    pacing: Option<&DiagnosticPacing>,
+    cancel: &CancellationToken,
+    sender: &mut T,
+    mut send: F,
+) -> Result<(), NetworkPeerProbeError>
+where
+    T: Send,
+    F: for<'a> FnMut(&'a mut T, Vec<u8>) -> PayloadSendFuture<'a> + Send,
+{
+    let mut remaining = traffic_bytes;
+    while remaining > 0 {
+        let bytes = if pacing.is_some() {
+            remaining.min(DIAGNOSTIC_CHUNK_BYTES)
+        } else {
+            remaining
+        };
+        if let Some(pacing) = pacing {
+            if cancel.is_cancelled() {
+                return Err(NetworkPeerProbeError::Cancelled);
+            }
+            if Instant::now() >= pacing.deadline {
+                return Err(NetworkPeerProbeError::TimedOut);
+            }
+            let due = pacing.reserve(bytes)?;
+            tokio::select! {
+                biased;
+                () = cancel.cancelled() => return Err(NetworkPeerProbeError::Cancelled),
+                () = tokio::time::sleep_until(pacing.deadline) => return Err(NetworkPeerProbeError::TimedOut),
+                () = tokio::time::sleep_until(due) => {},
+            }
+        }
+        let payload_len = usize::try_from(bytes).map_err(|_| NetworkPeerProbeError::LimitExceeded)?;
+        let payload = vec![0x5a; payload_len];
+        if let Some(pacing) = pacing {
+            // Check before polling the send future, including a cancellation/deadline tie.
+            tokio::select! {
+                biased;
+                () = cancel.cancelled() => return Err(NetworkPeerProbeError::Cancelled),
+                () = tokio::time::sleep_until(pacing.deadline) => return Err(NetworkPeerProbeError::TimedOut),
+                result = send(sender, payload) => result?,
+            }
+        } else {
+            send(sender, payload).await?;
+        }
+        remaining -= bytes;
+    }
+    Ok(())
+}
+
 async fn client(
    address: &str,
    class: ChannelClass,
@@ -208,6 +339,205 @@ mod tests {
    use crate::layout::endpoint::Endpoint;
    use crate::layout::endpoints::{Endpoints, PoolEndpoints};

+    fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
+        NetworkPeerProbeClient::from_endpoint_pools(&topology())
+            .with_diagnostic_pacing(Instant::now(), duration)
+            .expect("bounded diagnostic pacing window")
+            .diagnostic_pacing
+            .expect("opt-in pacing is present")
+    }
+
+    #[test]
+    fn pacing_uses_exact_millisecond_ceiling_at_n_and_n_plus_one() {
+        for duration in [Duration::ZERO, MAX_NETWORK_PROBE_DURATION + Duration::from_nanos(1)] {
+            assert!(matches!(
+                NetworkPeerProbeClient::from_endpoint_pools(&topology()).with_diagnostic_pacing(Instant::now(), duration),
+                Err(NetworkPeerProbeError::LimitExceeded)
+            ));
+        }
+        for (bytes, millis) in [(1, 1), (1_047_527, 999), (1_047_528, 1_000), (1_048_576, 1_000)] {
+            let pacing = pacing(Duration::from_secs(2));
+            assert_eq!(
+                pacing.reserve(bytes).expect("bounded reservation") - pacing.started,
+                Duration::from_millis(millis)
+            );
+        }
+        for bytes in [0, MAX_NETWORK_PROBE_BYTES + 1, u64::MAX] {
+            let pacing = pacing(Duration::from_secs(2));
+            assert_eq!(pacing.reserve(bytes), Err(NetworkPeerProbeError::LimitExceeded));
+            assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 0);
+        }
+    }
+
+    #[test]
+    fn cloned_diagnostic_clients_share_one_atomic_budget_without_reset() {
+        let client = NetworkPeerProbeClient::from_endpoint_pools(&topology())
+            .with_diagnostic_pacing(Instant::now(), Duration::from_secs(2))
+            .expect("pacing");
+        let cloned = client.clone();
+        let first = client.diagnostic_pacing.as_ref().expect("first budget");
+        let second = cloned.diagnostic_pacing.as_ref().expect("cloned budget");
+        assert!(Arc::ptr_eq(first, second));
+        let mut due = std::thread::scope(|scope| {
+            let first = scope.spawn(|| first.reserve(MAX_NETWORK_PROBE_BYTES / 2));
+            let second = scope.spawn(|| second.reserve(MAX_NETWORK_PROBE_BYTES / 2));
+            [
+                first.join().expect("first reservation thread").expect("first half"),
+                second.join().expect("second reservation thread").expect("second half"),
+            ]
+        });
+        due.sort();
+        assert_eq!(
+            due,
+            [
+                first.started + Duration::from_millis(500),
+                first.started + Duration::from_secs(1)
+            ]
+        );
+        assert_eq!(first.reserve(1), Err(NetworkPeerProbeError::LimitExceeded));
+        assert_eq!(first.charged_bytes.load(Ordering::Acquire), MAX_NETWORK_PROBE_BYTES);
+        assert!(matches!(
+            cloned.with_diagnostic_pacing(Instant::now(), Duration::from_secs(2)),
+            Err(NetworkPeerProbeError::LimitExceeded)
+        ));
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn failed_peer_payload_stays_charged_before_the_next_peer_send() {
+        let pacing = pacing(Duration::from_secs(2));
+        let cancel = CancellationToken::new();
+        let mut observed_partial_bytes = 0;
+        let failed = send_payload(32_768, Some(&pacing), &cancel, &mut observed_partial_bytes, |partial, payload| {
+            Box::pin(async move {
+                assert_eq!(payload.len(), 16_384);
+                // Model failure after a partial write: the sender cannot safely refund the remainder.
+                *partial += 8_192;
+                Err(NetworkPeerProbeError::Unreachable)
+            })
+        })
+        .await;
+        assert_eq!(failed, Err(NetworkPeerProbeError::Unreachable));
+        assert_eq!(observed_partial_bytes, 8_192);
+        assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 16_384);
+        let started = pacing.started;
+        let mut second_bytes = 0;
+        send_payload(16_384, Some(&pacing), &cancel, &mut second_bytes, move |bytes, payload| {
+            Box::pin(async move {
+                assert!(started.elapsed() >= Duration::from_millis(32));
+                *bytes += payload.len();
+                Ok(())
+            })
+        })
+        .await
+        .expect("second peer uses remaining shared budget");
+        assert_eq!(second_bytes, 16_384);
+        assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 32_768);
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn cancellation_and_deadline_take_priority_over_an_earned_send() {
+        for cancelled in [false, true] {
+            let pacing = pacing(Duration::from_millis(16));
+            let cancel = CancellationToken::new();
+            let mut sends = 0;
+            {
+                let mut sending =
+                    std::pin::pin!(send_payload(8_192, Some(&pacing), &cancel, &mut sends, |sends, _| Box::pin(async move {
+                        *sends += 1;
+                        Ok(())
+                    })));
+                assert!(futures::poll!(&mut sending).is_pending());
+                tokio::time::advance(Duration::from_millis(16)).await;
+                if cancelled {
+                    cancel.cancel();
+                }
+                let expected = if cancelled {
+                    NetworkPeerProbeError::Cancelled
+                } else {
+                    NetworkPeerProbeError::TimedOut
+                };
+                assert_eq!(sending.await, Err(expected));
+            }
+            assert_eq!(sends, 0);
+            assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 8_192);
+        }
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn cancellation_during_a_payload_rpc_retains_its_attempted_charge() {
+        let pacing = pacing(Duration::from_secs(2));
+        let cancel = CancellationToken::new();
+        let mut sends = 0;
+        {
+            let mut sending = std::pin::pin!(send_payload(16_384, Some(&pacing), &cancel, &mut sends, |sends, _| Box::pin(
+                async move {
+                    *sends += 1;
+                    std::future::pending().await
+                }
+            )));
+            assert!(futures::poll!(&mut sending).is_pending());
+            tokio::time::advance(Duration::from_millis(16)).await;
+            assert!(futures::poll!(&mut sending).is_pending());
+            cancel.cancel();
+            assert_eq!(sending.await, Err(NetworkPeerProbeError::Cancelled));
+        }
+        assert_eq!(sends, 1);
+        assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 16_384);
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn default_payload_send_remains_one_unpaced_rpc() {
+        let started = Instant::now();
+        let mut sends = 0;
+        send_payload(32_768, None, &CancellationToken::new(), &mut sends, move |sends, payload| {
+            Box::pin(async move {
+                assert_eq!(started.elapsed(), Duration::ZERO);
+                assert_eq!(payload.len(), 32_768);
+                *sends += 1;
+                Ok(())
+            })
+        })
+        .await
+        .expect("legacy admin send");
+        assert_eq!(sends, 1);
+        assert!(
+            NetworkPeerProbeClient::from_endpoint_pools(&topology())
+                .diagnostic_pacing
+                .is_none()
+        );
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn diagnostic_payload_is_charged_before_each_actual_send_poll() {
+        let started = tokio::time::Instant::now();
+        let pacing = DiagnosticPacing {
+            started,
+            deadline: started + Duration::from_secs(2),
+            charged_bytes: AtomicU64::new(0),
+        };
+        let mut attempted = 0_u64;
+        send_payload(
+            32_768,
+            Some(&pacing),
+            &CancellationToken::new(),
+            &mut attempted,
+            move |attempted, payload| {
+                Box::pin(async move {
+                    *attempted += u64::try_from(payload.len()).expect("bounded payload length");
+                    let available = u128::from(MAX_NETWORK_PROBE_BYTES) * started.elapsed().as_millis() / 1_000;
+                    assert!(
+                        u128::from(*attempted) <= available,
+                        "send boundary emitted {attempted} bytes with only {available} bytes earned"
+                    );
+                    Ok(())
+                })
+            },
+        )
+        .await
+        .expect("bounded diagnostic payload");
+        assert_eq!(attempted, 32_768);
+    }
+
    fn endpoint(value: &str, is_local: bool) -> Endpoint {
        let mut endpoint = Endpoint::try_from(value).expect("test endpoint should parse");
        endpoint.is_local = is_local;
@@ -230,6 +560,14 @@ mod tests {
        }])
    }

+    #[test]
+    fn probe_future_remains_send() {
+        fn require_send<T: Send>(_: T) {}
+        let client = NetworkPeerProbeClient::from_endpoint_pools(&topology());
+        let cancel = CancellationToken::new();
+        require_send(client.probe("peer-1", 1, Duration::from_secs(1), &cancel));
+    }
+
    #[test]
    fn targets_are_unique_sorted_remote_nodes_with_stable_aliases() {
        let client = NetworkPeerProbeClient::from_endpoint_pools(&topology());
@@ -993,7 +993,19 @@ impl RemoteDisk {
        // internode transport failure classified as retryable can be safely re-dialed. The
        // classifier is direction-agnostic — it inspects the InternodeHttpError kind — so it is
        // reused here from the write path.
-        err.is_retryable_internode_write_failure()
+        if err.is_retryable_internode_write_failure() {
+            return true;
+        }
+        // Header wait uses the body stall budget. That error is `BodyStalled`, not an
+        // `InternodeHttpError`, and the open has not consumed a shard byte yet.
+        matches!(
+            err,
+            DiskError::Io(error)
+                if error
+                    .get_ref()
+                    .and_then(|source| source.downcast_ref::<rustfs_rio::BodyStalled>())
+                    .is_some()
+        )
    }

    pub(crate) async fn new(ep: &Endpoint, opt: &DiskOption, data_transport: Arc<dyn InternodeDataTransport>) -> Result<Self> {
@@ -1102,7 +1114,14 @@ impl RemoteDisk {
        let mut attempt = 1;
        let mut last_retry_classification = None;
        loop {
-            match self.data_transport.open_read(request.clone()).await {
+            // The second attempt bypasses the pool. The first failure is often a
+            // stale kept-alive connection to a peer that just restarted.
+            let opened = if attempt == 1 {
+                self.data_transport.open_read(request.clone()).await
+            } else {
+                self.data_transport.open_read_fresh(request.clone()).await
+            };
+            match opened {
                Ok(reader) => {
                    if attempt > 1
                        && let Some(classification) = last_retry_classification
@@ -1138,7 +1157,12 @@ impl RemoteDisk {
        let mut attempt = 1;
        let mut last_retry_classification = None;
        loop {
-            match self.data_transport.open_read_chunks(request.clone()).await {
+            let opened = if attempt == 1 {
+                self.data_transport.open_read_chunks(request.clone()).await
+            } else {
+                self.data_transport.open_read_chunks_fresh(request.clone()).await
+            };
+            match opened {
                Ok(reader) => {
                    if attempt > 1
                        && let Some(classification) = last_retry_classification
@@ -1220,13 +1244,14 @@ impl RemoteDisk {
        {
            return;
        }
+        // Own the flag before spawning: shutdown can drop the task without polling it.
+        let lease = RecoveryMonitorLease {
+            active: Arc::clone(&active),
+        };
        let span = Self::recovery_monitor_span(&addr, &endpoint, handle_id);
        super::spawn_background_monitor(span, async move {
            #[cfg(test)]
            test_state.start_count.fetch_add(1, Ordering::AcqRel);
-            let lease = RecoveryMonitorLease {
-                active: Arc::clone(&active),
-            };
            Self::monitor_remote_disk_recovery(addr.clone(), endpoint.clone(), Arc::clone(&health), cancel_token.clone()).await;
            #[cfg(test)]
            if let Some(hook) = test_state.teardown_hook.lock().await.take() {
@@ -1532,7 +1557,8 @@ impl RemoteDisk {
        };

        if evict_cached_connection {
-            evict_failed_connection(addr).await;
+            // Cache contention must not hold the recovery lease indefinitely.
+            let _ = timeout(get_drive_active_check_timeout(), evict_failed_connection(addr)).await;
        }

        result
@@ -1691,7 +1717,7 @@ impl RemoteDisk {
        if timeout_duration == Duration::ZERO {
            let operation_result = operation().await;
            if operation_result.is_ok() {
-                self.health.log_success();
+                self.health.record_operation_success(&self.endpoint, "operation_success");
            }
            self.handle_network_like_error(op, timeout_duration, &operation_result, failure_health_action)
                .await;
@@ -1705,7 +1731,7 @@ impl RemoteDisk {
            Ok(operation_result) => {
                // Log success; the waiting guard balances every exit path.
                if operation_result.is_ok() {
-                    self.health.log_success();
+                    self.health.record_operation_success(&self.endpoint, "operation_success");
                }
                self.handle_network_like_error(op, timeout_duration, &operation_result, failure_health_action)
                    .await;
@@ -7729,6 +7755,25 @@ mod tests {
        assert_eq!(transport.calls().len(), 2, "read_file_stream should retry exactly once");
    }

+    #[tokio::test]
+    async fn test_remote_disk_read_file_stream_retries_header_stall_on_fresh_connection() {
+        let stall = rustfs_rio::BodyStalled {
+            timeout: Duration::from_millis(50),
+        };
+        let transport = RetryingOpenReadInternodeDataTransport::with_steps(vec![
+            OpenWriteTestStep::Error(DiskError::Io(std::io::Error::new(std::io::ErrorKind::TimedOut, stall))),
+            OpenWriteTestStep::Success,
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+
+        let _reader = remote_disk
+            .read_file_stream("bucket", "object/part.1", 0, 4096)
+            .await
+            .expect("a header stall must be retried once before the shard is failed");
+
+        assert_eq!(transport.calls().len(), 2, "header stall should re-dial exactly once");
+    }
+
    #[tokio::test]
    async fn test_remote_disk_read_file_stream_does_not_retry_non_retryable_open_read_error() {
        let transport = RetryingOpenReadInternodeDataTransport::with_steps(vec![OpenWriteTestStep::Error(DiskError::from(
@@ -8161,6 +8206,97 @@ mod tests {
        assert!(result.is_ok());
    }

+    #[tokio::test]
+    async fn successful_operation_recovers_suspect_remote_disk_without_monitor() {
+        let endpoint = Endpoint {
+            url: url::Url::parse("http://remote-recovery:9000/data").expect("valid endpoint"),
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+        let disk = RemoteDisk::new(
+            &endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+            Arc::new(TcpHttpInternodeDataTransport),
+        )
+        .await
+        .expect("create remote disk");
+
+        for duration in [Duration::ZERO, Duration::from_secs(1)] {
+            disk.health.mark_failure(&endpoint, "read_operation_deadline");
+            assert_eq!(disk.runtime_state(), RuntimeDriveHealthState::Suspect);
+            assert!(!disk.recovery_monitor_is_active());
+            let result = disk
+                .execute_with_timeout(|| async { Err::<(), _>(DiskError::FileNotFound) }, duration)
+                .await;
+            assert!(matches!(result, Err(DiskError::FileNotFound)));
+            assert_eq!(
+                disk.runtime_state(),
+                RuntimeDriveHealthState::Suspect,
+                "a failed operation must not restore readiness"
+            );
+
+            disk.execute_with_timeout(|| async { Ok(()) }, duration)
+                .await
+                .expect("successful disk RPC");
+            assert_eq!(
+                disk.runtime_state(),
+                RuntimeDriveHealthState::Online,
+                "successful traffic must restore readiness without a recovery monitor"
+            );
+            assert!(disk.offline_duration_secs().is_none());
+            assert_eq!(disk.health.waiting.load(Ordering::Acquire), 0);
+        }
+
+        disk.health.mark_offline(&endpoint, "test_offline");
+        let result = disk.execute_with_timeout(|| async { Ok(()) }, Duration::from_secs(1)).await;
+        assert!(
+            matches!(result, Err(DiskError::FaultyDisk)),
+            "offline handles still require recovery probes before data I/O"
+        );
+        assert_eq!(disk.runtime_state(), RuntimeDriveHealthState::Offline);
+    }
+
+    #[test]
+    fn recovery_monitor_releases_lease_when_dropped_before_first_poll() {
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("create test runtime");
+        let endpoint = Endpoint {
+            url: url::Url::parse("http://remote-unpolled:9000/data").expect("valid endpoint"),
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+        let active = Arc::new(AtomicBool::new(false));
+        let start_count = Arc::new(AtomicU32::new(0));
+        {
+            let _guard = runtime.enter();
+            RemoteDisk::schedule_recovery_monitor(
+                "http://remote-unpolled:9000".to_string(),
+                endpoint,
+                Uuid::new_v4(),
+                Arc::new(DiskHealthTracker::new()),
+                CancellationToken::new(),
+                Arc::clone(&active),
+                RecoveryMonitorTestState {
+                    start_count: Arc::clone(&start_count),
+                    teardown_hook: Arc::new(tokio::sync::Mutex::new(None)),
+                },
+            );
+            assert!(active.load(Ordering::Acquire));
+        }
+        drop(runtime);
+        assert_eq!(start_count.load(Ordering::Acquire), 0, "task must never be polled");
+        assert!(!active.load(Ordering::Acquire), "dropping an unpolled monitor must release its lease");
+    }
+
    #[tokio::test]
    async fn test_execute_with_timeout_marks_remote_disk_faulty() {
        let url = url::Url::parse("http://remote-timeout:9000").expect("operation should succeed");
@@ -3154,6 +3154,7 @@ mod tests {
                version_purge_status: Default::default(),
                replication_decision: String::new(),
                checksum: None,
+                multipart_completion_replayed: false,
            }
        }
    }
@@ -12976,7 +12976,7 @@ impl ECStore {
                .targets
                .iter()
                .find(|target| target.pool_index == permitted_target_pool_index)
-                .and_then(&candidate)
+                .and_then(candidate)
            {
                return Ok(pool_index);
            }
@@ -19542,7 +19542,10 @@ mod tests {
            .await
            .expect("second-node activation should not panic")
            .expect_err("the second reservation must observe and reject the committed first reservation");
-        assert!(err.to_string().contains("requires 60 bytes, but 40 bytes are available"));
+        assert!(
+            err.to_string().contains("requires 60 bytes, but 40 bytes are available"),
+            "the second reservation must report the committed capacity rejection, got {err:?}"
+        );

        let mut persisted = PoolMeta::default();
        persisted
@@ -19855,9 +19858,9 @@ mod tests {
                .reload_pool_meta()
                .await
                .expect("load the active generation on the remote node");
-            other_store
-                .decommission_cancel(0)
+            tokio::time::timeout(std::time::Duration::from_secs(30), other_store.decommission_cancel(0))
                .await
+                .expect("remote cancel must persist pool metadata without re-entering its state locks")
                .expect("the remote node should cancel the old generation");
            other_store
                .clear_decommission(0)
@@ -20122,9 +20125,9 @@ mod tests {
            .expect("the stale operation should have a valid terminal fence plan")
            .expect("the stale operation should have an active reservation");

-        first_node
-            .decommission_cancel(0)
+        tokio::time::timeout(std::time::Duration::from_secs(30), first_node.decommission_cancel(0))
            .await
+            .expect("cancel must persist pool metadata without re-entering its state locks")
            .expect("cancel the first durable operation");
        first_node
            .clear_decommission(0)
@@ -20627,8 +20630,9 @@ mod tests {
            .await
            .expect("second target mutation task should not panic")
            .expect("second target mutation should finalize");
-        cancel
+        tokio::time::timeout(std::time::Duration::from_secs(30), cancel)
            .await
+            .expect("cancel must finish after target mutations release their gates")
            .expect("decommission cancellation task should not panic")
            .expect("decommission cancellation should finish after target mutations");

@@ -2288,7 +2288,7 @@ mod decommission_lock_order_tests {
        let (lossy_store, refresh_calls) = store_with_capacity_lease_loss(&other_store).await;
        set_decommission_capacity_info_overrides_for_test(lossy_store.id, (0..40).map(|_| capacity_snapshot()).collect());
        let barrier = MultipartCommitBarrier::install(&bucket, object, MultipartCommitPause::AfterObjectPublication);
-        let migration = tokio::spawn({
+        let mut migration = tokio::spawn({
            let migration_store = Arc::clone(&lossy_store);
            let migration_bucket = bucket.clone();
            async move {
@@ -2304,7 +2304,12 @@ mod decommission_lock_order_tests {
                .await
            }
        });
-        barrier.wait_until_paused().await;
+        tokio::select! {
+            () = barrier.wait_until_paused() => {}
+            result = &mut migration => {
+                panic!("initial multipart migration completed before reaching the publication barrier: {result:?}");
+            }
+        }
        tokio::time::pause();
        let published = lossy_store.pools[2]
            .get_object_info(
@@ -2321,6 +2326,18 @@ mod decommission_lock_order_tests {
            .expect("the multipart target must publish before capacity progress save");
        assert!(published.is_multipart(), "published target must retain multipart identity");
        assert_eq!(published.version_id.map(|version| version.to_string()), Some(source_version.clone()));
+        for prefix in [
+            rustfs_utils::http::RUSTFS_INTERNAL_PREFIX,
+            rustfs_utils::http::MINIO_INTERNAL_PREFIX,
+        ] {
+            assert_eq!(
+                published
+                    .user_defined
+                    .get(&format!("{prefix}{}", rustfs_utils::http::SUFFIX_MULTIPART_UPLOAD_ID)),
+                Some(&source_upload.upload_id),
+                "migration must preserve the source completion identity under both internal prefixes"
+            );
+        }
        let mut published_reader = lossy_store.pools[2]
            .get_object_reader(
                &bucket,
@@ -6851,7 +6868,7 @@ mod decommission_lock_order_tests {
            let migration_store = Arc::clone(&store);
            let migration_bucket = bucket.clone();
            async move {
-                data_movement::migrate_decommission_object(
+                let result = data_movement::migrate_decommission_object(
                    migration_store,
                    0,
                    migration_bucket,
@@ -6860,16 +6877,19 @@ mod decommission_lock_order_tests {
                    "decommission_object_lock_order",
                    Some(owner),
                )
-                .await
+                .await;
+                (result, std::time::Instant::now())
            }
        });
        barrier.wait_until_owner_paused().await;

+        let ordinary_launched_at = std::time::Instant::now();
        let mut ordinary_mutation = tokio::spawn({
            let ordinary_store = Arc::clone(&other_store);
            let ordinary_bucket = bucket.clone();
            async move {
-                match mutation {
+                let ordinary_started_at = std::time::Instant::now();
+                let result = match mutation {
                    ExternalObjectMutation::Put => {
                        let mut data = PutObjReader::from_vec(b"ordinary replacement".to_vec());
                        ordinary_store
@@ -6941,7 +6961,8 @@ mod decommission_lock_order_tests {
                            .restore_transitioned_object(&ordinary_bucket, object, &opts)
                            .await
                    }
-                }
+                };
+                (result, ordinary_started_at, std::time::Instant::now())
            }
        });
        if let Some(get_barrier) = restore_get_barrier.as_ref() {
@@ -6989,8 +7010,10 @@ mod decommission_lock_order_tests {
        } else {
            barrier.wait_until_external_capacity_released().await;
        }
+        let ordinary_lock_phase_observed_at = std::time::Instant::now();
        assert!(!ordinary_mutation.is_finished());
-        let migration_result = if pause_restore_tail {
+        let owner_released_at = std::time::Instant::now();
+        let (migration_result, migration_finished_at) = if pause_restore_tail {
            barrier.release_owner();
            tokio::time::timeout(std::time::Duration::from_secs(30), &mut migration)
                .await
@@ -7005,7 +7028,7 @@ mod decommission_lock_order_tests {
        };
        migration_result.expect("migration should commit to its reserved target");

-        let ordinary_result = if pause_restore_tail {
+        let (ordinary_result, ordinary_started_at, ordinary_finished_at) = if pause_restore_tail {
            let handoff_barrier = PutObjectCommitBarrier::install(&bucket, object, PutObjectCommitPause::AfterRenameHandoff);
            let tail_barrier =
                crate::set_disk::rename_fanout_barrier::arm(object, 0, crate::set_disk::rename_fanout_barrier::PHASE_RENAME);
@@ -7077,7 +7100,20 @@ mod decommission_lock_order_tests {
                .expect("ordinary object mutation task should join")
        };
        drop(barrier);
-        ordinary_result.expect("ordinary object mutation should commit to the unreserved target");
+        ordinary_result.unwrap_or_else(|error| {
+            panic!(
+                "ordinary object mutation should commit to the unreserved target: {error:?}; \
+                 ordinary_start_after_launch={:?}, lock_phase_observed_after_launch={:?}, \
+                 owner_release_after_launch={:?}, migration_after_owner_release={:?}, \
+                 ordinary_finish_after_launch={:?}, ordinary_finished_before_migration={}",
+                ordinary_started_at.duration_since(ordinary_launched_at),
+                ordinary_lock_phase_observed_at.duration_since(ordinary_launched_at),
+                owner_released_at.duration_since(ordinary_launched_at),
+                migration_finished_at.duration_since(owner_released_at),
+                ordinary_finished_at.duration_since(ordinary_launched_at),
+                ordinary_finished_at < migration_finished_at,
+            );
+        });
        store.pools[1]
            .get_object_info(
                &bucket,
@@ -7252,7 +7288,7 @@ mod decommission_lock_order_tests {
            let migration_store = Arc::clone(&store);
            let migration_bucket = bucket.clone();
            async move {
-                data_movement::migrate_decommission_object(
+                let result = data_movement::migrate_decommission_object(
                    migration_store,
                    0,
                    migration_bucket,
@@ -7261,11 +7297,13 @@ mod decommission_lock_order_tests {
                    "same_object_copy_lock_order",
                    Some(owner),
                )
-                .await
+                .await;
+                (result, std::time::Instant::now())
            }
        });
        barrier.wait_until_owner_paused().await;

+        let ordinary_launched_at = std::time::Instant::now();
        let copy_task = tokio::spawn({
            let copy_store = Arc::clone(&other_store);
            let copy_bucket = bucket.clone();
@@ -7280,7 +7318,8 @@ mod decommission_lock_order_tests {
                ..Default::default()
            };
            async move {
-                copy_store
+                let ordinary_started_at = std::time::Instant::now();
+                let result = copy_store
                    .copy_object(
                        &copy_bucket,
                        object,
@@ -7291,27 +7330,44 @@ mod decommission_lock_order_tests {
                        &destination_opts,
                    )
                    .await
-                    .map(|_| ())
+                    .map(|_| ());
+                (result, ordinary_started_at, std::time::Instant::now())
            }
        });
        barrier.wait_until_external_capacity_released().await;
+        let ordinary_lock_phase_observed_at = std::time::Instant::now();
        assert!(
            !copy_task.is_finished(),
            "historical same-object CopyObject must wait behind the migration object fence"
        );
+        let owner_released_at = std::time::Instant::now();
        barrier.release_owner();
        drop(barrier);

-        tokio::time::timeout(std::time::Duration::from_secs(30), migration)
+        let (migration_result, migration_finished_at) = tokio::time::timeout(std::time::Duration::from_secs(30), migration)
            .await
            .expect("migration must not deadlock with historical same-object CopyObject")
-            .expect("migration task should join")
-            .expect("migration should commit to its reserved target");
-        tokio::time::timeout(std::time::Duration::from_secs(30), copy_task)
-            .await
-            .expect("historical same-object CopyObject must finish after the migration releases its object fence")
-            .expect("CopyObject task should join")
-            .expect("CopyObject should preserve the expected current-version precondition");
+            .expect("migration task should join");
+        migration_result.expect("migration should commit to its reserved target");
+        let (ordinary_result, ordinary_started_at, ordinary_finished_at) =
+            tokio::time::timeout(std::time::Duration::from_secs(30), copy_task)
+                .await
+                .expect("historical same-object CopyObject must finish after the migration releases its object fence")
+                .expect("CopyObject task should join");
+        ordinary_result.unwrap_or_else(|error| {
+            panic!(
+                "CopyObject should preserve the expected current-version precondition: {error:?}; \
+                 ordinary_start_after_launch={:?}, lock_phase_observed_after_launch={:?}, \
+                 owner_release_after_launch={:?}, migration_after_owner_release={:?}, \
+                 ordinary_finish_after_launch={:?}, ordinary_finished_before_migration={}",
+                ordinary_started_at.duration_since(ordinary_launched_at),
+                ordinary_lock_phase_observed_at.duration_since(ordinary_launched_at),
+                owner_released_at.duration_since(ordinary_launched_at),
+                migration_finished_at.duration_since(owner_released_at),
+                ordinary_finished_at.duration_since(ordinary_launched_at),
+                ordinary_finished_at < migration_finished_at,
+            );
+        });
    }

    #[tokio::test]
@@ -306,7 +306,7 @@ pub mod test_util {
        let Some(fault) = fault else { return Ok(None) };
        if fault
            .remaining
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |remaining| remaining.checked_sub(1))
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |remaining| remaining.checked_sub(1))
            != Ok(1)
        {
            return Ok(None);
@@ -43,7 +43,7 @@ use std::{
    future::Future,
    sync::{
        Arc, LazyLock, OnceLock,
-        atomic::{AtomicU64, Ordering},
+        atomic::{AtomicU64, AtomicUsize, Ordering},
    },
    time::{Duration, SystemTime},
 };
@@ -67,12 +67,14 @@ struct CachedBucketUsage {
    usage_updated_at: SystemTime,
    // Set by request-path mutations until a scanner snapshot catches up to the same core counts.
    dirty: bool,
-    // Set when a newer scanner snapshot was observed but did not include the dirty counts yet.
+    // Set when an observed snapshot cannot replace this absolute baseline.
    stale_snapshot_pending: bool,
    // First complete scanner generation observed after an unknown-baseline
    // mutation. A strictly later generation is required before the mutation
    // evidence can be discarded.
    pending_scanner_position: Option<(u64, u64)>,
+    // Generation that supplied this absolute baseline, retained across mutations.
+    scanner_position: Option<(u64, u64)>,
    // Deletes are visible to admin immediately, but quota admission keeps
    // them pending until a complete scanner generation reconciles the set.
    // This marker intentionally remains process-local: the delete request
@@ -83,6 +85,79 @@ struct CachedBucketUsage {
    // necessarily a complete scanner reconciliation and therefore creates a
    // fresh cache entry with no pending hold.
    pending_negative_delta: u64,
+    pending_negative_updated_at: Option<SystemTime>,
+    pending_negative_reconciliation: Option<PendingDeleteReconciliation>,
+    in_flight_expirations: Arc<AtomicUsize>,
+}
+
+#[derive(Debug, Clone)]
+struct PendingDeleteReconciliation {
+    bytes: u64,
+    deleted_at: SystemTime,
+    observed: Option<((u64, u64), SystemTime)>,
+}
+
+impl CachedBucketUsage {
+    fn reconcile_pending_deletes(&mut self, position: Option<(u64, u64)>, updated_at: SystemTime) {
+        let Some(position) = position else { return };
+        let Some(pending) = self.pending_negative_reconciliation.as_mut() else { return };
+        match pending.observed {
+            None if updated_at >= pending.deleted_at => pending.observed = Some((position, updated_at)),
+            Some((previous, observed_at)) if position.0 >= previous.0 && position.1 > previous.1 && updated_at >= observed_at => {
+                self.pending_negative_delta = self.pending_negative_delta.saturating_sub(pending.bytes);
+                self.pending_negative_reconciliation = self
+                    .pending_negative_updated_at
+                    .filter(|_| self.pending_negative_delta > 0)
+                    .map(|deleted_at| PendingDeleteReconciliation {
+                        bytes: self.pending_negative_delta,
+                        deleted_at,
+                        observed: (updated_at >= deleted_at).then_some((position, updated_at)),
+                    });
+            }
+            _ => {}
+        }
+    }
+}
+
+/// Keeps a scanner refresh from including a delete before its memory receipt.
+pub(crate) struct ExpiryUsageAccounting<'a> {
+    bucket: &'a str,
+    in_flight: Arc<AtomicUsize>,
+}
+
+impl ExpiryUsageAccounting<'_> {
+    pub(crate) async fn commit(self, deleted_size: u64) {
+        let mut cache = memory_cache().write().await;
+        if let Some(entry) = cache.get_mut(self.bucket)
+            && Arc::ptr_eq(&entry.in_flight_expirations, &self.in_flight)
+        {
+            apply_bucket_object_delete_memory(entry, deleted_size, true);
+        }
+    }
+}
+
+impl Drop for ExpiryUsageAccounting<'_> {
+    fn drop(&mut self) {
+        self.in_flight.fetch_sub(1, Ordering::AcqRel);
+    }
+}
+
+pub(crate) async fn begin_expiry_usage_accounting(bucket: &str) -> ExpiryUsageAccounting<'_> {
+    ensure_bucket_usage_cached(bucket).await;
+    let mut cache = memory_cache().write().await;
+    // The caller owns the bucket name through commit. Avoid allocating either
+    // a receipt name or a map key for every expiry in an already cached bucket.
+    let in_flight = match cache.get(bucket) {
+        Some(entry) => Arc::clone(&entry.in_flight_expirations),
+        None => {
+            let entry = cached_bucket_usage_now(BucketUsageInfo::default());
+            let in_flight = Arc::clone(&entry.in_flight_expirations);
+            cache.insert(bucket.to_owned(), entry);
+            in_flight
+        }
+    };
+    in_flight.fetch_add(1, Ordering::AcqRel);
+    ExpiryUsageAccounting { bucket, in_flight }
 }

 type UsageMemoryCache = Arc<RwLock<HashMap<String, CachedBucketUsage>>>;
@@ -1944,7 +2019,11 @@ fn cached_bucket_usage_from_backend(usage: BucketUsageInfo, updated_at: SystemTi
        dirty: false,
        stale_snapshot_pending: false,
        pending_scanner_position: None,
+        scanner_position: None,
        pending_negative_delta: 0,
+        pending_negative_updated_at: None,
+        pending_negative_reconciliation: None,
+        in_flight_expirations: Arc::new(AtomicUsize::new(0)),
    }
 }

@@ -1958,7 +2037,11 @@ fn cached_bucket_usage_now(usage: BucketUsageInfo) -> CachedBucketUsage {
        dirty: false,
        stale_snapshot_pending: false,
        pending_scanner_position: None,
+        scanner_position: None,
        pending_negative_delta: 0,
+        pending_negative_updated_at: None,
+        pending_negative_reconciliation: None,
+        in_flight_expirations: Arc::new(AtomicUsize::new(0)),
    }
 }

@@ -1974,27 +2057,26 @@ fn bucket_usage_counts_match(left: &BucketUsageInfo, right: &BucketUsageInfo) ->
 }

 fn preserve_unknown_dirty_usage(
-    existing: &CachedBucketUsage,
+    existing: &mut CachedBucketUsage,
    snapshot_position: Option<(u64, u64)>,
    snapshot_update: SystemTime,
-) -> Option<CachedBucketUsage> {
+) -> bool {
    if existing.authoritative || !existing.dirty {
-        return None;
+        return false;
    }
    if existing
        .pending_scanner_position
        .zip(snapshot_position)
        .is_some_and(|(previous, current)| current.0 > previous.0 || (current.0 == previous.0 && current.1 > previous.1))
    {
-        return None;
+        return false;
    }

-    let mut preserved = existing.clone();
-    preserved.stale_snapshot_pending = true;
-    if preserved.pending_scanner_position.is_none() && snapshot_update >= existing.usage_updated_at {
-        preserved.pending_scanner_position = snapshot_position;
+    existing.stale_snapshot_pending = true;
+    if existing.pending_scanner_position.is_none() && snapshot_update >= existing.usage_updated_at {
+        existing.pending_scanner_position = snapshot_position;
    }
-    Some(preserved)
+    true
 }

 #[cfg(any(test, feature = "test-util"))]
@@ -2110,6 +2192,10 @@ pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64,
        .entry(bucket.to_string())
        .or_insert_with(|| cached_bucket_usage_now(BucketUsageInfo::default()));

+    apply_bucket_object_delete_memory(entry, deleted_size, removed_current_object);
+}
+
+fn apply_bucket_object_delete_memory(entry: &mut CachedBucketUsage, deleted_size: u64, removed_current_object: bool) {
    entry.usage.size = entry.usage.size.saturating_sub(deleted_size);
    entry.pending_negative_delta = entry.pending_negative_delta.saturating_add(deleted_size);
    if removed_current_object {
@@ -2118,6 +2204,16 @@ pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64,
    }

    let now = SystemTime::now();
+    entry.pending_negative_updated_at = Some(now);
+    // Freeze this tranche even before its first observation. Later deletes
+    // must not keep moving the timestamp that the scanner needs to catch up to.
+    if entry.pending_negative_reconciliation.is_none() && entry.pending_negative_delta > 0 {
+        entry.pending_negative_reconciliation = Some(PendingDeleteReconciliation {
+            bytes: entry.pending_negative_delta,
+            deleted_at: now,
+            observed: None,
+        });
+    }
    entry.refreshed_at = now;
    entry.usage_updated_at = now;
    entry.dirty = true;
@@ -2245,23 +2341,48 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D

    let usage_updated_at = data_usage_info_updated_at(data_usage_info);
    let snapshot_position = data_usage_info.scanner_epoch.zip(data_usage_info.scanner_cycle);
+    let snapshot_converged = data_usage_info.usage_snapshot_converged == Some(true) && !data_usage_info.usage_snapshot_partial;
    let mut next_cache = HashMap::with_capacity(data_usage_info.buckets_usage.len());
    for (bucket, bucket_usage) in data_usage_info.buckets_usage.iter() {
-        next_cache.insert(
-            bucket.clone(),
-            cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at, true),
-        );
+        let mut entry = cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at, true);
+        entry.scanner_position = snapshot_position;
+        next_cache.insert(bucket.clone(), entry);
    }

    let mut cache = memory_cache().write().await;
    if expected_generation.is_some_and(|expected| usage_memory_generation() != expected) {
        return;
    }
-    for (bucket, existing) in cache.iter() {
-        match next_cache.entry(bucket.clone()) {
+    // Move preserved entries instead of cloning their histograms under the lock.
+    for (bucket, mut existing) in std::mem::take(&mut *cache) {
+        let advances_baseline = match (existing.scanner_position, snapshot_position) {
+            (Some(previous), Some(current)) if current.0 >= previous.0 && current.1 >= previous.1 => current.1 > previous.1,
+            (None, _) => true,
+            _ => {
+                existing.stale_snapshot_pending = true;
+                next_cache.insert(bucket, existing);
+                continue;
+            }
+        };
+        existing.reconcile_pending_deletes(snapshot_position, usage_updated_at);
+        if existing.in_flight_expirations.load(Ordering::Acquire) > 0 {
+            next_cache.insert(bucket, existing);
+            continue;
+        }
+        // Both convergence and the two-observation fallback must advance the
+        // baseline's cycle. Epoch-only changes cannot prove a new scan, and
+        // the timestamp check below still protects post-scan mutations.
+        let reconciled_dirty_usage = advances_baseline
+            && snapshot_position.is_some_and(|current| {
+                snapshot_converged
+                    || existing
+                        .pending_scanner_position
+                        .is_some_and(|previous| current.0 >= previous.0 && current.1 > previous.1)
+            });
+        match next_cache.entry(bucket) {
            Entry::Occupied(mut candidate) => {
-                if let Some(preserved) = preserve_unknown_dirty_usage(existing, snapshot_position, usage_updated_at) {
-                    candidate.insert(preserved);
+                if preserve_unknown_dirty_usage(&mut existing, snapshot_position, usage_updated_at) {
+                    candidate.insert(existing);
                    continue;
                }
                if existing.authoritative && existing.dirty && bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
@@ -2272,37 +2393,44 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
                    continue;
                }
                if existing.authoritative && existing.usage_updated_at > usage_updated_at {
-                    candidate.insert(existing.clone());
+                    candidate.insert(existing);
                    continue;
                }
-                if existing.authoritative && existing.dirty && !bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
+                if existing.authoritative
+                    && existing.dirty
+                    && !reconciled_dirty_usage
+                    && !bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
                {
                    // A scanner snapshot can be saved after newer writes but still miss them if it listed the bucket earlier.
-                    let mut preserved = existing.clone();
+                    let mut preserved = existing;
                    if preserved.pending_negative_delta > 0 && preserved.usage.size == candidate.get().usage.size {
                        // The complete scanner snapshot has reconciled the
                        // post-delete byte total. Keep the request-path object
                        // counts until they also converge, but release the
                        // conservative quota hold.
                        preserved.pending_negative_delta = 0;
+                        preserved.pending_negative_reconciliation = None;
                    }
                    preserved.stale_snapshot_pending = true;
+                    preserved.pending_scanner_position = preserved.pending_scanner_position.or(snapshot_position);
                    candidate.insert(preserved);
                }
            }
            Entry::Vacant(candidate) => {
-                if let Some(preserved) = preserve_unknown_dirty_usage(existing, snapshot_position, usage_updated_at) {
-                    candidate.insert(preserved);
-                } else if existing.authoritative && existing.usage_updated_at > usage_updated_at {
-                    candidate.insert(existing.clone());
-                } else if existing.authoritative && existing.dirty {
-                    let mut preserved = existing.clone();
+                if preserve_unknown_dirty_usage(&mut existing, snapshot_position, usage_updated_at)
+                    || (existing.authoritative && existing.usage_updated_at > usage_updated_at)
+                {
+                    candidate.insert(existing);
+                } else if existing.authoritative && existing.dirty && !reconciled_dirty_usage {
+                    let mut preserved = existing;
                    if preserved.pending_negative_delta > 0 && preserved.usage.size == 0 {
                        // Omitting the bucket from a complete snapshot confirms
                        // that its post-delete byte total is zero.
                        preserved.pending_negative_delta = 0;
+                        preserved.pending_negative_reconciliation = None;
                    }
                    preserved.stale_snapshot_pending = true;
+                    preserved.pending_scanner_position = preserved.pending_scanner_position.or(snapshot_position);
                    candidate.insert(preserved);
                }
            }
@@ -4352,6 +4480,7 @@ mod tests {
        let mut first_snapshot = data_usage_info_for_test(bucket, 10, 420, mutation_update + Duration::from_nanos(1));
        first_snapshot.scanner_epoch = Some(7);
        first_snapshot.scanner_cycle = Some(10);
+        first_snapshot.usage_snapshot_converged = Some(true);
        replace_bucket_usage_memory_from_info(&first_snapshot).await;

        let mut first_response = first_snapshot.clone();
@@ -5405,11 +5534,12 @@ mod tests {
            .expect("delete usage should remain cached")
            .usage_updated_at;

-        let stale = data_usage_info_for_test("bucket-a", 2, 768, mutation_update + Duration::from_nanos(1));
+        // Windows SystemTime has 100 ns resolution; retain strict ordering there.
+        let stale = data_usage_info_for_test("bucket-a", 2, 768, mutation_update + Duration::from_micros(1));
        replace_bucket_usage_memory_from_info(&stale).await;
        assert_eq!(get_bucket_usage_memory("bucket-a").await, Some(768));

-        let mut too_old_matching = data_usage_info_for_test("bucket-a", 1, 256, mutation_update - Duration::from_nanos(1));
+        let mut too_old_matching = data_usage_info_for_test("bucket-a", 1, 256, mutation_update - Duration::from_micros(1));
        too_old_matching
            .buckets_usage
            .get_mut("bucket-a")
@@ -5423,7 +5553,7 @@ mod tests {
            "a matching byte total from before the delete cannot release the hold"
        );

-        let mut reconciled = data_usage_info_for_test("bucket-a", 1, 256, mutation_update + Duration::from_nanos(2));
+        let mut reconciled = data_usage_info_for_test("bucket-a", 1, 256, mutation_update + Duration::from_micros(2));
        reconciled
            .buckets_usage
            .get_mut("bucket-a")
@@ -5759,6 +5889,503 @@ mod tests {
        );
    }

+    #[tokio::test]
+    #[serial]
+    async fn complete_scanner_cycles_reconcile_expired_dirty_usage() {
+        for (remaining_objects, omit_bucket) in [(0, false), (1, false), (0, true)] {
+            clear_usage_memory_cache_for_test().await;
+            let bucket = "expired-dirty-usage";
+            let baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+
+            // Lifecycle deletion bypasses request-layer memory accounting.
+            // Repeated complete scans must eventually replace that dirty overlay.
+            for cycle in 1..=3 {
+                let mut snapshot = data_usage_info_for_test(bucket, remaining_objects, remaining_objects * 42, SystemTime::now());
+                snapshot.scanner_epoch = Some(7);
+                snapshot.scanner_cycle = Some(cycle);
+                if omit_bucket {
+                    snapshot.buckets_usage.clear();
+                    snapshot.bucket_sizes.clear();
+                    snapshot.buckets_count = 0;
+                    snapshot.calculate_totals();
+                }
+                replace_bucket_usage_memory_from_info(&snapshot).await;
+                if cycle == 3 {
+                    apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+                    assert_eq!(snapshot.objects_total_count, remaining_objects);
+                    assert_eq!(snapshot.objects_total_size, remaining_objects * 42);
+                    if omit_bucket {
+                        assert!(!snapshot.buckets_usage.contains_key(bucket));
+                    } else {
+                        assert_eq!(snapshot.buckets_usage[bucket].objects_count, remaining_objects);
+                        assert_eq!(snapshot.buckets_usage[bucket].size, remaining_objects * 42);
+                    }
+                }
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_receipts_exclude_refresh_until_committed_or_cancelled() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-receipt";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 2, 84, SystemTime::now())).await;
+        let first = begin_expiry_usage_accounting(bucket).await;
+        let cancelled = begin_expiry_usage_accounting(bucket).await;
+        let mut after_delete = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        after_delete.scanner_epoch = Some(7);
+        after_delete.scanner_cycle = Some(10);
+        after_delete.usage_snapshot_converged = Some(true);
+        replace_bucket_usage_memory_from_info(&after_delete).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 84);
+        first.commit(42).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 42);
+
+        let mut omitted = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        omitted.buckets_usage.clear();
+        omitted.bucket_sizes.clear();
+        omitted.buckets_count = 0;
+        omitted.calculate_totals();
+        replace_bucket_usage_memory_from_info(&omitted).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 42);
+        drop(cancelled);
+        assert_eq!(
+            memory_cache().read().await[bucket]
+                .in_flight_expirations
+                .load(Ordering::Acquire),
+            0
+        );
+
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 1, 42, SystemTime::now())).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn cancelled_expiry_receipt_reconciles_without_another_mutation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-cancelled-after-delete";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 0, 0, SystemTime::now())).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let receipt = begin_expiry_usage_accounting(bucket).await;
+
+        // Storage has deleted one object, but the task has not committed its
+        // memory receipt. Even two complete scans must not replace its baseline.
+        let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        scanned.scanner_epoch = Some(7);
+        for cycle in [10, 11] {
+            scanned.scanner_cycle = Some(cycle);
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+        }
+        drop(receipt);
+
+        for (cycle, expected) in [(12, 84), (13, 42)] {
+            scanned.scanner_cycle = Some(cycle);
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(expected));
+            let mut displayed = scanned.clone();
+            apply_bucket_usage_memory_overlay(&mut displayed).await;
+            assert_eq!(displayed.buckets_usage[bucket].size, expected);
+            assert_eq!(displayed.buckets_usage[bucket].objects_count, expected / 42);
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_restart_reloads_durable_usage_without_persisting_the_overlay() {
+        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
+
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        let bucket = "expiry-restart";
+        clear_usage_memory_cache_for_test().await;
+        let baseline = data_usage_info_for_test(bucket, 2, 84, SystemTime::now());
+        store_data_usage_in_backend(baseline.clone(), store.clone())
+            .await
+            .expect("persist the pre-expiration scanner snapshot");
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        begin_expiry_usage_accounting(bucket).await.commit(42).await;
+        let mut displayed = baseline.clone();
+        apply_bucket_usage_memory_overlay(&mut displayed).await;
+        assert_eq!(displayed.buckets_usage[bucket].size, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+
+        // Recreate only process-local usage state, then use the real backend
+        // loader. This is a restart-state regression, not a process restart E2E.
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+        let recovered = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("reload the durable pre-expiration snapshot");
+        assert_eq!(recovered.buckets_usage[bucket].size, 84);
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+
+        let confirmed = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        store_data_usage_in_backend(confirmed, store.clone())
+            .await
+            .expect("persist scanner confirmation without another object mutation");
+        let recovered = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("reload the confirmed snapshot");
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+
+        // Historical overlay drift must not contaminate the persisted scan.
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+        let mut recovered = load_data_usage_from_backend_cached(store)
+            .await
+            .expect("reload the confirmed snapshot after losing the stale overlay");
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        apply_bucket_usage_memory_overlay(&mut recovered).await;
+        assert_eq!(recovered.buckets_usage[bucket].objects_count, 1);
+        assert_eq!(recovered.buckets_usage[bucket].size, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_receipt_cannot_credit_a_replaced_bucket_cache() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-recreated";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 1, 42, SystemTime::now())).await;
+        let receipt = begin_expiry_usage_accounting(bucket).await;
+        clear_usage_memory_cache_for_test().await;
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 2, 84, SystemTime::now())).await;
+        receipt.commit(42).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.objects_count, 2);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_delete_tranches_advance_despite_newer_mutations() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-busy-quota";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 10, 420, SystemTime::now())).await;
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        let mut observation = data_usage_info_for_test(bucket, 9, 378, SystemTime::now());
+        observation.scanner_epoch = Some(7);
+        observation.scanner_cycle = Some(10);
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        replace_bucket_usage_memory_from_info(&observation).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(462));
+
+        for (epoch, cycle, partial, old_time) in [
+            (8, 10, false, false),
+            (6, 11, false, false),
+            (7, 11, true, false),
+            (7, 11, false, true),
+        ] {
+            let mut rejected = observation.clone();
+            rejected.scanner_epoch = Some(epoch);
+            rejected.scanner_cycle = Some(cycle);
+            rejected.usage_snapshot_complete = !partial;
+            rejected.usage_snapshot_partial = partial;
+            if old_time {
+                rejected.last_update = Some(SystemTime::UNIX_EPOCH);
+            }
+            replace_bucket_usage_memory_from_info(&rejected).await;
+            assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 42);
+        }
+
+        observation.last_update = Some(SystemTime::now());
+        observation.scanner_cycle = Some(11);
+        // A newer delete must stay charged, without delaying the older receipt.
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        replace_bucket_usage_memory_from_info(&observation).await;
+        assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(462));
+
+        for cycle in [12, 13] {
+            observation.last_update = Some(SystemTime::now());
+            observation.scanner_cycle = Some(cycle);
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            replace_bucket_usage_memory_from_info(&observation).await;
+        }
+        assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 0);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(504));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_delete_hold_stays_bounded_with_deletes_after_every_observation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-continuous-quota";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 10, 420, SystemTime::now())).await;
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        for cycle in 1..=12 {
+            let mut observed = data_usage_info_for_test(bucket, 9, 378, SystemTime::now());
+            observed.scanner_epoch = Some(7);
+            observed.scanner_cycle = Some(cycle);
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            record_bucket_object_delete_memory(bucket, 42, true).await;
+            replace_bucket_usage_memory_from_info(&observed).await;
+            let cache = memory_cache().read().await;
+            assert_eq!(cache[bucket].usage.size, 378);
+            assert!(
+                cache[bucket].pending_negative_delta <= 126,
+                "confirmed older deletes must leave the quota hold"
+            );
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn dirty_usage_reconciliation_requires_a_later_complete_observation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "dirty-observation-fence";
+        let baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mutation_at = memory_cache().read().await[bucket].usage_updated_at;
+        let mut first = data_usage_info_for_test(bucket, 0, 0, mutation_at);
+        first.scanner_epoch = Some(7);
+        first.scanner_cycle = Some(10);
+        let mut partial = first.clone();
+        partial.usage_snapshot_partial = true;
+        partial.usage_snapshot_complete = false;
+        replace_bucket_usage_memory_from_info(&partial).await;
+        assert_eq!(memory_cache().read().await[bucket].pending_scanner_position, None);
+        replace_bucket_usage_memory_from_info(&first).await;
+
+        for (epoch, cycle, partial, update) in [
+            (Some(7), Some(10), false, mutation_at),
+            (Some(8), Some(10), false, mutation_at),
+            (Some(6), Some(11), false, mutation_at),
+            (Some(7), Some(9), false, mutation_at),
+            (Some(7), Some(11), true, mutation_at),
+            (None, Some(11), false, mutation_at),
+            (Some(7), None, false, mutation_at),
+            // Windows `SystemTime` stores 100 ns intervals, so a 1 ns offset is a no-op there.
+            (Some(7), Some(11), false, mutation_at - Duration::from_micros(1)),
+        ] {
+            let mut snapshot = first.clone();
+            snapshot.scanner_epoch = epoch;
+            snapshot.scanner_cycle = cycle;
+            snapshot.usage_snapshot_partial = partial;
+            snapshot.usage_snapshot_complete = !partial;
+            snapshot.last_update = Some(update);
+            replace_bucket_usage_memory_from_info(&snapshot).await;
+            apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+            assert_eq!(snapshot.objects_total_count, 1, "epoch={epoch:?}, cycle={cycle:?}, partial={partial}");
+            assert_eq!(snapshot.objects_total_size, 42);
+        }
+
+        let mut successor = first;
+        successor.scanner_epoch = Some(8);
+        successor.scanner_cycle = Some(11);
+        replace_bucket_usage_memory_from_info(&successor).await;
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut successor, true).await;
+        assert_eq!(successor.objects_total_count, 0);
+        assert_eq!(successor.objects_total_size, 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn a_new_authoritative_mutation_restarts_reconciliation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "dirty-mutation-fence";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 0, 0, SystemTime::now())).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mut snapshot = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        snapshot.scanner_epoch = Some(7);
+        snapshot.scanner_cycle = Some(10);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        snapshot.last_update = Some(SystemTime::now());
+        snapshot.scanner_cycle = Some(11);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+        let mut response = snapshot.clone();
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut response, true).await;
+        assert_eq!(response.objects_total_count, 2);
+        assert_eq!(response.objects_total_size, 84);
+
+        snapshot.scanner_cycle = Some(12);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+        assert_eq!(snapshot.objects_total_count, 0);
+        assert_eq!(snapshot.objects_total_size, 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_scans_repair_historical_usage_during_continuing_writes() {
+        let bucket = "historical-busy";
+        for position in [None, Some((7, 10))] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = position.map(|value| value.0);
+            baseline.scanner_cycle = position.map(|value| value.1);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            for objects in 1..=6 {
+                record_bucket_object_write_memory(bucket, None, 42).await;
+                let mut scanned = data_usage_info_for_test(bucket, objects, objects * 42, SystemTime::now());
+                scanned.scanner_epoch = Some(7);
+                scanned.scanner_cycle = Some(10 + objects);
+                scanned.usage_snapshot_converged = Some(true);
+                replace_bucket_usage_memory_from_info(&scanned).await;
+                assert_eq!(get_bucket_usage_memory(bucket).await, Some(objects * 42));
+                apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+                assert_eq!(scanned.objects_total_count, objects);
+                assert_eq!(scanned.objects_total_size, objects * 42);
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_usage_refresh_requires_a_new_valid_scanner_generation() {
+        let bucket = "converged-generation";
+        for (converged, epoch, cycle, partial) in [
+            (None, Some(7), Some(11), false),
+            (Some(false), Some(7), Some(11), false),
+            (Some(true), None, Some(11), false),
+            (Some(true), Some(7), None, false),
+            (Some(true), Some(7), Some(10), false),
+            (Some(true), Some(8), Some(10), false),
+            (Some(true), Some(6), Some(11), false),
+            (Some(true), Some(7), Some(9), false),
+            (Some(true), Some(7), Some(11), true),
+        ] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(10);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+            scanned.usage_snapshot_converged = converged;
+            scanned.scanner_epoch = epoch;
+            scanned.scanner_cycle = cycle;
+            scanned.usage_snapshot_partial = partial;
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(
+                get_bucket_usage_memory(bucket).await,
+                Some(4_587_688_554),
+                "converged={converged:?}, epoch={epoch:?}, cycle={cycle:?}, partial={partial}"
+            );
+            scanned.usage_snapshot_converged = Some(true);
+            scanned.scanner_epoch = Some(8);
+            scanned.scanner_cycle = Some(11);
+            scanned.usage_snapshot_partial = false;
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_usage_refresh_preserves_mutations_after_the_scan() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "converged-write-race";
+        let mut baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        baseline.scanner_epoch = Some(7);
+        baseline.scanner_cycle = Some(10);
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        scanned.scanner_epoch = Some(7);
+        scanned.scanner_cycle = Some(11);
+        scanned.usage_snapshot_converged = Some(true);
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let after_scan = scanned.last_update.expect("scanner timestamp") + Duration::from_secs(1);
+        memory_cache()
+            .write()
+            .await
+            .get_mut(bucket)
+            .expect("dirty cache")
+            .usage_updated_at = after_scan;
+        replace_bucket_usage_memory_from_info(&scanned).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+        assert_eq!(scanned.objects_total_count, 2);
+        assert_eq!(scanned.objects_total_size, 84);
+        let mut successor = data_usage_info_for_test(bucket, 2, 84, after_scan);
+        successor.scanner_epoch = Some(7);
+        successor.scanner_cycle = Some(12);
+        successor.usage_snapshot_converged = Some(true);
+        replace_bucket_usage_memory_from_info(&successor).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn fallback_observations_cannot_bypass_the_cached_scanner_generation() {
+        let bucket = "historical-generation-floor";
+        for (converged, epoch, first_cycle) in [(None, 7, 10), (Some(true), 7, 10), (None, 6, 21), (Some(true), 6, 21)] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(20);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            // Wall clocks may step back; a timestamp alone cannot establish
+            // that an asynchronously loaded generation includes this write.
+            let update = memory_cache().read().await[bucket].usage_updated_at + Duration::from_secs(1);
+            for cycle in first_cycle..first_cycle + 3 {
+                let mut scanned = data_usage_info_for_test(bucket, 0, 0, update);
+                scanned.scanner_epoch = Some(epoch);
+                scanned.scanner_cycle = Some(cycle);
+                scanned.usage_snapshot_converged = converged;
+                replace_bucket_usage_memory_from_info(&scanned).await;
+                assert_eq!(
+                    get_bucket_usage_memory(bucket).await,
+                    Some(4_587_688_554),
+                    "older epoch/cycle cannot release quota: converged={converged:?}, epoch={epoch}, cycle={cycle}"
+                );
+                apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+                assert_eq!(scanned.objects_total_count, 437);
+                assert_eq!(scanned.objects_total_size, 4_587_688_554);
+            }
+            let mut successor = data_usage_info_for_test(bucket, 1, 42, update);
+            successor.scanner_epoch = Some(8);
+            successor.scanner_cycle = Some(21);
+            successor.usage_snapshot_converged = Some(true);
+            replace_bucket_usage_memory_from_info(&successor).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn matching_counts_cannot_discard_the_cached_scanner_generation() {
+        let bucket = "matching-generation-floor";
+        for position in [None, Some((7, 10))] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 2, 84, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(20);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            let mut matching = baseline.clone();
+            matching.scanner_epoch = position.map(|value| value.0);
+            matching.scanner_cycle = position.map(|value| value.1);
+            replace_bucket_usage_memory_from_info(&matching).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            let update = memory_cache().read().await[bucket].usage_updated_at + Duration::from_secs(1);
+            let mut older = data_usage_info_for_test(bucket, 0, 0, update);
+            older.scanner_epoch = Some(7);
+            older.scanner_cycle = Some(12);
+            older.usage_snapshot_converged = Some(true);
+            replace_bucket_usage_memory_from_info(&older).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(126));
+            apply_bucket_usage_memory_overlay_if_authoritative(&mut older, true).await;
+            assert_eq!(older.objects_total_count, 3);
+            assert_eq!(older.objects_total_size, 126);
+        }
+    }
+
    #[tokio::test]
    #[serial]
    async fn authoritative_refresh_can_reduce_memory_usage() {
@@ -1236,6 +1236,17 @@ impl DiskHealthTracker {
                record_drive_recovery_class(classify_drive_recovery(duration));
            }
            self.offline_since_unix_secs.store(0, Ordering::Release);
+            info!(
+                event = EVENT_DISK_RECOVERY_PROBE_STATE,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DISK,
+                endpoint = %endpoint,
+                state = "recovered",
+                previous_state = current.as_str(),
+                runtime_state = next.as_str(),
+                reason,
+                "Disk recovered"
+            );
        } else if let Some(duration) = self.offline_duration() {
            record_drive_offline_duration(endpoint, duration);
        }
@@ -2074,7 +2085,9 @@ impl DiskAPI for LocalDiskWrapper {
                    }
                    let result = self.disk.disk_info(opts).await?;

-                    if let Some(current_disk_id) = *self.disk_id.read().await
+                    // Fresh capacity snapshots omit the disk ID; the stale-disk precheck above already verified it.
+                    if !opts.fresh_capacity
+                        && let Some(current_disk_id) = *self.disk_id.read().await
                        && Some(current_disk_id) != result.id
                    {
                        return Err(DiskError::DiskNotFound);
@@ -2560,7 +2573,9 @@ impl DiskAPI for LocalDiskWrapper {
 mod tests {
    use super::*;
    use crate::disk::endpoint::Endpoint;
+    use crate::disk::format::FormatV3;
    use crate::disk::health_state::RuntimeDriveHealthState;
+    use crate::disk::{FORMAT_CONFIG_FILE, RUSTFS_META_BUCKET};
    use std::{
        io,
        panic::{AssertUnwindSafe, catch_unwind},
@@ -2915,6 +2930,56 @@ mod tests {
        assert_eq!(snapshot.total_errors_availability, 1);
    }

+    #[tokio::test]
+    async fn fresh_capacity_keeps_disk_identity_precheck_without_requiring_id_in_snapshot() {
+        let dir = tempfile::tempdir().expect("temp dir should be created");
+        let mut endpoint =
+            Endpoint::try_from(dir.path().to_str().expect("temp dir should be valid UTF-8")).expect("endpoint should parse");
+        endpoint.set_pool_index(0);
+        endpoint.set_set_index(0);
+        endpoint.set_disk_index(0);
+        let meta_dir = dir.path().join(RUSTFS_META_BUCKET);
+        tokio::fs::create_dir_all(&meta_dir)
+            .await
+            .expect("metadata directory should be created");
+        let mut format = FormatV3::new(1, 1);
+        format.erasure.this = format.erasure.sets[0][0];
+        tokio::fs::write(meta_dir.join(FORMAT_CONFIG_FILE), format.to_json().expect("format should serialize"))
+            .await
+            .expect("format should be written");
+
+        let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("formatted disk should open"));
+        let disk_id = disk
+            .get_disk_id()
+            .await
+            .expect("disk ID should be readable")
+            .expect("formatted disk should have an ID");
+        let wrapper = LocalDiskWrapper::new(Arc::clone(&disk), false);
+        wrapper.set_disk_id_state(Some(disk_id)).await;
+
+        let snapshot = wrapper
+            .disk_info(&DiskInfoOptions {
+                fresh_capacity: true,
+                ..Default::default()
+            })
+            .await
+            .expect("fresh capacity should pass when the disk identity matches");
+        assert!(snapshot.fresh_capacity);
+        assert!(snapshot.total > 0);
+        assert!(snapshot.id.is_none(), "capacity-only snapshots should not expose the disk ID");
+
+        let stale_wrapper = LocalDiskWrapper::new(disk, false);
+        stale_wrapper.set_disk_id_state(Some(Uuid::nil())).await;
+        let error = stale_wrapper
+            .disk_info(&DiskInfoOptions {
+                fresh_capacity: true,
+                ..Default::default()
+            })
+            .await
+            .expect_err("a stale disk identity must still be rejected before the capacity probe");
+        assert_eq!(error, DiskError::DiskNotFound);
+    }
+
    #[tokio::test]
    async fn delete_versions_counts_returned_batch_error_classes() {
        let dir = tempfile::tempdir().expect("temp dir should be created");
@@ -428,6 +428,19 @@ impl DiskError {
        }
    }

+    /// Whether an internode RPC was cancelled, without classifying the peer as offline.
+    pub fn io_error_is_rpc_cancelled(error: &io::Error) -> bool {
+        error
+            .get_ref()
+            .and_then(|source| source.downcast_ref::<RpcStatusError>())
+            .is_some_and(|error| error.status().code() == tonic::Code::Cancelled)
+    }
+
+    pub(crate) fn clone_rpc_status_io_error(error: &io::Error) -> Option<io::Error> {
+        let status = error.get_ref()?.downcast_ref::<RpcStatusError>()?;
+        Some(io::Error::new(error.kind(), RpcStatusError(status.0.clone())))
+    }
+
    pub fn internode_http_error_kind(&self) -> Option<InternodeHttpErrorKind> {
        match self {
            DiskError::Io(io_error) => io_error
@@ -701,8 +714,8 @@ impl Clone for DiskError {
                DiskError::conditional_file_not_committed(io::Error::new(io_error.kind(), io_error.to_string())),
            ),
            DiskError::Io(io_error) => {
-                if let Some(status) = io_error.get_ref().and_then(|source| source.downcast_ref::<RpcStatusError>()) {
-                    return DiskError::Io(io::Error::new(io_error.kind(), RpcStatusError(status.0.clone())));
+                if let Some(error) = Self::clone_rpc_status_io_error(io_error) {
+                    return DiskError::Io(error);
                }
                DiskError::Io(
                    Self::clone_dangling_delete_grace(io_error)
@@ -900,6 +913,25 @@ mod tests {
    use super::*;
    use std::collections::HashMap;

+    #[test]
+    fn rpc_status_survives_disk_and_storage_clones() {
+        for code in [tonic::Code::Cancelled, tonic::Code::PermissionDenied] {
+            let status = tonic::Status::new(code, "operation was canceled");
+            let original = DiskError::Io(io::Error::new(io::ErrorKind::Interrupted, RpcStatusError::from(status)));
+            let display = original.to_string();
+            let disk = original.clone();
+            let storage = crate::error::StorageError::from(disk);
+            let cloned = storage.clone();
+            for error in [io::Error::from(original), io::Error::from(storage), io::Error::from(cloned)] {
+                assert_eq!(error.kind(), io::ErrorKind::Interrupted);
+                let status = error.get_ref().unwrap().downcast_ref::<RpcStatusError>().unwrap().status();
+                assert_eq!(status.code(), code);
+                assert_eq!(status.message(), "operation was canceled");
+                assert_eq!(DiskError::from(error).to_string(), display);
+            }
+        }
+    }
+
    #[test]
    fn retired_marker_deferral_survives_disk_and_storage_clones() {
        let original = DiskError::retired_marker_deferred("missing retirement record");
@@ -1365,6 +1365,10 @@ pub struct DiskInfoOptions {
    pub disk_id: String,
    pub metrics: bool,
    pub noop: bool,
+    /// Bypass the ordinary one-second admin cache and return a positive
+    /// acknowledgement only after a fresh capacity syscall completes.
+    #[serde(default)]
+    pub fresh_capacity: bool,
 }

 #[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
@@ -1389,6 +1393,10 @@ pub struct DiskInfo {
    pub rotational: bool,
    pub metrics: DiskMetrics,
    pub error: String,
+    /// Older peers omit this field. Callers must treat an absent/false value as
+    /// cached or otherwise unproven capacity evidence.
+    #[serde(default)]
+    pub fresh_capacity: bool,
 }

 #[derive(Clone, Debug, Default)]
@@ -1912,11 +1920,40 @@ mod tests {
            disk_id: "test-disk-id".to_string(),
            metrics: true,
            noop: false,
+            fresh_capacity: true,
        };

        assert_eq!(opts.disk_id, "test-disk-id");
        assert!(opts.metrics);
        assert!(!opts.noop);
+        assert!(opts.fresh_capacity);
+    }
+
+    #[test]
+    fn disk_info_fresh_capacity_ack_is_additive_for_old_peers() {
+        let old_options: DiskInfoOptions =
+            serde_json::from_str(r#"{"disk_id":"disk","metrics":false,"noop":false}"#).expect("old options");
+        assert!(!old_options.fresh_capacity);
+
+        let mut old_response_json = serde_json::to_value(DiskInfo {
+            total: 10,
+            free: 4,
+            used: 6,
+            ..Default::default()
+        })
+        .expect("response fixture");
+        old_response_json
+            .as_object_mut()
+            .expect("response object")
+            .remove("fresh_capacity");
+        let old_response: DiskInfo = serde_json::from_value(old_response_json).expect("old response");
+        assert!(!old_response.fresh_capacity);
+
+        let requested = DiskInfoOptions {
+            fresh_capacity: true,
+            ..Default::default()
+        };
+        assert_eq!(serde_json::to_value(requested).expect("new options")["fresh_capacity"], true);
    }

    /// Test ReadMultipleReq structure
@@ -73,7 +73,7 @@ impl ProcessResultBudget {
        }
        inner
            .available
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |available| available.checked_sub(bytes))
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |available| available.checked_sub(bytes))
            .map(|_| {
                Some(ResultBudgetReservation {
                    inner: Arc::clone(inner),
@@ -377,13 +377,21 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
    // Without a durable MRF consumer, partial PUTs fall back to the heal
    // admission channel and wait for its receipt before acknowledging the write.
    // Drive the test receiver alongside the PUT so neither waits on the other.
-    let (_put_dirs, put_set) = make_local_set_disks(4, 2).await;
+    let (_put_dirs, put_store) = crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks().await;
+    crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&put_store), Vec::new()).await;
+    let put_set = Arc::clone(&put_store.pools[0].disk_set[0]);
+    assert_eq!(put_set.set_drive_count, 4);
+    assert_eq!(put_set.default_parity_count, 2);
    let put_bucket = "bb-put-partial-convergence";
    let put_object = "object.bin";
-    put_set
+    put_store
        .make_bucket(put_bucket, &MakeBucketOptions::default())
        .await
        .expect("PUT bucket should be created");
+    let put_incarnation = put_store
+        .bucket_incarnation_id_from_disk(put_bucket)
+        .await
+        .expect("partial repair must bind the real persisted bucket generation");
    let offline_disk = {
        let mut disks = put_set.disks.write().await;
        disks[0].take()
@@ -400,6 +408,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
                        &ObjectOptions {
                            no_lock: true,
                            versioned: true,
+                            expected_bucket_incarnation_id: Some(put_incarnation),
                            ..Default::default()
                        },
                    )
@@ -419,6 +428,8 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
    assert_eq!(request.object_version_id.as_deref(), Some(committed_version.as_str()));
    assert_eq!(request.pool_index, Some(0));
    assert_eq!(request.set_index, Some(0));
+    assert_eq!(request.source, HealRequestSource::Mrf);
+    assert_eq!(request.expected_bucket_incarnation_id, Some(put_incarnation));

    let duplicate_request = tokio::time::timeout(std::time::Duration::from_millis(100), async {
        loop {
@@ -447,7 +458,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
    }

    let healthy_bucket = "bb-put-healthy-convergence";
-    put_set
+    put_store
        .make_bucket(healthy_bucket, &MakeBucketOptions::default())
        .await
        .expect("healthy PUT bucket should be created");
@@ -5267,6 +5267,71 @@ mod tests {
        assert_eq!(DATA_SHARDS + 1, bufs.iter().filter(|buf| buf.is_some()).count());
    }

+    /// A peer that sends part of a shard and then stops must not pin the stripe
+    /// for the full read timeout. The other EC 2+2 shards already hold a
+    /// decode-plus-verification quorum, so the lockstep hedge retires the
+    /// stalled reader and the stripe completes.
+    #[tokio::test]
+    async fn test_lockstep_hedges_shard_that_stops_mid_read() {
+        const NUM_SHARDS: usize = 1;
+        const BLOCK_SIZE: usize = 64;
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const SHARD_SIZE: usize = BLOCK_SIZE / DATA_SHARDS;
+
+        let hash_algo = HashAlgorithm::None;
+        let readers = vec![
+            Some(BitrotReader::new(
+                TestShardReader::PartialThenPending {
+                    data: vec![0xab],
+                    emitted: false,
+                },
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![1_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![2_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![3_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo,
+                false,
+            )),
+        ];
+
+        let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+        let mut parallel_reader = ParallelReader::new_with_metrics_path_read_costs_timeout_and_reconstruction_verification(
+            readers,
+            erasure,
+            0,
+            NUM_SHARDS * BLOCK_SIZE,
+            None,
+            vec![ShardReadCost::Unknown; 4],
+            Duration::from_secs(60),
+            true,
+        );
+
+        let (bufs, errs) = tokio::time::timeout(Duration::from_millis(500), parallel_reader.read())
+            .await
+            .expect("a shard that stops mid-read must be hedged instead of waiting out the 60s read timeout");
+
+        assert!(matches!(&errs[0], Some(DiskError::Io(err)) if err.kind() == ErrorKind::TimedOut));
+        assert!(parallel_reader.readers[0].is_none());
+        assert!(bufs[0].is_none());
+        assert_eq!(DATA_SHARDS + 1, bufs.iter().filter(|buf| buf.is_some()).count());
+    }
+
    #[tokio::test(start_paused = true)]
    async fn lockstep_reopens_hedged_shard_after_later_peer_loss() {
        const DATA_SHARDS: usize = 2;
--- a/Show More
+++ b/Show More