test(e2e): register paginator regression in Darwin inventory

fix(storage): publish delete rollback backups atomically
Stage rollback metadata outside the rollback directory and publish it only after the full write succeeds. A short write must not leave a backup that quorum rollback can rename over acknowledged version history. Add an isolated real short-write regression and register the backported ListParts SDK test in the smoke and Linux full inventories.
2026-10-27 14:27:10 +00:00 · 2026-10-03 07:07:41 +08:00 · 2026-10-03 06:43:25 +08:00 · 2026-10-03 06:32:24 +08:00 · 2026-10-03 06:32:24 +08:00 · 2026-10-02 19:59:28 +08:00
68 changed files with 4902 additions and 1040 deletions
@@ -1,6 +1,6 @@
 ---
 name: rustfs-release-publish
-description: "Run the RustFS console gate, source version bump, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
+description: "Run the RustFS console gate, source version bump, release-branch CI, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
 ---
 # RustFS Release Publish (preview-validated pipeline)

@@ -19,7 +19,9 @@ Pipeline shape:
 check console main against its latest Release
  -> if ahead: publish console -> wait for Release asset + latest API
  -> bump Cargo.toml and Cargo.lock to <target> -> merge
-  -> tag <preview-tag> at that commit -> CI green
+  -> cut release/<target> from a selected main commit containing the bump
+  -> CI green for that exact release-branch commit
+  -> tag <preview-tag> at that commit
  -> verify preview Release assets -> run binary locally + console checks
  -> validate with latest rc client
  -> report preview acceptance results -> STOP for explicit human confirmation
@@ -29,7 +31,7 @@ check console main against its latest Release
  -> update installation references on main -> update rustfs.com announcement
 ```

-On preview validation failure: fix lands on main via normal PR (Cargo versions are already at `<target>`, no new source bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2. Installation references remain on the previous published deliverable throughout preview validation.
+On candidate or preview validation failure: fix lands on main via normal PR, then backport only the needed fix to `release/<target>` through a reviewed PR. Re-run release-branch CI at the new commit and, if a preview was already tagged, tag `<preview-tag N+1>` there and restart acceptance. Installation references remain on the previous published deliverable throughout preview validation.

 ## Required inputs

@@ -68,14 +70,14 @@ Rules:
 - Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
 - Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
 - Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
+- The release branch is the candidate source during multi-day acceptance. Do not merge main wholesale into it, force-push it, or advance it with unrelated changes. The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD or an advanced release-branch HEAD, and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
 - When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
 - Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure blocks dependent steps. For a preview/source acceptance failure before the final tag exists, land the fix, verify Cargo still matches the confirmed target, and restart from Phase 2 with the next preview at the fixed commit. If main has advanced to another target, resolve that source/version mismatch before another preview.
+- Phases run in order; a failure blocks dependent steps. For a candidate or preview/source acceptance failure before the final tag exists, land the fix on main, backport it to the release branch, verify Cargo there still matches the confirmed target, and restart from release-branch CI in Phase 2. Use the next preview iteration if any preview was already tagged. Main may advance to another target without changing this release candidate.
 - Once the final tag exists, preserve its commit. Retry failed publication jobs for that exact tag; do not recreate the tag or restart preview on newer main. If a source change is required after final-tag publication, report the failure and obtain a new target version. Phase 7 failures resume only the failed follow-up after rechecking artifact availability and whether a newer release has superseded it.
 - Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
 - Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
+- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision. Do not delete the release branch or preview tags as part of abandonment without an explicit decision.
 - User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.

 ## Phase 0 — Preflight
@@ -83,6 +85,7 @@ Rules:
 - `git status --short` clean; `git fetch origin main --tags`.
 - `gh auth status` works; confirm you can view `gh release list -L 3`.
 - Confirm the exact final target version with the user if not explicit.
+- Check whether `release/<target>` already exists before creating it; a resumed release reuses the existing branch and its history. For an in-flight release that already has a preview tag but no branch, anchor the new branch to that tag's commit only after verifying its target versions and prior acceptance evidence.

 ### Console release gate

@@ -90,18 +93,19 @@ Read and complete [the Console gate](references/console-gate.md) before Phase 1.

 ## Phase 1 — Source version bump to the final target (once)

- If Cargo.toml and all workspace members in Cargo.lock already read `<target>` (e.g. this is a restart after a failed preview), verify both and skip the source bump. Installation versions intentionally differ during preview; do not treat them as leftovers. If a previous preparation advanced installation references to an unavailable target, restore those references to the verified published baseline before selecting the next preview commit.
- Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- Get the PR merged into main. Record the resulting main commit:
+- If `release/<target>` exists, verify its Cargo.toml and workspace members in Cargo.lock read `<target>` and skip the source bump, even if main has advanced. If the branch versions differ, stop and resolve the mismatch without downgrading main.
+- If no branch exists but an unfinished preview tag does, verify its commit has `<target>` in both version files and skip the bump; Phase 2 anchors the branch to that commit. Stop if those files disagree. Otherwise, if main already has `<target>` in both files, skip the bump. Installation references intentionally retain the previous published deliverable during preview; if they were advanced to an unavailable target, restore that baseline before choosing the next candidate.
+- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
+- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.

-```bash
-git fetch origin main
-PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
-```
+Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.

-`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
+## Phase 2 — Pin and validate the release candidate, then publish the preview tag

-## Phase 2 — Publish the preview tag
+- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
+- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
+- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
+- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.

 ```bash
 git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
@@ -112,11 +116,12 @@ Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `d

 The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.

-On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
+On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.

 ## Phase 3 — CI and preview Release verification

 - Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
+- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
 - Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
 - Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
 - Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
@@ -128,13 +133,13 @@ Read and complete [preview acceptance](references/preview-acceptance.md): verify

 ### Manual confirmation gate

-After every Phase 3–5 check passes, report the target, preview tag, `PREVIEW_HASH`, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
+After every Phase 3–5 check passes, verify the release branch still points to `PREVIEW_HASH`. Report the target, release branch, preview tag, `PREVIEW_HASH`, release-branch CI run, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.

 Continue to Phase 6 only after a new user reply explicitly confirms the reported target, preview tag, and commit. A clear affirmative reply to that exact report, such as `确认继续`, is sufficient; if the reply is ambiguous or any reported value changed, ask again.

 ## Phase 6 — Publish the final tag on the validated commit

-No second source version bump. The final tag goes on the exact commit the preview validated:
+No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:

 ```bash
 git fetch origin --tags
@@ -158,7 +163,7 @@ Only after Phase 6 succeeds, complete [post-release updates](references/post-rel
 Always report:

 - Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
+- Target version, release branch, release-branch CI run and validated SHA, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
 - Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
 - Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
 - Post-release installation PR and verification results; website banner target, text, link, PR, and observed deployment state. Report any remaining merge authorization or failed deployment explicitly rather than claiming the banner is live.
@@ -1,2 +1,2 @@
-sha256-darwin=3b6ff112ff16acbaa4148c4736095a1b6b71cc11af0dd3513ff28c6bdc334ca8
-sha256-linux=4e8bfc01870565a0a608a1634e6919fc593ce120a970aa04d974f7edf2675c21
+sha256-darwin=6b8e35ef69456bd244d6a6b19179e408763763e1e163aaaf90055490ae12da4b
+sha256-linux=ff4c40f288ab5e91fd28d533c277538c0e3281fa353328ab52f5a8f88e0dc931
@@ -1 +1 @@
-sha256=476cb92aa9e385c9bb509ee5df3e5bd07e8c929730040b60d16791b6c0616d00
+sha256=7e3989625e2e6087c2e7db7ee02d30624b566d5a99832122032fb413ff456662
@@ -82,6 +82,13 @@ setup = 'lifecycle-large-stack'
 filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
 setup = 'ecstore-large-stack'

+# These real-disk scanner fixtures assert quiet-workload publication and
+# durable MRF admission. Reserve the pool while their bounded CAS/fsync
+# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
+[[profile.default.overrides]]
+filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
+threads-required = "num-test-threads"
+
 [[profile.default.overrides]]
 filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
@@ -301,6 +308,13 @@ setup = 'lifecycle-large-stack'
 filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
 setup = 'ecstore-large-stack'

+# These real-disk scanner fixtures assert quiet-workload publication and
+# durable MRF admission. Reserve the pool while their bounded CAS/fsync
+# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
+threads-required = "num-test-threads"
+
 # ===========================================================================
 # QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
 #
@@ -542,10 +556,11 @@ fail-fast = false
 [profile.e2e-smoke.junit]
 path = "junit.xml"

-# These storage-heavy fixtures reserve capacity. Pagination tests bound listing
-# requests separately, after their seed objects have been written.
+# Bound the whole pagination fixture, including startup, seeding, and cleanup;
+# its LIST deadline begins only after the seed objects have been written.
 [[profile.e2e-smoke.overrides]]
 filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
+slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
 threads-required = "num-test-threads"

 [[profile.e2e-smoke.overrides]]
@@ -551,54 +551,6 @@ jobs:
          if-no-files-found: error
          retention-days: 30

-  # Docker Hub's overview is a separate `full_description` field that
-  # `docker push` never touches, so it drifted from README.md and kept
-  # linking to pages that no longer exist (#7995). Republish it from the
-  # commit the images were built from, only when images were pushed.
-  sync-dockerhub-description:
-    name: Sync Docker Hub Description
-    needs: [ build-check, build-docker ]
-    if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
-    runs-on: sm-standard-2
-    timeout-minutes: 10
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-          ref: ${{ needs.build-check.outputs.source_ref }}
-          sparse-checkout: README.md
-          sparse-checkout-cone-mode: false
-
-      - name: Sync Docker Hub description
-        uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
-        with:
-          username: ${{ env.DOCKERHUB_USERNAME }}
-          password: ${{ secrets.DOCKERHUB_TOKEN }}
-          repository: ${{ env.REGISTRY_DOCKERHUB }}
-          readme-filepath: ./README.md
-          # README.md links to docs/ and CONTRIBUTING.md by relative path;
-          # rewrite those to github.com URLs so they resolve on Docker Hub.
-          enable-url-completion: true
-
-      # Docker Hub caps `full_description` at 25,000 bytes and the action
-      # truncates to fit with only a warning, which would publish a README
-      # cut off mid-sentence. README.md is already near the cap, so read the
-      # published overview back and fail if it hit the limit.
-      - name: Check published description was not truncated
-        env:
-          REPOSITORY: ${{ env.REGISTRY_DOCKERHUB }}
-        run: |
-          limit=25000
-          published=$(curl -sSf "https://hub.docker.com/v2/repositories/${REPOSITORY}/" \
-            | jq -j '.full_description' | wc -c | tr -d ' ')
-          echo "Published overview: ${published} bytes (Docker Hub limit: ${limit})"
-          # Truncation lands within one UTF-8 character of the cap.
-          if [ "$published" -ge $((limit - 4)) ]; then
-            echo "::error file=README.md::Docker Hub overview was truncated at ${limit} bytes; trim README.md before the next release."
-            exit 1
-          fi
-
  # Docker build summary
  docker-summary:
    name: Docker Build Summary
@@ -685,9 +685,9 @@ dependencies = [

 [[package]]
 name = "async-compression"
-version = "0.4.48"
+version = "0.4.50"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fb61aea1a7def73ee7c350a184f0e70b32c182344e2e75bf70c9b621b83417fd"
+checksum = "ee19bd99b43e3691acbad4e840420a4881cea6c0b66a208125a824f8fd53f5a1"
 dependencies = [
 "compression-codecs",
 "compression-core",
@@ -771,17 +771,6 @@ dependencies = [
 "url",
 ]

-[[package]]
-name = "async-recursion"
-version = "1.1.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 2.0.119",
-]
-
 [[package]]
 name = "async-rs"
 version = "0.8.12"
@@ -1012,9 +1001,9 @@ dependencies = [

 [[package]]
 name = "aws-sdk-s3"
-version = "1.150.0"
+version = "1.151.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2fecaa77c862fa1e9d636d54fa06e0049a19a4f071b0d72ea1b9086887d09359"
+checksum = "c82d4b41fe41bfca3c9e1de55645d9a525868ef5ce5e14cee9499ccaf1fc7225"
 dependencies = [
 "arc-swap",
 "aws-credential-types",
@@ -1215,9 +1204,9 @@ dependencies = [

 [[package]]
 name = "aws-smithy-http-client"
-version = "1.4.2"
+version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7bd25384a4e437aa8d8f339afad4b69e786b936a7cb10db668a7aaf66717b1a8"
+checksum = "51c89cc3f1f281d659a67a519a1b5c6d445b5ce09fa7e5aee40c2c2707e9509d"
 dependencies = [
 "aws-smithy-async",
 "aws-smithy-protocol-test",
@@ -1297,9 +1286,9 @@ dependencies = [

 [[package]]
 name = "aws-smithy-runtime"
-version = "1.15.0"
+version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "591e0024cdf2a8de8711860f7e1d5cfbca26b800ce13793a9aa2f4f5918e6d95"
+checksum = "d6e302ac1d88b99652489df31abdec6ac42a2ab2ac3982ad0ac49f64dfaf28ba"
 dependencies = [
 "aws-smithy-async",
 "aws-smithy-http",
@@ -1321,9 +1310,9 @@ dependencies = [

 [[package]]
 name = "aws-smithy-runtime-api"
-version = "1.18.0"
+version = "1.19.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9bb0deb40a69de59f7809251cfe1ab16de135f42c49d222d1202b4d0ce19720f"
+checksum = "c0730c16f91124c6a2abb4932c77e299288b3dd9f967ea2e9ec48cc6731e87a4"
 dependencies = [
 "aws-smithy-async",
 "aws-smithy-runtime-api-macros",
@@ -1544,9 +1533,9 @@ dependencies = [

 [[package]]
 name = "bigdecimal"
-version = "0.4.10"
+version = "0.4.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695"
+checksum = "8fa3f3d8cbf4dffcfe4991de61d012bef509a409ecbe9dd41049bfe32b4d4653"
 dependencies = [
 "autocfg",
 "libm",
@@ -2209,11 +2198,11 @@ dependencies = [

 [[package]]
 name = "compression-codecs"
-version = "0.4.43"
+version = "0.4.45"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bef16c47ba2797aa6a909cc37d39911f3a6743811fe7408ac0b0cc0276b656e9"
+checksum = "98fc98460ba0ad5317075d3632b8dfc45d0be8c4a49347c2a38272019717614a"
 dependencies = [
- "brotli 8.0.4",
+ "brotli 9.0.0",
 "bzip2",
 "compression-core",
 "flate2",
@@ -4804,9 +4793,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-gax-internal"
-version = "0.7.19"
+version = "0.7.20"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c47c73e95dc5e1de74bebe21b406394b53ff5ccfdfe0e81a0443cdef468e6744"
+checksum = "e873f7e6e39030c07b6d9697bf3b999a269175c8b433e058f56ae10d129f12f7"
 dependencies = [
 "bytes",
 "futures",
@@ -4820,9 +4809,9 @@ dependencies = [
 "http-body-util",
 "hyper",
 "lazy_static",
- "opentelemetry 0.32.0",
- "opentelemetry-semantic-conventions 0.32.1",
- "opentelemetry_sdk 0.32.1",
+ "opentelemetry",
+ "opentelemetry-semantic-conventions",
+ "opentelemetry_sdk",
 "percent-encoding",
 "pin-project",
 "prost 0.14.4",
@@ -4838,7 +4827,7 @@ dependencies = [
 "tonic-prost",
 "tower",
 "tracing",
- "tracing-opentelemetry 0.33.0",
+ "tracing-opentelemetry",
 ]

 [[package]]
@@ -4908,9 +4897,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-storage"
-version = "1.19.0"
+version = "1.20.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5f7fa1e66d6b47220dbfc5f243b634136e5b8d14d64cb13419bd59f976681da3"
+checksum = "570eb8d0784118a5f4ae1219fd76ff3b14e90b7b0f877478c02b1374e9bbcb8c"
 dependencies = [
 "async-trait",
 "base64 0.23.1",
@@ -4961,9 +4950,9 @@ dependencies = [

 [[package]]
 name = "google-cloud-wkt"
-version = "1.8.0"
+version = "1.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ad4b36fefda4485f0922de7a58a3e562e984d513442a1aae2281dce25124be2e"
+checksum = "7f3037907e542cf8b8580aa4cf8ad68568d7ac8e7274519a7d0a121867d434da"
 dependencies = [
 "base64 0.23.1",
 "bytes",
@@ -5282,9 +5271,9 @@ dependencies = [

 [[package]]
 name = "hotpath"
-version = "0.26.1"
+version = "0.28.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "036ff36fa1c4ae8dfb686606537eaa15b5d5da0c51bf94abeaf9dfc6ee54cac7"
+checksum = "d8345e171844f80e5ae4cbed8bc8276e094f6ff2293a6f8e547c200183976674"
 dependencies = [
 "arc-swap",
 "async-channel",
@@ -5310,6 +5299,7 @@ dependencies = [
 "rustc-demangle",
 "serde",
 "serde_json",
+ "time",
 "tiny_http",
 "tokio",
 ]
@@ -5322,9 +5312,9 @@ checksum = "e149b660b611650bdf3307941ef3e9b81be3c30ed1dfdbe7f670319674f023a6"

 [[package]]
 name = "hotpath-macros"
-version = "0.26.1"
+version = "0.28.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1ff68b15fdbcddfbad685f445597580d6fcbf28b49f01ff975e934351079e13b"
+checksum = "bf73e28fe4a7595c322e4265c1608cb10c215cbbce11e49cfcc258766ba686d6"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -6044,9 +6034,9 @@ dependencies = [

 [[package]]
 name = "lazy_static"
-version = "1.5.0"
+version = "1.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
+checksum = "20870f649af7073d53e38067b2a84312175d56ea15217e1b15bc83506ec50afb"
 dependencies = [
 "spin 0.9.9",
 ]
@@ -7318,20 +7308,6 @@ version = "0.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"

-[[package]]
-name = "opentelemetry"
-version = "0.32.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0142c63252a9e054e68a4c61a5778f7b14f576274d593f8ce883d191a099682"
-dependencies = [
- "futures-core",
- "futures-sink",
- "js-sys",
- "pin-project-lite",
- "thiserror 2.0.21",
- "tracing",
-]
-
 [[package]]
 name = "opentelemetry"
 version = "0.33.0"
@@ -7352,7 +7328,7 @@ version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e2b304e8d8f11326ba366a99ca5c7aaf7cd397b67268a332437175aeb42a40db"
 dependencies = [
- "opentelemetry 0.33.0",
+ "opentelemetry",
 "tracing",
 "tracing-core",
 "tracing-log",
@@ -7368,7 +7344,7 @@ dependencies = [
 "async-trait",
 "bytes",
 "http 1.5.0",
- "opentelemetry 0.33.0",
+ "opentelemetry",
 "reqwest",
 ]

@@ -7381,10 +7357,10 @@ dependencies = [
 "flate2",
 "http 1.5.0",
 "httpdate",
- "opentelemetry 0.33.0",
+ "opentelemetry",
 "opentelemetry-http",
 "opentelemetry-proto",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry_sdk",
 "prost 0.14.4",
 "reqwest",
 "thiserror 2.0.21",
@@ -7396,17 +7372,11 @@ version = "0.33.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "25da1ac11a0aeccf38d7f77ee0348715adaf8340f65ad46c94a02c6b20e2f65d"
 dependencies = [
- "opentelemetry 0.33.0",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry",
+ "opentelemetry_sdk",
 "prost 0.14.4",
 ]

-[[package]]
-name = "opentelemetry-semantic-conventions"
-version = "0.32.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c913ac17a6c451661ee255f4625d143e51647ae78ebd969b75e41c4442f4fe47"
-
 [[package]]
 name = "opentelemetry-semantic-conventions"
 version = "0.33.0"
@@ -7420,24 +7390,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "9f71803f481d3f5ccbe477edb42a78f473613712b2327835911a9ea0ab069824"
 dependencies = [
 "chrono",
- "opentelemetry 0.33.0",
- "opentelemetry_sdk 0.33.0",
-]
-
-[[package]]
-name = "opentelemetry_sdk"
-version = "0.32.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9b59f80e1ac4d5ff7a2db8fb6c80badb7f0f3f858211fba08dd9aaec750894f9"
-dependencies = [
- "futures-channel",
- "futures-executor",
- "futures-util",
- "opentelemetry 0.32.0",
- "percent-encoding",
- "portable-atomic",
- "rand 0.9.5",
- "thiserror 2.0.21",
+ "opentelemetry",
+ "opentelemetry_sdk",
 ]

 [[package]]
@@ -7449,7 +7403,7 @@ dependencies = [
 "futures-channel",
 "futures-executor",
 "futures-util",
- "opentelemetry 0.33.0",
+ "opentelemetry",
 "percent-encoding",
 "portable-atomic",
 "rand 0.9.5",
@@ -7939,9 +7893,9 @@ checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"

 [[package]]
 name = "pin-utils"
-version = "0.1.0"
+version = "0.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
+checksum = "13bee6c73da26345c729282832b60b0363cf3dd9f4bfd81d8551b7a1c889a113"

 [[package]]
 name = "piper"
@@ -8473,9 +8427,9 @@ dependencies = [

 [[package]]
 name = "pulldown-cmark-to-cmark"
-version = "22.0.1"
+version = "22.0.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ab1ad36992cead65f02aa399a373a42730922f1525d988172634fdefdecb8a60"
+checksum = "84bbb29c624230c4bd1047bbdb2aa47e41c860e9665ce62ba9504eebe91bf867"
 dependencies = [
 "pulldown-cmark",
 ]
@@ -8710,9 +8664,9 @@ dependencies = [

 [[package]]
 name = "quinn-proto"
-version = "0.11.18"
+version = "0.11.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc"
+checksum = "0e750cca55fe4f0439a15d0bb529da9651e79993e8e72c61a899a36d462befbe"
 dependencies = [
 "aws-lc-rs",
 "bytes",
@@ -8733,9 +8687,9 @@ dependencies = [

 [[package]]
 name = "quinn-udp"
-version = "0.5.15"
+version = "0.5.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "35a133f956daabe89a61a685c2649f13d82d5aa4bd5d12d1277e1072a21c0694"
+checksum = "af66907df18639dcf4db56ca65490cabc4b27a97dbadd96f2926cca73298f016"
 dependencies = [
 "cfg_aliases",
 "libc",
@@ -9416,9 +9370,9 @@ dependencies = [

 [[package]]
 name = "russh-sftp"
-version = "3.0.0"
+version = "3.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "093197e526668d92bba562e2bbbe98d1af9831bf080b619c736316ca1fa35101"
+checksum = "9bb17f72aba4314656b359ffba72c68dcbd70ed9e97497898b75498fe7198042"
 dependencies = [
 "bitflags 2.13.2",
 "bytes",
@@ -9554,9 +9508,9 @@ dependencies = [
 "mime_guess",
 "moka",
 "object 0.40.0",
- "opentelemetry 0.33.0",
+ "opentelemetry",
 "opentelemetry-proto",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry_sdk",
 "p256 0.14.0",
 "parking_lot",
 "percent-encoding",
@@ -9647,7 +9601,7 @@ dependencies = [
 "tower",
 "tower-http 0.7.1",
 "tracing",
- "tracing-opentelemetry 0.34.0",
+ "tracing-opentelemetry",
 "tracing-subscriber",
 "url",
 "urlencoding",
@@ -9797,7 +9751,6 @@ version = "1.0.1"
 dependencies = [
 "arc-swap",
 "async-channel",
- "async-recursion",
 "async-trait",
 "aws-credential-types",
 "aws-sdk-s3",
@@ -9837,8 +9790,8 @@ dependencies = [
 "metrics-util",
 "moka",
 "num_cpus",
- "opentelemetry 0.33.0",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry",
+ "opentelemetry_sdk",
 "parking_lot",
 "path-absolutize",
 "pin-project-lite",
@@ -9899,7 +9852,7 @@ dependencies = [
 "tower",
 "tracing",
 "tracing-core",
- "tracing-opentelemetry 0.34.0",
+ "tracing-opentelemetry",
 "tracing-subscriber",
 "url",
 "urlencoding",
@@ -10416,12 +10369,12 @@ dependencies = [
 "metrics-util",
 "num_cpus",
 "nvml-wrapper",
- "opentelemetry 0.33.0",
+ "opentelemetry",
 "opentelemetry-appender-tracing",
 "opentelemetry-otlp",
- "opentelemetry-semantic-conventions 0.33.0",
+ "opentelemetry-semantic-conventions",
 "opentelemetry-stdout",
- "opentelemetry_sdk 0.33.0",
+ "opentelemetry_sdk",
 "percent-encoding",
 "pyroscope",
 "rcgen",
@@ -10453,7 +10406,7 @@ dependencies = [
 "tracing",
 "tracing-appender",
 "tracing-error",
- "tracing-opentelemetry 0.34.0",
+ "tracing-opentelemetry",
 "tracing-subscriber",
 "url",
 "zstd 0.14.0",
@@ -10762,7 +10715,6 @@ dependencies = [
 name = "rustfs-s3select-query"
 version = "1.0.1"
 dependencies = [
- "async-recursion",
 "async-trait",
 "bytes",
 "datafusion",
@@ -11170,9 +11122,9 @@ dependencies = [

 [[package]]
 name = "rustls-connector"
-version = "0.23.8"
+version = "0.23.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1babecfcc65b139b812e74bcc7f9ec7b4e00db659fd42d99567b7e77f0c714c6"
+checksum = "524994fd6957827c775abc941f41a146ce351bbd0a9b8cf632ee37f43a13c205"
 dependencies = [
 "futures-io",
 "futures-rustls",
@@ -12283,9 +12235,9 @@ checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"

 [[package]]
 name = "suppaftp"
-version = "12.1.0"
+version = "12.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d79f066c76927a775d61b5642e0b712a17582ebe7f62b7e5abba246e619878f7"
+checksum = "b161d71217da0cc91bd09c8f98ac5ad2a5e5b2735c6dfe1c6cb08b4471612f35"
 dependencies = [
 "async-trait",
 "chrono",
@@ -12776,9 +12728,9 @@ dependencies = [

 [[package]]
 name = "tokio-rustls"
-version = "0.26.5"
+version = "0.26.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0c85f2c3ef0b1cd58b36682f4b17aaa995f0e5db534d85692b4903abce21f67"
+checksum = "c9cc2678c2cdd569ef8215e2afd7954ada2ae20b4fdd2c5fe6139a3b02d105db"
 dependencies = [
 "rustls",
 "tokio",
@@ -13086,20 +13038,6 @@ dependencies = [
 "tracing-core",
 ]

-[[package]]
-name = "tracing-opentelemetry"
-version = "0.33.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "adbc64cba7137545b8044cb1fe9814f7aacf3c6b5f9b45be8bb5db538befdb26"
-dependencies = [
- "js-sys",
- "opentelemetry 0.32.0",
- "tracing",
- "tracing-core",
- "tracing-subscriber",
- "web-time",
-]
-
 [[package]]
 name = "tracing-opentelemetry"
 version = "0.34.0"
@@ -13107,7 +13045,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0a904802a1b902f43638b677ff2a650847e3b4404101b6c586d648e8c1e3e8fe"
 dependencies = [
 "js-sys",
- "opentelemetry 0.33.0",
+ "opentelemetry",
 "smallvec",
 "tracing",
 "tracing-core",
@@ -14042,9 +13980,9 @@ dependencies = [

 [[package]]
 name = "xxhash-rust"
-version = "0.8.18"
+version = "0.8.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aee1b19627c7c60102ab80d3a9cbe18de90bfe03bfa6c3715447681f0e8c8af6"
+checksum = "550a2b930b62486a393c52d5c3b84bff264b28aa437ed64694d31e93b1757af7"

 [[package]]
 name = "yansi"
@@ -14075,9 +14013,9 @@ dependencies = [

 [[package]]
 name = "yoke-derive"
-version = "0.8.3"
+version = "0.8.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "33811428bee40dbceb6d545e95754741d17a6aef9a4849f0fd62e2ba4f412a78"
+checksum = "ec8ebde2db3681e8c9980cc27822030e68752690ddfa9473e739aeb4dbde6d71"
 dependencies = [
 "proc-macro2",
 "quote",
@@ -146,8 +146,7 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
 async-channel = "2.5.0"
 async_zip = { default-features = false, version = "0.0.19" }
 mysql_async = { default-features = false, version = "0.37.1" }
-async-compression = { version = "0.4.48" }
-async-recursion = "1.1.1"
+async-compression = { version = "0.4.50" }
 async-trait = "0.1.92"
 async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
@@ -170,7 +169,7 @@ reqwest = "0.13.5"
 rustfs-kafka-async = { version = "1.3.1" }
 socket2 = { version = "0.6.5" }
 tokio = { version = "1.53.1" }
-tokio-rustls = { default-features = false, version = "0.26.5" }
+tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
 tokio-util = { version = "0.7.19" }
@@ -246,11 +245,11 @@ atomic_enum = "0.3.0"
 aws-config = { version = "1.12.0" }
 aws-credential-types = { version = "1.3.0" }
 aws-sdk-kms = { default-features = false, version = "1.122.0" }
-aws-sdk-s3 = { default-features = false, version = "1.150.0" }
+aws-sdk-s3 = { default-features = false, version = "1.151.0" }
 aws-sdk-sts = { default-features = false, version = "1.118.0" }
 aws-smithy-async = { version = "1.3.0" }
-aws-smithy-http-client = { default-features = false, version = "1.4.2" }
-aws-smithy-runtime-api = { version = "1.18.0" }
+aws-smithy-http-client = { default-features = false, version = "1.5.0" }
+aws-smithy-runtime-api = { version = "1.19.0" }
 aws-smithy-types = { version = "1.8.1" }
 base64-simd = "0.8.0"
 brotli = "9.0.0"
@@ -269,7 +268,7 @@ enumset = "1.1.14"
 faster-hex = "1.0.0"
 flate2 = "1.1.10"
 glob = "0.3.4"
-google-cloud-storage = "1.19.0"
+google-cloud-storage = "1.20.0"
 google-cloud-auth = "1.17.0"
 hashbrown = { version = "0.17.1" }
 # Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
@@ -279,7 +278,7 @@ hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
 hostname = "0.4.2"
 ipnetwork = { version = "0.21.1" }
-lazy_static = "1.5.0"
+lazy_static = "1.5.1"
 libc = "0.2.189"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
@@ -346,7 +345,7 @@ walkdir = "2.5.0"
 winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
-xxhash-rust = { version = "0.8.18" }
+xxhash-rust = { version = "0.8.19" }
 zip = "8.6.0"
 zstd = "0.14.0"

@@ -366,10 +365,10 @@ pyroscope = { version = "2.1.1" }
 # FTP and SFTP
 libunftp = { version = "0.23.1" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.1.0" }
+suppaftp = { version = "12.1.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.63.3" }
-russh-sftp = "3.0.0"
+russh-sftp = "3.0.1"

 # WebDAV
 dav-server = "0.11.0"
@@ -377,7 +376,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.26.1", default-features = false }
+hotpath = { version = "0.28.0", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }

@@ -168,9 +168,10 @@ Scanner cycle budget controls:

 ## Foreground write admission environment variables

-Large direct `PutObject` requests and multipart `UploadPart` requests share one
-per-process permit pool that bounds concurrent body ingest and storage writes.
-Small direct PUTs stay on the legacy path.
+Large direct `PutObject` requests, multipart `UploadPart` requests, and
+server-side multipart `UploadPartCopy` requests share one per-process permit
+pool that bounds concurrent body ingest and storage writes. Small direct PUTs
+stay on the legacy path.

 - `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE`
  - enables the default-on pool; `false` keeps only the soft request counter.
@@ -190,11 +191,11 @@ Small direct PUTs stay on the legacy path.
  - smallest `UploadPart` that takes a permit; `0` gates every part.
  - default is `0`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
-  - how long an `UploadPart` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
-  - default is `10000`. Parts wait before body ingest, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
-  - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. Keep it with margin below the shortest such timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
+  - how long an `UploadPart` or `UploadPartCopy` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
+  - default is `10000`. Parts wait before body ingest or source-object reads, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
+  - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. `UploadPartCopy` waits before taking bucket lifecycle locks or opening source readers, and uses the unknown-size weight because the authoritative copy length is available only after source metadata and range validation. Keep the wait with margin below the shortest client/proxy timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING`
-  - maximum `UploadPart` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
+  - maximum `UploadPart` and `UploadPartCopy` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
  - default is `0`, which derives 16 times the large-write slot count, or the explicit request-count limit (512 at stock settings). Automatic subdivision does not enlarge this queue.
  - each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory). Autotuned receive buffers can remain large on reused connections; queue depth does not imply a fixed per-connection memory cost.
 - `RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE`, `RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT`, `RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
@@ -387,7 +387,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 10
 /// Multipart pressure is often many moderate-sized parts rather than one very
 /// large request. The default gates every multipart part through the same permit
 /// pool as large/unknown-size PutObject while keeping small direct PUTs on the
-/// legacy path.
+/// legacy path. Server-side UploadPartCopy requests use this same multipart
+/// admission path with unknown-size weighting because source metadata and range
+/// validation happen after admission.
 pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str =
    "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
 pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 0;
@@ -415,7 +417,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
 /// wait with a 10 s request deadline. The wait must leave margin under the
 /// shortest of those, not merely fall below an SDK default, so the part
 /// receives S3 `SlowDown`/503 for the client to retry instead of losing its
-/// connection (issue #7385). `0` rejects immediately when the pool is full.
+/// connection (issue #7385). Server-side UploadPartCopy waits in the same queue
+/// before taking bucket lifecycle locks or opening source readers. `0` rejects
+/// immediately when the pool is full.
 pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str =
    "RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
 pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_000;
@@ -427,7 +431,7 @@ pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_0
 // the wait past any client timeout.
 const _: () = assert!(DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS * 3 <= 30_000);

-/// Maximum multipart UploadPart requests waiting for a foreground write permit per process.
+/// Maximum multipart UploadPart or UploadPartCopy requests waiting for a foreground write permit per process.
 ///
 /// Parts beyond this queue depth are rejected with S3 `SlowDown`/503 without
 /// waiting, so a genuinely saturated node still fails fast instead of holding
@@ -1450,8 +1450,11 @@ mod tests {
            cluster.set_env("RUSTFS_HEAL_PAGE_PARALLEL_ENABLE", "false");
        }
        // Keep every node's Heal runtime enabled for normal disk registration.
-        let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG")
-            .unwrap_or_else(|_| "rustfs::heal::task=info,rustfs=error".to_string());
+        // Capture durable handoff and marker cleanup failures during shutdown.
+        let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG").unwrap_or_else(|_| {
+            "rustfs::heal::task=info,rustfs::heal::manager=info,rustfs::main::handle_shutdown=info,rustfs_heal::heal=warn,rustfs=error"
+                .to_string()
+        });
        cluster.set_env("RUST_LOG", server_rust_log);
        let log_dir = std::env::var("RUSTFS_HEAL_CHAOS_LOG_DIR").unwrap_or_else(|_| {
            // Keep explicit directories compatible and isolate parallel suite captures.
@@ -886,6 +886,8 @@ fn configure_reader_metric_cluster(cluster: &mut RustFSTestClusterEnvironment, c
    cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "true");
    cluster.set_env("RUSTFS_OBS_TRACES_EXPORT_ENABLED", "false");
    cluster.set_env("RUSTFS_OBS_LOGS_EXPORT_ENABLED", "false");
+    // Keep startup diagnostics available when OTLP log export is disabled.
+    cluster.set_env("RUSTFS_OBS_LOG_STDOUT_ENABLED", "true");
    cluster.set_env("RUSTFS_OBS_METER_INTERVAL", "1");
    cluster.set_env("RUSTFS_OBS_USE_STDOUT", "false");
    cluster.set_env("RUSTFS_GET_CODEC_STREAMING_ENABLE", "false");
@@ -1029,6 +1029,8 @@ mod tests {
    /// tests the non-delimiter case.
    #[tokio::test]
    async fn test_list_objects_v2_maxkeys_above_limit_with_delimiter() {
+        use futures::{StreamExt, stream};
+
        init_logging();
        info!("Starting test: ListObjectsV2 MaxKeys above limit with delimiter");

@@ -1043,27 +1045,39 @@ mod tests {
        // 12 dirs × 100 files = 1200 raw keys
        let dir_count = 12;
        let files_per_dir = 100;
-        for d in 0..dir_count {
-            for f in 0..files_per_dir {
-                let key = format!("dir-{:02}/file{:03}.txt", d, f);
-                client
-                    .put_object()
-                    .bucket(bucket)
-                    .key(&key)
-                    .body(ByteStream::from_static(b"x"))
-                    .send()
-                    .await
-                    .expect("Failed to put object");
-            }
-        }
-
+        const FIXTURE_CONCURRENCY: usize = 8;
+        let fixture_started = std::time::Instant::now();
        eprintln!(
-            "Seeded {} objects in {bucket}; starting ListObjectsV2 pagination",
+            "Preparing {} objects with at most {FIXTURE_CONCURRENCY} concurrent PUTs",
            dir_count * files_per_dir
        );
+        // Object creation order is irrelevant to this listing contract. Bound
+        // fixture I/O and await every PUT before issuing the LIST request.
+        stream::iter((0..dir_count).flat_map(|d| (0..files_per_dir).map(move |f| (d, f))))
+            .for_each_concurrent(FIXTURE_CONCURRENCY, |(d, f)| {
+                let client = &client;
+                async move {
+                    let key = format!("dir-{d:02}/file{f:03}.txt");
+                    client
+                        .put_object()
+                        .bucket(bucket)
+                        .key(&key)
+                        .body(ByteStream::from_static(b"x"))
+                        .send()
+                        .await
+                        .unwrap_or_else(|err| panic!("Failed to prepare fixture object {key}: {err:?}"));
+                }
+            })
+            .await;
+        eprintln!(
+            "Seeded {} objects in {bucket} in {:?}; starting ListObjectsV2 pagination",
+            dir_count * files_per_dir,
+            fixture_started.elapsed()
+        );
        let deadline = Instant::now() + PAGINATION_TIMEOUT;

        // With delimiter: 12 CommonPrefixes visible, all fit within capped 1000
+        let list_started = std::time::Instant::now();
        let output =
            listing_request_before(deadline, client.list_objects_v2().bucket(bucket).delimiter("/").max_keys(2000).send())
                .await
@@ -1077,6 +1091,15 @@ mod tests {
            dir_count,
            output.common_prefixes().len()
        );
+        let prefixes = output
+            .common_prefixes()
+            .iter()
+            .map(|entry| entry.prefix().expect("each CommonPrefix must carry its prefix").to_owned())
+            .collect::<Vec<_>>();
+        let expected_prefixes = (0..dir_count).map(|d| format!("dir-{d:02}/")).collect::<Vec<_>>();
+        assert_eq!(prefixes, expected_prefixes);
+        assert!(output.contents().is_empty(), "all fixture objects must collapse into prefixes");
+        assert_eq!(output.key_count(), Some(i32::try_from(dir_count).expect("fixture count fits i32")));
        assert_eq!(output.max_keys(), Some(1000));
        // 12 visible < 1000 capped MaxKeys → not truncated
        assert!(
@@ -1085,6 +1108,11 @@ mod tests {
            output.common_prefixes().len()
        );

+        assert!(
+            output.next_continuation_token().is_none(),
+            "a complete page must not carry a continuation token"
+        );
+        eprintln!("ListObjectsV2 assertions completed in {:?}", list_started.elapsed());
        info!("MaxKeys above limit with delimiter test passed");

        env.stop_server();
@@ -65,6 +65,130 @@ mod tests {
        Ok(())
    }

+    #[tokio::test]
+    async fn list_parts_sdk_paginator_terminates_and_preserves_completion() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
+    {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server(Vec::new()).await?;
+        let client = env.create_s3_client();
+        let bucket = "multipart-pagination";
+        let key = "sparse-parts.bin";
+        env.create_test_bucket(bucket).await?;
+        let upload = client.create_multipart_upload().bucket(bucket).key(key).send().await?;
+        let upload_id = upload.upload_id().ok_or("CreateMultipartUpload returned no upload ID")?;
+
+        let mut empty_pages = client
+            .list_parts()
+            .bucket(bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .into_paginator()
+            .stop_on_duplicate_token(false)
+            .send();
+        let empty = tokio::time::timeout(std::time::Duration::from_secs(30), empty_pages.next())
+            .await?
+            .ok_or("empty upload must return one page")??;
+        assert!(empty.parts().is_empty());
+        assert_eq!(empty.is_truncated(), Some(false));
+        assert_eq!(empty.next_part_number_marker(), None);
+        assert!(
+            tokio::time::timeout(std::time::Duration::from_secs(30), empty_pages.next())
+                .await?
+                .is_none()
+        );
+
+        let mut expected_body = Vec::new();
+        let mut completed_parts = Vec::new();
+        for (part_number, body) in [(1, vec![b'a'; PART_SIZE]), (3, vec![b'b'; PART_SIZE]), (10, b"tail".to_vec())] {
+            expected_body.extend_from_slice(&body);
+            let uploaded = client
+                .upload_part()
+                .bucket(bucket)
+                .key(key)
+                .upload_id(upload_id)
+                .part_number(part_number)
+                .body(ByteStream::from(body))
+                .send()
+                .await?;
+            completed_parts.push(
+                CompletedPart::builder()
+                    .part_number(part_number)
+                    .set_e_tag(uploaded.e_tag().map(str::to_owned))
+                    .build(),
+            );
+        }
+
+        for (page_size, expected_pages) in [(1, 3), (2, 2), (3, 1), (1000, 1)] {
+            let mut pages = client
+                .list_parts()
+                .bucket(bucket)
+                .key(key)
+                .upload_id(upload_id)
+                .into_paginator()
+                .page_size(page_size)
+                // The server must terminate pagination even without the SDK's duplicate-token safeguard.
+                .stop_on_duplicate_token(false)
+                .send();
+            let mut page_count = 0;
+            let mut listed_parts = Vec::new();
+            while let Some(page) = tokio::time::timeout(std::time::Duration::from_secs(30), pages.next()).await? {
+                assert!(page_count < expected_pages, "paginator must terminate after the final page");
+                let page = page?;
+                page_count += 1;
+                let truncated = page_count < expected_pages;
+                assert_eq!(page.is_truncated(), Some(truncated));
+                let last_part = page.parts().last().and_then(|part| part.part_number());
+                let expected_marker = truncated.then(|| last_part.expect("truncated page must contain parts").to_string());
+                assert_eq!(page.next_part_number_marker(), expected_marker.as_deref());
+                listed_parts.extend(
+                    page.parts()
+                        .iter()
+                        .map(|part| part.part_number().expect("part number must be present")),
+                );
+            }
+            assert_eq!(page_count, expected_pages);
+            assert_eq!(listed_parts, [1, 3, 10]);
+        }
+
+        let mut resumed_pages = client
+            .list_parts()
+            .bucket(bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .part_number_marker("2")
+            .into_paginator()
+            .page_size(1)
+            .stop_on_duplicate_token(false)
+            .send();
+        for (part_number, next_marker) in [(3, Some("3")), (10, None)] {
+            let page = tokio::time::timeout(std::time::Duration::from_secs(30), resumed_pages.next())
+                .await?
+                .ok_or("parts above a missing marker must still be listed")??;
+            assert_eq!(page.parts().len(), 1);
+            assert_eq!(page.parts()[0].part_number(), Some(part_number));
+            assert_eq!(page.is_truncated(), Some(next_marker.is_some()));
+            assert_eq!(page.next_part_number_marker(), next_marker);
+        }
+        assert!(
+            tokio::time::timeout(std::time::Duration::from_secs(30), resumed_pages.next())
+                .await?
+                .is_none()
+        );
+
+        client
+            .complete_multipart_upload()
+            .bucket(bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed_parts)).build())
+            .send()
+            .await?;
+        assert_completed_object(&client, bucket, key, "STANDARD", &expected_body).await?;
+        env.stop_server();
+        Ok(())
+    }
+
    #[tokio::test]
    async fn multipart_upload_preserves_standard_and_rrs_across_retry_and_resume()
    -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
@@ -52,6 +52,9 @@ const VERSIONED_BUCKET: &str = "upgrade-versioned-data";
 const MIXED_BUCKET: &str = "upgrade-mixed-version-data";
 const MIXED_NODE_COUNT: usize = 4;
 const UPGRADE_READINESS_BODY: &[u8] = b"upgrade write readiness";
+// Written by the "previous-seed" readiness probe through node 0 before any
+// current binary starts; every later phase must still read it.
+const PREVIOUS_RELEASE_SEED_KEY: &str = ".upgrade-readiness/previous-seed/node-0";
 const MULTIPART_WORKERS: usize = 16;
 const MULTIPART_UPLOADS_PER_WORKER: usize = 16;
 // Peers keep a restarted node's drive in Suspect/Returning for roughly
@@ -68,6 +71,7 @@ const CONFIG_REPLICA_BUCKET: &str = "upgrade-config-replica";
 const ROLLBACK_BUCKET: &str = "rollback-config-data";
 const ROLLBACK_REPLICA_BUCKET: &str = "rollback-config-replica";
 const BUCKET_QUOTA_BYTES: u64 = 64 * 1024 * 1024;
+const QUOTA_USAGE_READINESS_TIMEOUT: Duration = Duration::from_secs(90);
 const LIFECYCLE_RULE_ID: &str = "upgrade-expire-logs";
 const LIFECYCLE_PREFIX: &str = "logs/";
 const LIFECYCLE_DAYS: i32 = 30;
@@ -425,6 +429,11 @@ async fn exercise_mixed_cluster(
    let multipart_keys = write_multipart_load(&clients, phase).await?;
    let expected_count = multipart_keys.len() + 2;
    for (label, client) in [("current", current_client), ("previous", previous_client)] {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "{phase}: the {label} node must retain the previous-release seed"
+        );
        wait_for_phase_listing(
            client,
            phase,
@@ -523,9 +532,7 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro
    wait_for_upgrade_write_readiness(&clients, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
    for (reader, client) in clients.iter().enumerate() {
        assert_eq!(
-            read_object(client, MIXED_BUCKET, ".upgrade-readiness/previous-seed/node-0", None)
-                .await?
-                .1,
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
            UPGRADE_READINESS_BODY,
            "previous-release node {reader} must retain the seed across its rolling restart"
        );
@@ -935,6 +942,11 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
    cluster.start_node_from_binary(3, &current_binary).await?;

    for (node_idx, client) in cluster.create_all_clients()?.iter().enumerate() {
+        assert_eq!(
+            read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
+            UPGRADE_READINESS_BODY,
+            "current node {node_idx} must retain the previous-release seed"
+        );
        for phase in ["one-current-node", "one-previous-node"] {
            wait_for_phase_listing(
                client,
@@ -962,7 +974,8 @@ async fn rolling_upgrade_from_rc2_preserves_mixed_version_contracts() -> TestRes
                "assertions": [
                    "current node reads objects written through previous-release client",
                    "previous-release node reads objects written through current client",
-                    "all nodes list every mixed-version object after homogeneous-current convergence"
+                    "all nodes list every mixed-version object after homogeneous-current convergence",
+                    "the previous-release seed survives the baseline restart and every upgrade phase"
                ],
            }),
        )?;
@@ -1067,12 +1080,23 @@ async fn put_object_through_quota_warmup(client: &Client, bucket: &str, key: &st

 async fn get_bucket_quota(env: &RustFSTestEnvironment, bucket: &str) -> Result<Option<u64>, BoxError> {
    let path = format!("/rustfs/admin/v3/quota/{bucket}");
-    let (status, response) = admin_request(&env.url, Method::GET, &path, None, &env.access_key, &env.secret_key).await?;
-    if status != StatusCode::OK {
-        return Err(format!("reading the quota of {bucket} failed: {status} {response}").into());
+    let deadline = Instant::now() + QUOTA_USAGE_READINESS_TIMEOUT;
+    loop {
+        let (status, response) = tokio::time::timeout_at(
+            deadline,
+            admin_request(&env.url, Method::GET, &path, None, &env.access_key, &env.secret_key),
+        )
+        .await
+        .map_err(|_| format!("reading the quota of {bucket} exceeded the usage-readiness timeout"))??;
+        if status == StatusCode::OK {
+            let quota: serde_json::Value = serde_json::from_str(&response)?;
+            return Ok(quota.get("quota").and_then(serde_json::Value::as_u64));
+        }
+        if status != StatusCode::SERVICE_UNAVAILABLE || Instant::now() >= deadline {
+            return Err(format!("reading the quota of {bucket} failed: {status} {response}").into());
+        }
+        sleep(Duration::from_millis(500)).await;
    }
-    let quota: serde_json::Value = serde_json::from_str(&response)?;
-    Ok(quota.get("quota").and_then(serde_json::Value::as_u64))
 }

 /// `GET /rustfs/admin/v3/list-remote-targets?bucket=...`.
@@ -208,7 +208,6 @@ aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a",
 urlencoding = { workspace = true }
 smallvec = { workspace = true, features = ["serde"] }
 shadow-rs = { workspace = true, default-features = false }
-async-recursion.workspace = true
 aws-credential-types = { workspace = true }
 aws-smithy-types = { workspace = true }
 aws-smithy-runtime-api = { workspace = true, features = ["http-1x"] }
@@ -5606,6 +5606,11 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(

    //debug!("lc_event.action: {:?}", lc_event.action);
    debug!("expiry_on_non_transitioned_objects opts: {:?}", opts);
+    let usage_accounting = if lc_event.action == IlmAction::DeleteAction && !versioned && !version_suspended && !oi.is_dir {
+        Some(crate::data_usage::begin_expiry_usage_accounting(&oi.bucket).await)
+    } else {
+        None
+    };
    let mut dobj = match api
        .delete_object_with_tier_delete_journal(&oi.bucket, &encode_dir_object(&oi.name), opts)
        .await
@@ -5625,6 +5630,14 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
            return false;
        }
    };
+    if let Some(accounting) = usage_accounting
+        && !dobj.name.is_empty()
+        && !dobj.delete_marker
+        && !dobj.is_dir
+        && let Ok(size) = crate::data_usage::quota_object_size(&dobj)
+    {
+        accounting.commit(size).await;
+    }
    schedule_lifecycle_replication_delete_if_needed(oi, &dobj).await;

    // The object (or all its versions, for delete_all) was expired; evict any
@@ -12891,6 +12904,130 @@ mod tests {
        );
    }

+    #[tokio::test]
+    #[serial]
+    async fn ordinary_expiry_accounts_the_committed_receipt_once() {
+        use crate::bucket::quota::{QuotaOperation, checker::QuotaChecker};
+        use crate::data_usage::{apply_bucket_usage_memory_overlay, replace_bucket_usage_memory_from_info};
+        use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
+
+        let (_disk_paths, ecstore) = setup_test_env().await;
+        for size in [0usize, 42] {
+            let bytes = u64::try_from(size).expect("fixture size fits");
+            let bucket = format!("expiry-accounting-{}", Uuid::new_v4().simple());
+            create_test_bucket(&ecstore, &bucket).await;
+            let mut reader = PutObjReader::from_vec(vec![0; size]);
+            let mut queued = ecstore
+                .put_object(&bucket, "expire", &mut reader, &ObjectOptions::default())
+                .await
+                .expect("create expiring object");
+            let mut reader = PutObjReader::from_vec(vec![0; size]);
+            ecstore
+                .put_object(&bucket, "keep", &mut reader, &ObjectOptions::default())
+                .await
+                .expect("create retained object");
+            let mut baseline = DataUsageInfo {
+                last_update: Some(std::time::SystemTime::now()),
+                usage_snapshot_complete: true,
+                buckets_count: 1,
+                ..Default::default()
+            };
+            baseline.buckets_usage.insert(
+                bucket.clone(),
+                BucketUsageInfo {
+                    objects_count: 2,
+                    versions_count: 2,
+                    size: bytes * 2,
+                    ..Default::default()
+                },
+            );
+            baseline.bucket_sizes.insert(bucket.clone(), bytes * 2);
+            baseline.calculate_totals();
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            // Legacy quota admission consumes the same conservative overlay.
+            // Durable reservations have their own storage-commit accounting.
+            metadata_sys::update(
+                &bucket,
+                rustfs_config::QUOTA_CONFIG_FILE,
+                format!(r#"{{"quota":{},"quota_type":"Hard"}}"#, bytes * 2).into_bytes(),
+            )
+            .await
+            .expect("configure the legacy quota");
+            let checker =
+                QuotaChecker::new(Arc::new(tokio::sync::RwLock::new(metadata_sys::BucketMetadataSys::new(ecstore.clone()))));
+            let attempted_size = bytes.max(1);
+            let quota = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                .await
+                .expect("check quota before expiration");
+            assert!(!quota.allowed);
+            assert!(!quota.uses_durable_reservations);
+            assert_eq!(quota.current_usage, Some(bytes * 2));
+            // Queued metadata is not the accounting receipt from the storage commit.
+            queued.size = 1;
+            let event = lifecycle::Event {
+                action: IlmAction::DeleteAction,
+                ..Default::default()
+            };
+            let incarnation = ecstore
+                .bucket_incarnation_id_from_disk(&bucket)
+                .await
+                .expect("bucket incarnation");
+            for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
+                assert_eq!(
+                    super::apply_expiry_on_non_transitioned_objects(
+                        ecstore.clone(),
+                        &queued,
+                        &event,
+                        &LcEventSrc::Scanner,
+                        attempt_incarnation,
+                    )
+                    .await,
+                    expected
+                );
+                let mut response = baseline.clone();
+                apply_bucket_usage_memory_overlay(&mut response).await;
+                let remaining = if attempt_incarnation == incarnation { 1 } else { 2 };
+                assert_eq!(response.buckets_usage[&bucket].objects_count, remaining);
+                assert_eq!(response.buckets_usage[&bucket].size, remaining * bytes);
+                let quota = checker
+                    .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                    .await
+                    .expect("check quota before scanner confirmation");
+                assert!(!quota.allowed, "expiry must not release unconfirmed quota credit");
+                assert_eq!(quota.current_usage, Some(bytes * 2));
+            }
+            let mut confirmed = baseline.clone();
+            confirmed.last_update = Some(std::time::SystemTime::now());
+            confirmed.scanner_epoch = Some(7);
+            confirmed.scanner_cycle = Some(10);
+            let usage = confirmed.buckets_usage.get_mut(&bucket).expect("bucket is in the snapshot");
+            usage.size = bytes;
+            usage.objects_count = 1;
+            usage.versions_count = 0; // The scanner does not count unversioned objects as versions.
+            confirmed.bucket_sizes.insert(bucket.clone(), bytes);
+            confirmed.calculate_totals();
+            replace_bucket_usage_memory_from_info(&confirmed).await;
+            let quota = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, attempted_size)
+                .await
+                .expect("check quota after scanner confirmation");
+            assert_eq!(quota.allowed, bytes > 0);
+            assert_eq!(quota.current_usage, Some(bytes));
+            let too_large = checker
+                .check_quota(&bucket, QuotaOperation::PutObject, bytes + 1)
+                .await
+                .expect("check the first byte above the quota limit");
+            assert!(!too_large.allowed);
+            assert!(
+                ecstore
+                    .get_object_info(&bucket, "keep", &ObjectOptions::default())
+                    .await
+                    .is_ok()
+            );
+        }
+    }
+
    #[tokio::test]
    #[serial]
    async fn queued_delete_all_rechecks_a_same_id_rule_moved_into_the_future() {
@@ -56,7 +56,7 @@ impl DiagnosticPacing {
        // Reservation is never refunded: a failed or dropped RPC may have sent some payload.
        let previous = self
            .charged_bytes
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |charged| {
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |charged| {
                charged.checked_add(bytes).filter(|total| *total <= MAX_NETWORK_PROBE_BYTES)
            })
            .map_err(|_| NetworkPeerProbeError::LimitExceeded)?;
@@ -12976,7 +12976,7 @@ impl ECStore {
                .targets
                .iter()
                .find(|target| target.pool_index == permitted_target_pool_index)
-                .and_then(&candidate)
+                .and_then(candidate)
            {
                return Ok(pool_index);
            }
@@ -19858,9 +19858,9 @@ mod tests {
                .reload_pool_meta()
                .await
                .expect("load the active generation on the remote node");
-            other_store
-                .decommission_cancel(0)
+            tokio::time::timeout(std::time::Duration::from_secs(30), other_store.decommission_cancel(0))
                .await
+                .expect("remote cancel must persist pool metadata without re-entering its state locks")
                .expect("the remote node should cancel the old generation");
            other_store
                .clear_decommission(0)
@@ -20125,9 +20125,9 @@ mod tests {
            .expect("the stale operation should have a valid terminal fence plan")
            .expect("the stale operation should have an active reservation");

-        first_node
-            .decommission_cancel(0)
+        tokio::time::timeout(std::time::Duration::from_secs(30), first_node.decommission_cancel(0))
            .await
+            .expect("cancel must persist pool metadata without re-entering its state locks")
            .expect("cancel the first durable operation");
        first_node
            .clear_decommission(0)
@@ -20630,8 +20630,9 @@ mod tests {
            .await
            .expect("second target mutation task should not panic")
            .expect("second target mutation should finalize");
-        cancel
+        tokio::time::timeout(std::time::Duration::from_secs(30), cancel)
            .await
+            .expect("cancel must finish after target mutations release their gates")
            .expect("decommission cancellation task should not panic")
            .expect("decommission cancellation should finish after target mutations");

@@ -306,7 +306,7 @@ pub mod test_util {
        let Some(fault) = fault else { return Ok(None) };
        if fault
            .remaining
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |remaining| remaining.checked_sub(1))
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |remaining| remaining.checked_sub(1))
            != Ok(1)
        {
            return Ok(None);
@@ -43,7 +43,7 @@ use std::{
    future::Future,
    sync::{
        Arc, LazyLock, OnceLock,
-        atomic::{AtomicU64, Ordering},
+        atomic::{AtomicU64, AtomicUsize, Ordering},
    },
    time::{Duration, SystemTime},
 };
@@ -67,12 +67,14 @@ struct CachedBucketUsage {
    usage_updated_at: SystemTime,
    // Set by request-path mutations until a scanner snapshot catches up to the same core counts.
    dirty: bool,
-    // Set when a newer scanner snapshot was observed but did not include the dirty counts yet.
+    // Set when an observed snapshot cannot replace this absolute baseline.
    stale_snapshot_pending: bool,
    // First complete scanner generation observed after an unknown-baseline
    // mutation. A strictly later generation is required before the mutation
    // evidence can be discarded.
    pending_scanner_position: Option<(u64, u64)>,
+    // Generation that supplied this absolute baseline, retained across mutations.
+    scanner_position: Option<(u64, u64)>,
    // Deletes are visible to admin immediately, but quota admission keeps
    // them pending until a complete scanner generation reconciles the set.
    // This marker intentionally remains process-local: the delete request
@@ -83,6 +85,79 @@ struct CachedBucketUsage {
    // necessarily a complete scanner reconciliation and therefore creates a
    // fresh cache entry with no pending hold.
    pending_negative_delta: u64,
+    pending_negative_updated_at: Option<SystemTime>,
+    pending_negative_reconciliation: Option<PendingDeleteReconciliation>,
+    in_flight_expirations: Arc<AtomicUsize>,
+}
+
+#[derive(Debug, Clone)]
+struct PendingDeleteReconciliation {
+    bytes: u64,
+    deleted_at: SystemTime,
+    observed: Option<((u64, u64), SystemTime)>,
+}
+
+impl CachedBucketUsage {
+    fn reconcile_pending_deletes(&mut self, position: Option<(u64, u64)>, updated_at: SystemTime) {
+        let Some(position) = position else { return };
+        let Some(pending) = self.pending_negative_reconciliation.as_mut() else { return };
+        match pending.observed {
+            None if updated_at >= pending.deleted_at => pending.observed = Some((position, updated_at)),
+            Some((previous, observed_at)) if position.0 >= previous.0 && position.1 > previous.1 && updated_at >= observed_at => {
+                self.pending_negative_delta = self.pending_negative_delta.saturating_sub(pending.bytes);
+                self.pending_negative_reconciliation = self
+                    .pending_negative_updated_at
+                    .filter(|_| self.pending_negative_delta > 0)
+                    .map(|deleted_at| PendingDeleteReconciliation {
+                        bytes: self.pending_negative_delta,
+                        deleted_at,
+                        observed: (updated_at >= deleted_at).then_some((position, updated_at)),
+                    });
+            }
+            _ => {}
+        }
+    }
+}
+
+/// Keeps a scanner refresh from including a delete before its memory receipt.
+pub(crate) struct ExpiryUsageAccounting<'a> {
+    bucket: &'a str,
+    in_flight: Arc<AtomicUsize>,
+}
+
+impl ExpiryUsageAccounting<'_> {
+    pub(crate) async fn commit(self, deleted_size: u64) {
+        let mut cache = memory_cache().write().await;
+        if let Some(entry) = cache.get_mut(self.bucket)
+            && Arc::ptr_eq(&entry.in_flight_expirations, &self.in_flight)
+        {
+            apply_bucket_object_delete_memory(entry, deleted_size, true);
+        }
+    }
+}
+
+impl Drop for ExpiryUsageAccounting<'_> {
+    fn drop(&mut self) {
+        self.in_flight.fetch_sub(1, Ordering::AcqRel);
+    }
+}
+
+pub(crate) async fn begin_expiry_usage_accounting(bucket: &str) -> ExpiryUsageAccounting<'_> {
+    ensure_bucket_usage_cached(bucket).await;
+    let mut cache = memory_cache().write().await;
+    // The caller owns the bucket name through commit. Avoid allocating either
+    // a receipt name or a map key for every expiry in an already cached bucket.
+    let in_flight = match cache.get(bucket) {
+        Some(entry) => Arc::clone(&entry.in_flight_expirations),
+        None => {
+            let entry = cached_bucket_usage_now(BucketUsageInfo::default());
+            let in_flight = Arc::clone(&entry.in_flight_expirations);
+            cache.insert(bucket.to_owned(), entry);
+            in_flight
+        }
+    };
+    in_flight.fetch_add(1, Ordering::AcqRel);
+    ExpiryUsageAccounting { bucket, in_flight }
 }

 type UsageMemoryCache = Arc<RwLock<HashMap<String, CachedBucketUsage>>>;
@@ -1944,7 +2019,11 @@ fn cached_bucket_usage_from_backend(usage: BucketUsageInfo, updated_at: SystemTi
        dirty: false,
        stale_snapshot_pending: false,
        pending_scanner_position: None,
+        scanner_position: None,
        pending_negative_delta: 0,
+        pending_negative_updated_at: None,
+        pending_negative_reconciliation: None,
+        in_flight_expirations: Arc::new(AtomicUsize::new(0)),
    }
 }

@@ -1958,7 +2037,11 @@ fn cached_bucket_usage_now(usage: BucketUsageInfo) -> CachedBucketUsage {
        dirty: false,
        stale_snapshot_pending: false,
        pending_scanner_position: None,
+        scanner_position: None,
        pending_negative_delta: 0,
+        pending_negative_updated_at: None,
+        pending_negative_reconciliation: None,
+        in_flight_expirations: Arc::new(AtomicUsize::new(0)),
    }
 }

@@ -1974,27 +2057,26 @@ fn bucket_usage_counts_match(left: &BucketUsageInfo, right: &BucketUsageInfo) ->
 }

 fn preserve_unknown_dirty_usage(
-    existing: &CachedBucketUsage,
+    existing: &mut CachedBucketUsage,
    snapshot_position: Option<(u64, u64)>,
    snapshot_update: SystemTime,
-) -> Option<CachedBucketUsage> {
+) -> bool {
    if existing.authoritative || !existing.dirty {
-        return None;
+        return false;
    }
    if existing
        .pending_scanner_position
        .zip(snapshot_position)
        .is_some_and(|(previous, current)| current.0 > previous.0 || (current.0 == previous.0 && current.1 > previous.1))
    {
-        return None;
+        return false;
    }

-    let mut preserved = existing.clone();
-    preserved.stale_snapshot_pending = true;
-    if preserved.pending_scanner_position.is_none() && snapshot_update >= existing.usage_updated_at {
-        preserved.pending_scanner_position = snapshot_position;
+    existing.stale_snapshot_pending = true;
+    if existing.pending_scanner_position.is_none() && snapshot_update >= existing.usage_updated_at {
+        existing.pending_scanner_position = snapshot_position;
    }
-    Some(preserved)
+    true
 }

 #[cfg(any(test, feature = "test-util"))]
@@ -2110,6 +2192,10 @@ pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64,
        .entry(bucket.to_string())
        .or_insert_with(|| cached_bucket_usage_now(BucketUsageInfo::default()));

+    apply_bucket_object_delete_memory(entry, deleted_size, removed_current_object);
+}
+
+fn apply_bucket_object_delete_memory(entry: &mut CachedBucketUsage, deleted_size: u64, removed_current_object: bool) {
    entry.usage.size = entry.usage.size.saturating_sub(deleted_size);
    entry.pending_negative_delta = entry.pending_negative_delta.saturating_add(deleted_size);
    if removed_current_object {
@@ -2118,6 +2204,16 @@ pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64,
    }

    let now = SystemTime::now();
+    entry.pending_negative_updated_at = Some(now);
+    // Freeze this tranche even before its first observation. Later deletes
+    // must not keep moving the timestamp that the scanner needs to catch up to.
+    if entry.pending_negative_reconciliation.is_none() && entry.pending_negative_delta > 0 {
+        entry.pending_negative_reconciliation = Some(PendingDeleteReconciliation {
+            bytes: entry.pending_negative_delta,
+            deleted_at: now,
+            observed: None,
+        });
+    }
    entry.refreshed_at = now;
    entry.usage_updated_at = now;
    entry.dirty = true;
@@ -2245,23 +2341,48 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D

    let usage_updated_at = data_usage_info_updated_at(data_usage_info);
    let snapshot_position = data_usage_info.scanner_epoch.zip(data_usage_info.scanner_cycle);
+    let snapshot_converged = data_usage_info.usage_snapshot_converged == Some(true) && !data_usage_info.usage_snapshot_partial;
    let mut next_cache = HashMap::with_capacity(data_usage_info.buckets_usage.len());
    for (bucket, bucket_usage) in data_usage_info.buckets_usage.iter() {
-        next_cache.insert(
-            bucket.clone(),
-            cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at, true),
-        );
+        let mut entry = cached_bucket_usage_from_backend(bucket_usage.clone(), usage_updated_at, true);
+        entry.scanner_position = snapshot_position;
+        next_cache.insert(bucket.clone(), entry);
    }

    let mut cache = memory_cache().write().await;
    if expected_generation.is_some_and(|expected| usage_memory_generation() != expected) {
        return;
    }
-    for (bucket, existing) in cache.iter() {
-        match next_cache.entry(bucket.clone()) {
+    // Move preserved entries instead of cloning their histograms under the lock.
+    for (bucket, mut existing) in std::mem::take(&mut *cache) {
+        let advances_baseline = match (existing.scanner_position, snapshot_position) {
+            (Some(previous), Some(current)) if current.0 >= previous.0 && current.1 >= previous.1 => current.1 > previous.1,
+            (None, _) => true,
+            _ => {
+                existing.stale_snapshot_pending = true;
+                next_cache.insert(bucket, existing);
+                continue;
+            }
+        };
+        existing.reconcile_pending_deletes(snapshot_position, usage_updated_at);
+        if existing.in_flight_expirations.load(Ordering::Acquire) > 0 {
+            next_cache.insert(bucket, existing);
+            continue;
+        }
+        // Both convergence and the two-observation fallback must advance the
+        // baseline's cycle. Epoch-only changes cannot prove a new scan, and
+        // the timestamp check below still protects post-scan mutations.
+        let reconciled_dirty_usage = advances_baseline
+            && snapshot_position.is_some_and(|current| {
+                snapshot_converged
+                    || existing
+                        .pending_scanner_position
+                        .is_some_and(|previous| current.0 >= previous.0 && current.1 > previous.1)
+            });
+        match next_cache.entry(bucket) {
            Entry::Occupied(mut candidate) => {
-                if let Some(preserved) = preserve_unknown_dirty_usage(existing, snapshot_position, usage_updated_at) {
-                    candidate.insert(preserved);
+                if preserve_unknown_dirty_usage(&mut existing, snapshot_position, usage_updated_at) {
+                    candidate.insert(existing);
                    continue;
                }
                if existing.authoritative && existing.dirty && bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
@@ -2272,37 +2393,44 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
                    continue;
                }
                if existing.authoritative && existing.usage_updated_at > usage_updated_at {
-                    candidate.insert(existing.clone());
+                    candidate.insert(existing);
                    continue;
                }
-                if existing.authoritative && existing.dirty && !bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
+                if existing.authoritative
+                    && existing.dirty
+                    && !reconciled_dirty_usage
+                    && !bucket_usage_counts_match(&existing.usage, &candidate.get().usage)
                {
                    // A scanner snapshot can be saved after newer writes but still miss them if it listed the bucket earlier.
-                    let mut preserved = existing.clone();
+                    let mut preserved = existing;
                    if preserved.pending_negative_delta > 0 && preserved.usage.size == candidate.get().usage.size {
                        // The complete scanner snapshot has reconciled the
                        // post-delete byte total. Keep the request-path object
                        // counts until they also converge, but release the
                        // conservative quota hold.
                        preserved.pending_negative_delta = 0;
+                        preserved.pending_negative_reconciliation = None;
                    }
                    preserved.stale_snapshot_pending = true;
+                    preserved.pending_scanner_position = preserved.pending_scanner_position.or(snapshot_position);
                    candidate.insert(preserved);
                }
            }
            Entry::Vacant(candidate) => {
-                if let Some(preserved) = preserve_unknown_dirty_usage(existing, snapshot_position, usage_updated_at) {
-                    candidate.insert(preserved);
-                } else if existing.authoritative && existing.usage_updated_at > usage_updated_at {
-                    candidate.insert(existing.clone());
-                } else if existing.authoritative && existing.dirty {
-                    let mut preserved = existing.clone();
+                if preserve_unknown_dirty_usage(&mut existing, snapshot_position, usage_updated_at)
+                    || (existing.authoritative && existing.usage_updated_at > usage_updated_at)
+                {
+                    candidate.insert(existing);
+                } else if existing.authoritative && existing.dirty && !reconciled_dirty_usage {
+                    let mut preserved = existing;
                    if preserved.pending_negative_delta > 0 && preserved.usage.size == 0 {
                        // Omitting the bucket from a complete snapshot confirms
                        // that its post-delete byte total is zero.
                        preserved.pending_negative_delta = 0;
+                        preserved.pending_negative_reconciliation = None;
                    }
                    preserved.stale_snapshot_pending = true;
+                    preserved.pending_scanner_position = preserved.pending_scanner_position.or(snapshot_position);
                    candidate.insert(preserved);
                }
            }
@@ -4352,6 +4480,7 @@ mod tests {
        let mut first_snapshot = data_usage_info_for_test(bucket, 10, 420, mutation_update + Duration::from_nanos(1));
        first_snapshot.scanner_epoch = Some(7);
        first_snapshot.scanner_cycle = Some(10);
+        first_snapshot.usage_snapshot_converged = Some(true);
        replace_bucket_usage_memory_from_info(&first_snapshot).await;

        let mut first_response = first_snapshot.clone();
@@ -5405,11 +5534,12 @@ mod tests {
            .expect("delete usage should remain cached")
            .usage_updated_at;

-        let stale = data_usage_info_for_test("bucket-a", 2, 768, mutation_update + Duration::from_nanos(1));
+        // Windows SystemTime has 100 ns resolution; retain strict ordering there.
+        let stale = data_usage_info_for_test("bucket-a", 2, 768, mutation_update + Duration::from_micros(1));
        replace_bucket_usage_memory_from_info(&stale).await;
        assert_eq!(get_bucket_usage_memory("bucket-a").await, Some(768));

-        let mut too_old_matching = data_usage_info_for_test("bucket-a", 1, 256, mutation_update - Duration::from_nanos(1));
+        let mut too_old_matching = data_usage_info_for_test("bucket-a", 1, 256, mutation_update - Duration::from_micros(1));
        too_old_matching
            .buckets_usage
            .get_mut("bucket-a")
@@ -5423,7 +5553,7 @@ mod tests {
            "a matching byte total from before the delete cannot release the hold"
        );

-        let mut reconciled = data_usage_info_for_test("bucket-a", 1, 256, mutation_update + Duration::from_nanos(2));
+        let mut reconciled = data_usage_info_for_test("bucket-a", 1, 256, mutation_update + Duration::from_micros(2));
        reconciled
            .buckets_usage
            .get_mut("bucket-a")
@@ -5759,6 +5889,503 @@ mod tests {
        );
    }

+    #[tokio::test]
+    #[serial]
+    async fn complete_scanner_cycles_reconcile_expired_dirty_usage() {
+        for (remaining_objects, omit_bucket) in [(0, false), (1, false), (0, true)] {
+            clear_usage_memory_cache_for_test().await;
+            let bucket = "expired-dirty-usage";
+            let baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+
+            // Lifecycle deletion bypasses request-layer memory accounting.
+            // Repeated complete scans must eventually replace that dirty overlay.
+            for cycle in 1..=3 {
+                let mut snapshot = data_usage_info_for_test(bucket, remaining_objects, remaining_objects * 42, SystemTime::now());
+                snapshot.scanner_epoch = Some(7);
+                snapshot.scanner_cycle = Some(cycle);
+                if omit_bucket {
+                    snapshot.buckets_usage.clear();
+                    snapshot.bucket_sizes.clear();
+                    snapshot.buckets_count = 0;
+                    snapshot.calculate_totals();
+                }
+                replace_bucket_usage_memory_from_info(&snapshot).await;
+                if cycle == 3 {
+                    apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+                    assert_eq!(snapshot.objects_total_count, remaining_objects);
+                    assert_eq!(snapshot.objects_total_size, remaining_objects * 42);
+                    if omit_bucket {
+                        assert!(!snapshot.buckets_usage.contains_key(bucket));
+                    } else {
+                        assert_eq!(snapshot.buckets_usage[bucket].objects_count, remaining_objects);
+                        assert_eq!(snapshot.buckets_usage[bucket].size, remaining_objects * 42);
+                    }
+                }
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_receipts_exclude_refresh_until_committed_or_cancelled() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-receipt";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 2, 84, SystemTime::now())).await;
+        let first = begin_expiry_usage_accounting(bucket).await;
+        let cancelled = begin_expiry_usage_accounting(bucket).await;
+        let mut after_delete = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        after_delete.scanner_epoch = Some(7);
+        after_delete.scanner_cycle = Some(10);
+        after_delete.usage_snapshot_converged = Some(true);
+        replace_bucket_usage_memory_from_info(&after_delete).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 84);
+        first.commit(42).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 42);
+
+        let mut omitted = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        omitted.buckets_usage.clear();
+        omitted.bucket_sizes.clear();
+        omitted.buckets_count = 0;
+        omitted.calculate_totals();
+        replace_bucket_usage_memory_from_info(&omitted).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.size, 42);
+        drop(cancelled);
+        assert_eq!(
+            memory_cache().read().await[bucket]
+                .in_flight_expirations
+                .load(Ordering::Acquire),
+            0
+        );
+
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 1, 42, SystemTime::now())).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn cancelled_expiry_receipt_reconciles_without_another_mutation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-cancelled-after-delete";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 0, 0, SystemTime::now())).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let receipt = begin_expiry_usage_accounting(bucket).await;
+
+        // Storage has deleted one object, but the task has not committed its
+        // memory receipt. Even two complete scans must not replace its baseline.
+        let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        scanned.scanner_epoch = Some(7);
+        for cycle in [10, 11] {
+            scanned.scanner_cycle = Some(cycle);
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+        }
+        drop(receipt);
+
+        for (cycle, expected) in [(12, 84), (13, 42)] {
+            scanned.scanner_cycle = Some(cycle);
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(expected));
+            let mut displayed = scanned.clone();
+            apply_bucket_usage_memory_overlay(&mut displayed).await;
+            assert_eq!(displayed.buckets_usage[bucket].size, expected);
+            assert_eq!(displayed.buckets_usage[bucket].objects_count, expected / 42);
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_restart_reloads_durable_usage_without_persisting_the_overlay() {
+        use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
+
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        let bucket = "expiry-restart";
+        clear_usage_memory_cache_for_test().await;
+        let baseline = data_usage_info_for_test(bucket, 2, 84, SystemTime::now());
+        store_data_usage_in_backend(baseline.clone(), store.clone())
+            .await
+            .expect("persist the pre-expiration scanner snapshot");
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        begin_expiry_usage_accounting(bucket).await.commit(42).await;
+        let mut displayed = baseline.clone();
+        apply_bucket_usage_memory_overlay(&mut displayed).await;
+        assert_eq!(displayed.buckets_usage[bucket].size, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+
+        // Recreate only process-local usage state, then use the real backend
+        // loader. This is a restart-state regression, not a process restart E2E.
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+        let recovered = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("reload the durable pre-expiration snapshot");
+        assert_eq!(recovered.buckets_usage[bucket].size, 84);
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+
+        let confirmed = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        store_data_usage_in_backend(confirmed, store.clone())
+            .await
+            .expect("persist scanner confirmation without another object mutation");
+        let recovered = load_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("reload the confirmed snapshot");
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+
+        // Historical overlay drift must not contaminate the persisted scan.
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+        let mut recovered = load_data_usage_from_backend_cached(store)
+            .await
+            .expect("reload the confirmed snapshot after losing the stale overlay");
+        replace_bucket_usage_memory_from_info(&recovered).await;
+        apply_bucket_usage_memory_overlay(&mut recovered).await;
+        assert_eq!(recovered.buckets_usage[bucket].objects_count, 1);
+        assert_eq!(recovered.buckets_usage[bucket].size, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        clear_usage_memory_cache_for_test().await;
+        invalidate_data_usage_snapshot_cache().await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn expiry_receipt_cannot_credit_a_replaced_bucket_cache() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-recreated";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 1, 42, SystemTime::now())).await;
+        let receipt = begin_expiry_usage_accounting(bucket).await;
+        clear_usage_memory_cache_for_test().await;
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 2, 84, SystemTime::now())).await;
+        receipt.commit(42).await;
+        assert_eq!(memory_cache().read().await[bucket].usage.objects_count, 2);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_delete_tranches_advance_despite_newer_mutations() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-busy-quota";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 10, 420, SystemTime::now())).await;
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        let mut observation = data_usage_info_for_test(bucket, 9, 378, SystemTime::now());
+        observation.scanner_epoch = Some(7);
+        observation.scanner_cycle = Some(10);
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        replace_bucket_usage_memory_from_info(&observation).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(462));
+
+        for (epoch, cycle, partial, old_time) in [
+            (8, 10, false, false),
+            (6, 11, false, false),
+            (7, 11, true, false),
+            (7, 11, false, true),
+        ] {
+            let mut rejected = observation.clone();
+            rejected.scanner_epoch = Some(epoch);
+            rejected.scanner_cycle = Some(cycle);
+            rejected.usage_snapshot_complete = !partial;
+            rejected.usage_snapshot_partial = partial;
+            if old_time {
+                rejected.last_update = Some(SystemTime::UNIX_EPOCH);
+            }
+            replace_bucket_usage_memory_from_info(&rejected).await;
+            assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 42);
+        }
+
+        observation.last_update = Some(SystemTime::now());
+        observation.scanner_cycle = Some(11);
+        // A newer delete must stay charged, without delaying the older receipt.
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        replace_bucket_usage_memory_from_info(&observation).await;
+        assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 42);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(462));
+
+        for cycle in [12, 13] {
+            observation.last_update = Some(SystemTime::now());
+            observation.scanner_cycle = Some(cycle);
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            replace_bucket_usage_memory_from_info(&observation).await;
+        }
+        assert_eq!(memory_cache().read().await[bucket].pending_negative_delta, 0);
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(504));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn quota_delete_hold_stays_bounded_with_deletes_after_every_observation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "expiry-continuous-quota";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 10, 420, SystemTime::now())).await;
+        record_bucket_object_delete_memory(bucket, 42, true).await;
+        for cycle in 1..=12 {
+            let mut observed = data_usage_info_for_test(bucket, 9, 378, SystemTime::now());
+            observed.scanner_epoch = Some(7);
+            observed.scanner_cycle = Some(cycle);
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            record_bucket_object_delete_memory(bucket, 42, true).await;
+            replace_bucket_usage_memory_from_info(&observed).await;
+            let cache = memory_cache().read().await;
+            assert_eq!(cache[bucket].usage.size, 378);
+            assert!(
+                cache[bucket].pending_negative_delta <= 126,
+                "confirmed older deletes must leave the quota hold"
+            );
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn dirty_usage_reconciliation_requires_a_later_complete_observation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "dirty-observation-fence";
+        let baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mutation_at = memory_cache().read().await[bucket].usage_updated_at;
+        let mut first = data_usage_info_for_test(bucket, 0, 0, mutation_at);
+        first.scanner_epoch = Some(7);
+        first.scanner_cycle = Some(10);
+        let mut partial = first.clone();
+        partial.usage_snapshot_partial = true;
+        partial.usage_snapshot_complete = false;
+        replace_bucket_usage_memory_from_info(&partial).await;
+        assert_eq!(memory_cache().read().await[bucket].pending_scanner_position, None);
+        replace_bucket_usage_memory_from_info(&first).await;
+
+        for (epoch, cycle, partial, update) in [
+            (Some(7), Some(10), false, mutation_at),
+            (Some(8), Some(10), false, mutation_at),
+            (Some(6), Some(11), false, mutation_at),
+            (Some(7), Some(9), false, mutation_at),
+            (Some(7), Some(11), true, mutation_at),
+            (None, Some(11), false, mutation_at),
+            (Some(7), None, false, mutation_at),
+            // Windows `SystemTime` stores 100 ns intervals, so a 1 ns offset is a no-op there.
+            (Some(7), Some(11), false, mutation_at - Duration::from_micros(1)),
+        ] {
+            let mut snapshot = first.clone();
+            snapshot.scanner_epoch = epoch;
+            snapshot.scanner_cycle = cycle;
+            snapshot.usage_snapshot_partial = partial;
+            snapshot.usage_snapshot_complete = !partial;
+            snapshot.last_update = Some(update);
+            replace_bucket_usage_memory_from_info(&snapshot).await;
+            apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+            assert_eq!(snapshot.objects_total_count, 1, "epoch={epoch:?}, cycle={cycle:?}, partial={partial}");
+            assert_eq!(snapshot.objects_total_size, 42);
+        }
+
+        let mut successor = first;
+        successor.scanner_epoch = Some(8);
+        successor.scanner_cycle = Some(11);
+        replace_bucket_usage_memory_from_info(&successor).await;
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut successor, true).await;
+        assert_eq!(successor.objects_total_count, 0);
+        assert_eq!(successor.objects_total_size, 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn a_new_authoritative_mutation_restarts_reconciliation() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "dirty-mutation-fence";
+        replace_bucket_usage_memory_from_info(&data_usage_info_for_test(bucket, 0, 0, SystemTime::now())).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mut snapshot = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        snapshot.scanner_epoch = Some(7);
+        snapshot.scanner_cycle = Some(10);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        snapshot.last_update = Some(SystemTime::now());
+        snapshot.scanner_cycle = Some(11);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+        let mut response = snapshot.clone();
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut response, true).await;
+        assert_eq!(response.objects_total_count, 2);
+        assert_eq!(response.objects_total_size, 84);
+
+        snapshot.scanner_cycle = Some(12);
+        replace_bucket_usage_memory_from_info(&snapshot).await;
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut snapshot, true).await;
+        assert_eq!(snapshot.objects_total_count, 0);
+        assert_eq!(snapshot.objects_total_size, 0);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_scans_repair_historical_usage_during_continuing_writes() {
+        let bucket = "historical-busy";
+        for position in [None, Some((7, 10))] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = position.map(|value| value.0);
+            baseline.scanner_cycle = position.map(|value| value.1);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            for objects in 1..=6 {
+                record_bucket_object_write_memory(bucket, None, 42).await;
+                let mut scanned = data_usage_info_for_test(bucket, objects, objects * 42, SystemTime::now());
+                scanned.scanner_epoch = Some(7);
+                scanned.scanner_cycle = Some(10 + objects);
+                scanned.usage_snapshot_converged = Some(true);
+                replace_bucket_usage_memory_from_info(&scanned).await;
+                assert_eq!(get_bucket_usage_memory(bucket).await, Some(objects * 42));
+                apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+                assert_eq!(scanned.objects_total_count, objects);
+                assert_eq!(scanned.objects_total_size, objects * 42);
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_usage_refresh_requires_a_new_valid_scanner_generation() {
+        let bucket = "converged-generation";
+        for (converged, epoch, cycle, partial) in [
+            (None, Some(7), Some(11), false),
+            (Some(false), Some(7), Some(11), false),
+            (Some(true), None, Some(11), false),
+            (Some(true), Some(7), None, false),
+            (Some(true), Some(7), Some(10), false),
+            (Some(true), Some(8), Some(10), false),
+            (Some(true), Some(6), Some(11), false),
+            (Some(true), Some(7), Some(9), false),
+            (Some(true), Some(7), Some(11), true),
+        ] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(10);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+            scanned.usage_snapshot_converged = converged;
+            scanned.scanner_epoch = epoch;
+            scanned.scanner_cycle = cycle;
+            scanned.usage_snapshot_partial = partial;
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(
+                get_bucket_usage_memory(bucket).await,
+                Some(4_587_688_554),
+                "converged={converged:?}, epoch={epoch:?}, cycle={cycle:?}, partial={partial}"
+            );
+            scanned.usage_snapshot_converged = Some(true);
+            scanned.scanner_epoch = Some(8);
+            scanned.scanner_cycle = Some(11);
+            scanned.usage_snapshot_partial = false;
+            replace_bucket_usage_memory_from_info(&scanned).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn converged_usage_refresh_preserves_mutations_after_the_scan() {
+        clear_usage_memory_cache_for_test().await;
+        let bucket = "converged-write-race";
+        let mut baseline = data_usage_info_for_test(bucket, 0, 0, SystemTime::now());
+        baseline.scanner_epoch = Some(7);
+        baseline.scanner_cycle = Some(10);
+        replace_bucket_usage_memory_from_info(&baseline).await;
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let mut scanned = data_usage_info_for_test(bucket, 1, 42, SystemTime::now());
+        scanned.scanner_epoch = Some(7);
+        scanned.scanner_cycle = Some(11);
+        scanned.usage_snapshot_converged = Some(true);
+        record_bucket_object_write_memory(bucket, None, 42).await;
+        let after_scan = scanned.last_update.expect("scanner timestamp") + Duration::from_secs(1);
+        memory_cache()
+            .write()
+            .await
+            .get_mut(bucket)
+            .expect("dirty cache")
+            .usage_updated_at = after_scan;
+        replace_bucket_usage_memory_from_info(&scanned).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+        apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+        assert_eq!(scanned.objects_total_count, 2);
+        assert_eq!(scanned.objects_total_size, 84);
+        let mut successor = data_usage_info_for_test(bucket, 2, 84, after_scan);
+        successor.scanner_epoch = Some(7);
+        successor.scanner_cycle = Some(12);
+        successor.usage_snapshot_converged = Some(true);
+        replace_bucket_usage_memory_from_info(&successor).await;
+        assert_eq!(get_bucket_usage_memory(bucket).await, Some(84));
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn fallback_observations_cannot_bypass_the_cached_scanner_generation() {
+        let bucket = "historical-generation-floor";
+        for (converged, epoch, first_cycle) in [(None, 7, 10), (Some(true), 7, 10), (None, 6, 21), (Some(true), 6, 21)] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 436, 4_587_688_512, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(20);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            // Wall clocks may step back; a timestamp alone cannot establish
+            // that an asynchronously loaded generation includes this write.
+            let update = memory_cache().read().await[bucket].usage_updated_at + Duration::from_secs(1);
+            for cycle in first_cycle..first_cycle + 3 {
+                let mut scanned = data_usage_info_for_test(bucket, 0, 0, update);
+                scanned.scanner_epoch = Some(epoch);
+                scanned.scanner_cycle = Some(cycle);
+                scanned.usage_snapshot_converged = converged;
+                replace_bucket_usage_memory_from_info(&scanned).await;
+                assert_eq!(
+                    get_bucket_usage_memory(bucket).await,
+                    Some(4_587_688_554),
+                    "older epoch/cycle cannot release quota: converged={converged:?}, epoch={epoch}, cycle={cycle}"
+                );
+                apply_bucket_usage_memory_overlay_if_authoritative(&mut scanned, true).await;
+                assert_eq!(scanned.objects_total_count, 437);
+                assert_eq!(scanned.objects_total_size, 4_587_688_554);
+            }
+            let mut successor = data_usage_info_for_test(bucket, 1, 42, update);
+            successor.scanner_epoch = Some(8);
+            successor.scanner_cycle = Some(21);
+            successor.usage_snapshot_converged = Some(true);
+            replace_bucket_usage_memory_from_info(&successor).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(42));
+        }
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn matching_counts_cannot_discard_the_cached_scanner_generation() {
+        let bucket = "matching-generation-floor";
+        for position in [None, Some((7, 10))] {
+            clear_usage_memory_cache_for_test().await;
+            let mut baseline = data_usage_info_for_test(bucket, 2, 84, SystemTime::now());
+            baseline.scanner_epoch = Some(7);
+            baseline.scanner_cycle = Some(20);
+            replace_bucket_usage_memory_from_info(&baseline).await;
+            let mut matching = baseline.clone();
+            matching.scanner_epoch = position.map(|value| value.0);
+            matching.scanner_cycle = position.map(|value| value.1);
+            replace_bucket_usage_memory_from_info(&matching).await;
+            record_bucket_object_write_memory(bucket, None, 42).await;
+            let update = memory_cache().read().await[bucket].usage_updated_at + Duration::from_secs(1);
+            let mut older = data_usage_info_for_test(bucket, 0, 0, update);
+            older.scanner_epoch = Some(7);
+            older.scanner_cycle = Some(12);
+            older.usage_snapshot_converged = Some(true);
+            replace_bucket_usage_memory_from_info(&older).await;
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some(126));
+            apply_bucket_usage_memory_overlay_if_authoritative(&mut older, true).await;
+            assert_eq!(older.objects_total_count, 3);
+            assert_eq!(older.objects_total_size, 126);
+        }
+    }
+
    #[tokio::test]
    #[serial]
    async fn authoritative_refresh_can_reduce_memory_usage() {
@@ -73,7 +73,7 @@ impl ProcessResultBudget {
        }
        inner
            .available
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |available| available.checked_sub(bytes))
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |available| available.checked_sub(bytes))
            .map(|_| {
                Some(ResultBudgetReservation {
                    inner: Arc::clone(inner),
@@ -408,6 +408,7 @@ async fn blackbox_heal_requests_preserve_repair_scope() {
                        &ObjectOptions {
                            no_lock: true,
                            versioned: true,
+                            expected_bucket_incarnation_id: Some(put_incarnation),
                            ..Default::default()
                        },
                    )
@@ -908,7 +908,7 @@ fn explicit_url_port(raw: &str) -> Result<Option<u16>> {
        return Err(invalid());
    }

-    let (scheme, remainder) = raw.split_once("://").ok_or_else(&invalid)?;
+    let (scheme, remainder) = raw.split_once("://").ok_or_else(invalid)?;
    if !scheme.eq_ignore_ascii_case("http") && !scheme.eq_ignore_ascii_case("https") {
        return Err(invalid());
    }
@@ -916,17 +916,17 @@ fn explicit_url_port(raw: &str) -> Result<Option<u16>> {
        .split('/')
        .next()
        .filter(|authority| !authority.is_empty())
-        .ok_or_else(&invalid)?;
+        .ok_or_else(invalid)?;
    if authority.contains('@') {
        return Err(invalid());
    }

    let port = if let Some(bracketed) = authority.strip_prefix('[') {
-        let (_, suffix) = bracketed.split_once(']').ok_or_else(&invalid)?;
+        let (_, suffix) = bracketed.split_once(']').ok_or_else(invalid)?;
        if suffix.is_empty() {
            return Ok(None);
        }
-        suffix.strip_prefix(':').ok_or_else(&invalid)?
+        suffix.strip_prefix(':').ok_or_else(invalid)?
    } else if let Some((_, port)) = authority.rsplit_once(':') {
        port
    } else {
@@ -1001,11 +1001,7 @@ fn parse_explicit_local_endpoint_host(raw: &str) -> Result<Host<String>> {

    let host = Host::parse(raw).map_err(|_| invalid())?;
    let host = match host {
-        Host::Domain(domain) => Host::Domain(
-            domain_without_optional_trailing_dot(&domain)
-                .ok_or_else(&invalid)?
-                .to_string(),
-        ),
+        Host::Domain(domain) => Host::Domain(domain_without_optional_trailing_dot(&domain).ok_or_else(invalid)?.to_string()),
        host => host,
    };
    if matches!(&host, Host::Domain(domain) if domain.contains('*'))
@@ -396,7 +396,7 @@ impl InstanceContext {
    pub(crate) fn begin_namespace_commit(self: &Arc<Self>) -> Arc<NamespaceCommitGuard> {
        let counted = self
            .namespace_commits
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |count| count.checked_add(1))
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |count| count.checked_add(1))
            .is_ok();
        if counted {
            self.advance_namespace_commit_generation();
@@ -412,7 +412,7 @@ impl InstanceContext {
    fn advance_namespace_commit_generation(&self) {
        let _ = self
            .namespace_commit_generation
-            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |generation| Some(generation.saturating_add(1)));
+            .try_update(Ordering::AcqRel, Ordering::Acquire, |generation| Some(generation.saturating_add(1)));
    }

    pub(crate) fn namespace_commit_generation(&self) -> u64 {
@@ -1350,6 +1350,8 @@ mod prepared_get_object_metadata_tests {
                ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
            ],
            async {
+                // Isolate late body recovery from the independent metadata slow-tail hedge.
+                let _hedge_timer = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_NON_INLINE_HEDGE_TIMER);
                let calls = disk_call_counters::observe(&object);
                let mut reader = set_disks
                    .get_object_reader(bucket, &object, None, HeaderMap::new(), &opts)
@@ -1414,6 +1416,8 @@ mod prepared_get_object_metadata_tests {
                ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
            ],
            async {
+                // Isolate late body recovery from the independent metadata slow-tail hedge.
+                let _hedge_timer = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_NON_INLINE_HEDGE_TIMER);
                let calls = disk_call_counters::observe(&object);
                let mut reader = set_disks
                    .get_object_reader(bucket, &object, None, HeaderMap::new(), &opts)
@@ -1427,6 +1431,11 @@ mod prepared_get_object_metadata_tests {
                    .expect("late parity should restore the exact GET body");
                assert_eq!(restored, payload);
                assert_eq!(calls.total(disk_call_counters::KIND_READ_VERSION), 7);
+                assert_eq!(
+                    calls.for_disk(disk_call_counters::KIND_READ_VERSION, order[3]),
+                    1,
+                    "the omitted parity disk must only be read by the late metadata refresh"
+                );
            },
        )
        .await;
@@ -1491,6 +1500,8 @@ mod prepared_get_object_metadata_tests {
                ("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", Some("true")),
            ],
            async {
+                // Isolate late body recovery from the independent metadata slow-tail hedge.
+                let _hedge_timer = rename_fanout_barrier::arm(&object, 0, rename_fanout_barrier::PHASE_NON_INLINE_HEDGE_TIMER);
                let calls = disk_call_counters::observe(&object);
                let mut reader = set_disks
                    .get_object_reader(bucket, &object, None, HeaderMap::new(), &opts)
@@ -7171,10 +7182,10 @@ fn parts_after_marker(part_numbers: &[usize], part_number_marker: usize) -> Opti
        return Some(part_numbers);
    }

-    part_numbers
-        .iter()
-        .position(|&part_number| part_number != 0 && part_number == part_number_marker)
-        .map(|index| &part_numbers[index + 1..])
+    // reduce_quorum_part_numbers returns sorted numbers; the marker need not exist.
+    part_numbers.last().filter(|&&last| part_number_marker <= last)?;
+    let index = part_numbers.partition_point(|&part_number| part_number <= part_number_marker);
+    Some(&part_numbers[index..])
 }

 pub fn canonicalize_etag(etag: &str) -> String {
@@ -13972,6 +13983,24 @@ mod tests {
        assert!(parts_after_marker(&part_numbers, 4).is_none());
    }

+    #[test]
+    fn parts_after_marker_uses_exclusive_numeric_boundary_for_sparse_parts() {
+        let part_numbers = [1, 3, 10];
+        for (marker, expected) in [
+            (0, Some(&part_numbers[..])),
+            (1, Some(&part_numbers[1..])),
+            (2, Some(&part_numbers[1..])),
+            (3, Some(&part_numbers[2..])),
+            (9, Some(&part_numbers[2..])),
+            (10, Some(&part_numbers[3..])),
+            (11, None),
+            (usize::MAX, None),
+        ] {
+            assert_eq!(parts_after_marker(&part_numbers, marker), expected, "marker {marker}");
+        }
+        assert_eq!(parts_after_marker(&[], 1), None);
+    }
+
    #[test]
    fn delete_file_info_version_id_maps_explicit_null_version_to_stored_null() {
        assert_eq!(delete_file_info_version_id(Some(Uuid::nil())), None);
@@ -2125,7 +2125,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {

        if object_parts.len() > ret.parts.len() {
            ret.is_truncated = true;
-            ret.next_part_number_marker = ret.parts.last().map(|v| v.part_num).unwrap_or_default();
+            ret.next_part_number_marker = ret.parts.last().map(|v| v.part_num);
        }

        ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
@@ -4505,6 +4505,54 @@ mod tests {
        }
    }

+    #[tokio::test]
+    async fn list_object_parts_pagination_preserves_sparse_markers_and_terminates() {
+        let (_temp_dirs, disks, set_disks) = hermetic_set_disks(4).await;
+        let bucket = "multipart-pagination";
+        let object = "object";
+        make_bucket_on_all(&disks, bucket).await;
+        let opts = ObjectOptions::default();
+        let upload = set_disks
+            .new_multipart_upload(bucket, object, &opts)
+            .await
+            .expect("multipart upload should be created");
+
+        let empty = set_disks
+            .list_object_parts(bucket, object, &upload.upload_id, None, 2, &opts)
+            .await
+            .expect("empty upload should be listable");
+        assert!(empty.parts.is_empty());
+        assert!(!empty.is_truncated);
+        assert_eq!(empty.next_part_number_marker, None);
+
+        for part_number in [10, 1, 3] {
+            put_test_part(&set_disks, bucket, object, &upload.upload_id, part_number, b"part", 4).await;
+        }
+
+        for (marker, max_parts, expected_parts, next_marker) in [
+            (None, 0, vec![], None),
+            (None, 1, vec![1], Some(1)),
+            (None, 2, vec![1, 3], Some(3)),
+            (None, 3, vec![1, 3, 10], None),
+            (None, MAX_PARTS_COUNT + 1, vec![1, 3, 10], None),
+            (Some(1), 2, vec![3, 10], None),
+            (Some(2), 1, vec![3], Some(3)),
+            (Some(3), 2, vec![10], None),
+            (Some(10), 2, vec![], None),
+            (Some(11), 2, vec![], None),
+        ] {
+            let page = set_disks
+                .list_object_parts(bucket, object, &upload.upload_id, marker, max_parts, &opts)
+                .await
+                .expect("uploaded parts should be listable");
+            assert_eq!(page.part_number_marker, marker.unwrap_or_default());
+            assert_eq!(page.max_parts, max_parts.min(MAX_PARTS_COUNT));
+            assert_eq!(page.parts.iter().map(|part| part.part_num).collect::<Vec<_>>(), expected_parts);
+            assert_eq!(page.is_truncated, next_marker.is_some());
+            assert_eq!(page.next_part_number_marker, next_marker);
+        }
+    }
+
    async fn object_transaction_epochs(disks: &[DiskStore], bucket: &str, object: &str) -> Vec<Option<Uuid>> {
        let mut epochs = Vec::with_capacity(disks.len());
        let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
@@ -4362,7 +4362,7 @@ mod tests {
            // successful attempts. Only injected faults spend this global budget.
            // A real failure may consume an attempt, so preserve the final chance.
            faults
-                .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |faults| {
+                .try_update(Ordering::SeqCst, Ordering::SeqCst, |faults| {
                    (faults < crate::core::pools::DECOMMISSION_VERSION_COPY_ATTEMPTS.saturating_sub(1))
                        .then_some(faults.saturating_add(1))
                })
@@ -918,10 +918,19 @@ async fn partial_write_sigkill_replay_scenario(protected: bool) {
            "replayed responsibility must be released only after verified repair"
        );
    } else {
+        // A repair can finish before the next legacy check parks its responsibility.
        assert!(
            wait_until(|| async {
                let snapshot = manager.operations_snapshot().await;
-                snapshot.queue_length == 0 && snapshot.active_tasks == 0
+                snapshot.queue_length == 0
+                    && snapshot.active_tasks == 0
+                    && mrf_queue::list_legacy_responsibilities(None, 32).await.is_ok_and(|state| {
+                        state.responsibilities.iter().any(|entry| {
+                            entry.bucket == "partial-crash"
+                                && entry.object == "crash.bin"
+                                && entry.status == "held_unverified_legacy"
+                        })
+                    })
            })
            .await,
            "legacy replay attempts must finish"
@@ -2415,6 +2415,13 @@ pub fn record_allocator_memory_observation(backend: &'static str, observation: A
    if let Some(page_committed_bytes) = observation.page_committed_bytes {
        gauge!("rustfs_memory_allocator_page_committed_bytes", "backend" => backend).set(page_committed_bytes as f64);
    }
+    gauge!("rustfs_memory_allocator_malloc_requested_bytes_available", "backend" => backend).set(
+        if observation.malloc_requested_bytes.is_some() {
+            1.0
+        } else {
+            0.0
+        },
+    );
    if let Some(malloc_requested_bytes) = observation.malloc_requested_bytes {
        gauge!("rustfs_memory_allocator_malloc_requested_bytes", "backend" => backend).set(malloc_requested_bytes as f64);
    }
@@ -2795,6 +2802,46 @@ mod tests {
        samples
    }

+    #[test]
+    fn allocator_requested_availability_distinguishes_missing_from_zero() {
+        let _guard = METRICS_FLAG_LOCK.lock().unwrap_or_else(|e| e.into_inner());
+        for (previous, requested, expected_value, expected_availability) in [
+            (None, None, None, 0.0),
+            (None, Some(0), Some(0.0), 1.0),
+            (Some(128), None, Some(128.0), 0.0),
+            (Some(128), Some(0), Some(0.0), 1.0),
+        ] {
+            let recorder = DebuggingRecorder::new();
+            let snapshotter = recorder.snapshotter();
+            metrics::with_local_recorder(&recorder, || {
+                set_metrics_enabled(true);
+                if let Some(previous) = previous {
+                    record_allocator_memory_observation(
+                        "mimalloc",
+                        AllocatorMemoryObservation {
+                            malloc_requested_bytes: Some(previous),
+                            ..Default::default()
+                        },
+                    );
+                }
+                record_allocator_memory_observation(
+                    "mimalloc",
+                    AllocatorMemoryObservation {
+                        malloc_requested_bytes: requested,
+                        ..Default::default()
+                    },
+                );
+                set_metrics_enabled(false);
+            });
+            let rows = snapshotter.snapshot().into_vec();
+            assert_eq!(
+                gauge_value(&rows, "rustfs_memory_allocator_malloc_requested_bytes_available"),
+                Some(expected_availability),
+            );
+            assert_eq!(gauge_value(&rows, "rustfs_memory_allocator_malloc_requested_bytes"), expected_value);
+        }
+    }
+
    /// Replaces four smoke tests that called the zero-copy and bytes-pool recorders
    /// and asserted nothing (rustfs/backlog#1836). The same calls now run against a
    /// local recorder: every metric name these helpers own must be emitted, the
@@ -43,7 +43,6 @@ hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-s3select-api/hotpath-cp
 [dependencies]
 hotpath.workspace = true
 rustfs-s3select-api = { workspace = true }
-async-recursion = { workspace = true }
 async-trait.workspace = true
 bytes.workspace = true
 datafusion = { workspace = true, default-features = false, features = ["parquet", "recursive_protection", "sql"] }
@@ -14,7 +14,6 @@

 use std::{convert::Infallible, ops::ControlFlow};

-use async_recursion::async_recursion;
 use async_trait::async_trait;
 use datafusion::sql::{
    planner::{IdentNormalizer, SqlToRel},
@@ -58,7 +57,6 @@ impl<'a, S: ContextProviderExtension + Send + Sync + 'a> SqlPlanner<'a, S> {
    }

    /// Generate a logical plan from an  Extent SQL statement
-    #[async_recursion]
    pub(crate) async fn statement_to_plan(&self, statement: ExtStatement, session: &SessionCtx) -> QueryResult<Plan> {
        match statement {
            ExtStatement::SqlStatement(stmt) => self.df_sql_to_plan(*stmt, session).await,
@@ -87,7 +87,8 @@ pub use runtime_config::{apply_scanner_runtime_config, scanner_runtime_config_st
 pub use rustfs_scanner_metrics::last_minute;
 pub use scanner::{
    SCANNER_RECOVERY_INTENT_ACTION_USAGE_FULL_REBUILD, ScannerCycleRecoveryMarker, ScannerCycleRecoveryStatus,
-    ScannerCycleScheduleStatus, ScannerPauseBacklogAlertReason, ScannerPauseBacklogPhase, ScannerPauseBacklogStatus,
+    ScannerCycleScheduleStatus, ScannerPauseBacklogAlertReason, ScannerPauseBacklogPhase, ScannerPauseBacklogReplicaId,
+    ScannerPauseBacklogReplicaStatus, ScannerPauseBacklogReplicaStatusState, ScannerPauseBacklogStatus,
    ScannerPauseBacklogThresholds, ScannerRecoveryIntentAcceptResult, ScannerRecoveryIntentConflict, ScannerRecoveryIntentRecord,
    ScannerRecoveryIntentRequest, ScannerUsageStateResetResult, accept_scanner_usage_recovery_intent,
    get_scanner_usage_recovery_intent, init_data_scanner, init_scanner_with_recovery, register_scanner_pause_backlog_retirement,
@@ -840,6 +840,7 @@ async fn scan_and_persist_local_bucket(
                }
                match persist_scanner_checkpoint(
                    set.clone(),
+                    store.clone(),
                    ScannerCheckpointPersistContext {
                        ctx: &scan_ctx,
                        expected_publication_epoch,
@@ -3800,6 +3800,7 @@ mod usage_store;

 use activity::*;
 use backlog::*;
+pub use backlog::{ScannerPauseBacklogReplicaId, ScannerPauseBacklogReplicaStatus, ScannerPauseBacklogReplicaStatusState};
 use cycle_state::*;
 use leadership::*;
 pub(crate) use usage_store::RootPublicationProof;
@@ -144,12 +144,39 @@ pub struct ScannerPauseBacklogStatus {
    pub replica_count: usize,
    pub healthy_replicas: usize,
    pub stale_or_unavailable_replicas: usize,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub replica_diagnostics: Option<Vec<ScannerPauseBacklogReplicaStatus>>,
    pub alerting: bool,
    pub alert_reasons: Vec<ScannerPauseBacklogAlertReason>,
    pub thresholds: ScannerPauseBacklogThresholds,
    pub error: Option<String>,
 }

+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
+#[serde(rename_all = "snake_case")]
+pub enum ScannerPauseBacklogReplicaStatusState {
+    Missing,
+    Valid,
+    Invalid,
+    FutureSchema,
+    Unavailable,
+}
+
+#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
+pub struct ScannerPauseBacklogReplicaStatus {
+    pub pool_index: usize,
+    pub set_index: usize,
+    pub state: ScannerPauseBacklogReplicaStatusState,
+    pub revision: Option<String>,
+    pub stable_generation: Option<u64>,
+    pub stable_writer_epoch: Option<u64>,
+    pub committed_generation: Option<u64>,
+    pub committed_writer_epoch: Option<u64>,
+    pub committed_replica_count: Option<usize>,
+    pub committed_replicas: Option<Vec<ScannerPauseBacklogReplicaId>>,
+    pub error: Option<String>,
+}
+
 #[derive(Clone, Debug, Deserialize, PartialEq, Eq, Serialize)]
 #[serde(deny_unknown_fields)]
 struct ScannerPauseBacklogLedger {
@@ -656,9 +683,9 @@ pub(super) enum ScannerPauseBacklogAttemptDecision {
 }

 #[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize)]
-struct ScannerPauseBacklogReplicaId {
-    pool_index: usize,
-    set_index: usize,
+pub struct ScannerPauseBacklogReplicaId {
+    pub pool_index: usize,
+    pub set_index: usize,
 }

 #[derive(Clone, Debug, Deserialize, PartialEq, Eq, Serialize)]
@@ -780,6 +807,7 @@ impl LoadedScannerPauseBacklog {
            replica_degraded,
            persistence_unavailable,
            error,
+            None,
        )
    }
 }
@@ -796,6 +824,7 @@ fn status_from_ledger(
    replica_degraded: bool,
    persistence_unavailable: bool,
    error: Option<String>,
+    replica_diagnostics: Option<Vec<ScannerPauseBacklogReplicaStatus>>,
 ) -> ScannerPauseBacklogStatus {
    let alert_reasons = ledger.alert_reasons(now, replica_degraded, persistence_unavailable);
    let status = ScannerPauseBacklogStatus {
@@ -833,6 +862,7 @@ fn status_from_ledger(
        replica_count,
        healthy_replicas,
        stale_or_unavailable_replicas,
+        replica_diagnostics,
        alerting: !alert_reasons.is_empty(),
        alert_reasons,
        thresholds: ScannerPauseBacklogThresholds::default(),
@@ -947,6 +977,59 @@ async fn read_scanner_pause_backlog_replica(store: Arc<SetDisks>) -> ScannerPaus
    ScannerPauseBacklogReplica { id, revision, state }
 }

+fn scanner_pause_backlog_replica_diagnostics(replicas: Vec<ScannerPauseBacklogReplica>) -> Vec<ScannerPauseBacklogReplicaStatus> {
+    replicas
+        .into_iter()
+        .map(|replica| {
+            let mut status = ScannerPauseBacklogReplicaStatus {
+                pool_index: replica.id.pool_index,
+                set_index: replica.id.set_index,
+                state: ScannerPauseBacklogReplicaStatusState::Missing,
+                revision: match replica.revision {
+                    Some(DataUsageCacheRevision::Etag(etag)) => Some(etag),
+                    Some(DataUsageCacheRevision::Missing) | None => None,
+                },
+                stable_generation: None,
+                stable_writer_epoch: None,
+                committed_generation: None,
+                committed_writer_epoch: None,
+                committed_replica_count: None,
+                committed_replicas: None,
+                error: None,
+            };
+            match replica.state {
+                ScannerPauseBacklogReplicaState::Missing => {}
+                ScannerPauseBacklogReplicaState::Valid(record) => {
+                    status.state = ScannerPauseBacklogReplicaStatusState::Valid;
+                    if let Some(stable) = record.stable {
+                        status.stable_generation = Some(stable.generation);
+                        status.stable_writer_epoch = Some(stable.writer_epoch);
+                    }
+                    if let Some(committed) = record.committed {
+                        status.committed_generation = Some(committed.ledger.generation);
+                        status.committed_writer_epoch = Some(committed.ledger.writer_epoch);
+                        status.committed_replica_count = Some(committed.replicas.len());
+                        status.committed_replicas = Some(committed.replicas);
+                    }
+                }
+                ScannerPauseBacklogReplicaState::Invalid(reason) => {
+                    status.state = ScannerPauseBacklogReplicaStatusState::Invalid;
+                    status.error = Some(reason);
+                }
+                ScannerPauseBacklogReplicaState::FutureSchema(version) => {
+                    status.state = ScannerPauseBacklogReplicaStatusState::FutureSchema;
+                    status.error = Some(format!("unsupported future schema {version}"));
+                }
+                ScannerPauseBacklogReplicaState::Unavailable(reason) => {
+                    status.state = ScannerPauseBacklogReplicaStatusState::Unavailable;
+                    status.error = Some(reason);
+                }
+            }
+            status
+        })
+        .collect()
+}
+
 fn decode_scanner_pause_backlog_ledger(data: &[u8]) -> ScannerPauseBacklogReplicaState {
    let value = match serde_json::from_slice::<serde_json::Value>(data) {
        Ok(value) => value,
@@ -1016,6 +1099,33 @@ enum ScannerPauseBacklogSelectionError {
    InvalidRecord(String),
 }

+#[derive(Clone)]
+struct ScannerPauseBacklogSelectionFailure {
+    error: ScannerPauseBacklogSelectionError,
+    replicas: Vec<ScannerPauseBacklogReplica>,
+}
+
+impl fmt::Debug for ScannerPauseBacklogSelectionFailure {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.debug_struct("ScannerPauseBacklogSelectionFailure")
+            .field("error", &self.error)
+            .finish()
+    }
+}
+
+impl fmt::Display for ScannerPauseBacklogSelectionFailure {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        fmt::Display::fmt(&self.error, f)
+    }
+}
+
+fn scanner_pause_backlog_selection_failure(
+    replicas: Vec<ScannerPauseBacklogReplica>,
+    error: ScannerPauseBacklogSelectionError,
+) -> ScannerPauseBacklogSelectionFailure {
+    ScannerPauseBacklogSelectionFailure { error, replicas }
+}
+
 impl fmt::Display for ScannerPauseBacklogSelectionError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
@@ -1034,11 +1144,12 @@ impl From<ScannerPauseBacklogCommitSelectionError> for ScannerPauseBacklogSelect
 }

 fn scanner_pause_backlog_retirement_selection_error(
-    err: ScannerPauseBacklogSelectionError,
+    error: ScannerPauseBacklogSelectionError,
 ) -> ScannerPauseBacklogRetirementError {
-    match err {
+    let reason = error.to_string();
+    match error {
        ScannerPauseBacklogSelectionError::RetryableHandoffConflict => {
-            ScannerPauseBacklogRetirementError::HandoffConflict { reason: err.to_string() }
+            ScannerPauseBacklogRetirementError::HandoffConflict { reason }
        }
        ScannerPauseBacklogSelectionError::AuthorityConflict(reason) => {
            ScannerPauseBacklogRetirementError::AuthorityConflict { reason }
@@ -1123,24 +1234,33 @@ fn select_scanner_pause_backlog_commit(

 fn select_scanner_pause_backlog_replicas(
    replicas: Vec<ScannerPauseBacklogReplica>,
-) -> Result<LoadedScannerPauseBacklog, ScannerPauseBacklogSelectionError> {
+) -> Result<LoadedScannerPauseBacklog, ScannerPauseBacklogSelectionFailure> {
    if replicas.is_empty() {
-        return Err(ScannerPauseBacklogSelectionError::AuthorityConflict(
-            "scanner pause backlog has no storage replicas".to_string(),
+        return Err(scanner_pause_backlog_selection_failure(
+            replicas,
+            ScannerPauseBacklogSelectionError::AuthorityConflict("scanner pause backlog has no storage replicas".to_string()),
        ));
    }
-    for replica in &replicas {
-        if let ScannerPauseBacklogReplicaState::FutureSchema(version) = &replica.state {
-            return Err(ScannerPauseBacklogSelectionError::InvalidRecord(format!(
+    if let Some((replica_id, version)) = replicas.iter().find_map(|replica| match &replica.state {
+        ScannerPauseBacklogReplicaState::FutureSchema(version) => Some((replica.id, *version)),
+        _ => None,
+    }) {
+        return Err(scanner_pause_backlog_selection_failure(
+            replicas,
+            ScannerPauseBacklogSelectionError::InvalidRecord(format!(
                "scanner pause backlog pool {} set {} uses future schema {version}",
-                replica.id.pool_index, replica.id.set_index
-            )));
-        }
+                replica_id.pool_index, replica_id.set_index
+            )),
+        ));
    }

    let replica_ids = scanner_pause_backlog_replica_ids(&replicas);
-    let authoritative_commit =
-        select_scanner_pause_backlog_commit(&replicas, &replica_ids).map_err(ScannerPauseBacklogSelectionError::from)?;
+    let authoritative_commit = match select_scanner_pause_backlog_commit(&replicas, &replica_ids) {
+        Ok(commit) => commit,
+        Err(error) => {
+            return Err(scanner_pause_backlog_selection_failure(replicas, error.into()));
+        }
+    };
    let stable_consensus = if replicas.iter().all(|replica| {
        matches!(
            &replica.state,
@@ -1175,16 +1295,21 @@ fn select_scanner_pause_backlog_replicas(
                    "scanner pause backlog pool {} set {} is unavailable: {reason}",
                    replica.id.pool_index, replica.id.set_index
                );
-                return Err(match &replica.state {
+                let error = match &replica.state {
                    ScannerPauseBacklogReplicaState::Invalid(_) => ScannerPauseBacklogSelectionError::InvalidRecord(reason),
                    _ => ScannerPauseBacklogSelectionError::AuthorityConflict(reason),
-                });
+                };
+                return Err(scanner_pause_backlog_selection_failure(replicas, error));
            }
            match &stable_consensus {
                Ok(stable) => stable.clone(),
                Err(()) => {
-                    return Err(ScannerPauseBacklogSelectionError::AuthorityConflict(
-                        "scanner pause backlog has neither a surviving membership commit nor a stable rollback point".to_string(),
+                    return Err(scanner_pause_backlog_selection_failure(
+                        replicas,
+                        ScannerPauseBacklogSelectionError::AuthorityConflict(
+                            "scanner pause backlog has neither a surviving membership commit nor a stable rollback point"
+                                .to_string(),
+                        ),
                    ));
                }
            }
@@ -1340,13 +1465,15 @@ fn verify_scanner_pause_backlog_retirement(
        .filter(|replica| replica.id.pool_index != source_pool_index)
        .cloned()
        .collect::<Vec<_>>();
-    let selected = select_scanner_pause_backlog_replicas(surviving).map_err(scanner_pause_backlog_retirement_selection_error)?;
+    let selected = select_scanner_pause_backlog_replicas(surviving)
+        .map_err(|failure| scanner_pause_backlog_retirement_selection_error(failure.error))?;
    if !selected.durable || !selected.stable_matches_ledger {
        return Err(scanner_pause_backlog_retirement_authority_conflict(
            "scanner pause backlog retirement requires stable authority on the complete surviving membership",
        ));
    }
-    let before = select_scanner_pause_backlog_replicas(decoded).map_err(scanner_pause_backlog_retirement_selection_error)?;
+    let before = select_scanner_pause_backlog_replicas(decoded)
+        .map_err(|failure| scanner_pause_backlog_retirement_selection_error(failure.error))?;
    if !before.durable || before.ledger != selected.ledger {
        return Err(scanner_pause_backlog_retirement_authority_conflict(
            "scanner pause backlog retirement would change the native ledger authority",
@@ -1439,7 +1566,7 @@ fn plan_scanner_pause_backlog_retirement(
        // Use the same native selector that validates the full commit, rather
        // than inferring authority from source epoch or physical object time.
        full_selection
-            .map_err(scanner_pause_backlog_retirement_selection_error)?
+            .map_err(|failure| scanner_pause_backlog_retirement_selection_error(failure.error))?
            .ledger
    } else {
        // An interrupted cohort switch can invalidate the old full commit
@@ -1448,7 +1575,7 @@ fn plan_scanner_pause_backlog_retirement(
        // must have been rejected before this recovery path.
        let current = selected
            .as_ref()
-            .map_err(|err| scanner_pause_backlog_retirement_selection_error(err.clone()))?;
+            .map_err(|failure| scanner_pause_backlog_retirement_selection_error(failure.error.clone()))?;
        if !current.durable {
            return Err(scanner_pause_backlog_retirement_authority_conflict(
                "scanner pause backlog retirement has no proven native authority",
@@ -1548,12 +1675,19 @@ async fn load_scanner_pause_backlog<S>(storeapi: Arc<S>) -> Result<LoadedScanner
 where
    S: ScannerStorage,
 {
-    let writable = storeapi.scanner_pause_backlog_writable_set_disks().await;
-    if writable.is_empty() {
+    let replicas = read_scanner_pause_backlog_replicas(storeapi).await;
+    if replicas.is_empty() {
        return Err("scanner pause backlog has no surviving storage replicas".to_string());
    }
-    let replicas = join_all(writable.into_iter().map(read_scanner_pause_backlog_replica)).await;
-    select_scanner_pause_backlog_replicas(replicas).map_err(|err| err.to_string())
+    select_scanner_pause_backlog_replicas(replicas).map_err(|failure| failure.to_string())
+}
+
+async fn read_scanner_pause_backlog_replicas<S>(storeapi: Arc<S>) -> Vec<ScannerPauseBacklogReplica>
+where
+    S: ScannerStorage,
+{
+    let writable = storeapi.scanner_pause_backlog_writable_set_disks().await;
+    join_all(writable.into_iter().map(read_scanner_pause_backlog_replica)).await
 }

 async fn write_scanner_pause_backlog_record<S>(
@@ -1929,21 +2063,35 @@ fn runtime_error() -> Option<String> {

 pub async fn scanner_pause_backlog_status(storeapi: Arc<ECStore>) -> ScannerPauseBacklogStatus {
    let now = unix_now();
-    match load_scanner_pause_backlog(storeapi).await {
-        Ok(loaded) => loaded.status(now, runtime_error()),
-        Err(error) => {
-            let error = runtime_error().map_or(error.clone(), |runtime| format!("{error}; {runtime}"));
+    let replicas = read_scanner_pause_backlog_replicas(storeapi).await;
+    let replica_count = replicas.len();
+    match select_scanner_pause_backlog_replicas(replicas) {
+        Ok(loaded) => {
+            let mut status = loaded.status(now, runtime_error());
+            if status.error.is_some()
+                || !status.durable
+                || status.stale_or_unavailable_replicas > 0
+                || status.persistence_state != "healthy"
+            {
+                status.replica_diagnostics = Some(scanner_pause_backlog_replica_diagnostics(loaded.replicas));
+            }
+            status
+        }
+        Err(failure) => {
+            let error = runtime_error().map_or(failure.to_string(), |runtime| format!("{failure}; {runtime}"));
+            let replica_diagnostics = scanner_pause_backlog_replica_diagnostics(failure.replicas);
            status_from_ledger(
                &ScannerPauseBacklogLedger::default(),
                now,
                "unavailable".to_string(),
                false,
+                replica_count,
                0,
-                0,
-                0,
+                replica_count,
                false,
                true,
                Some(error),
+                Some(replica_diagnostics),
            )
        }
    }
@@ -1959,6 +2107,62 @@ mod tests {

    const NATIVE_RETIREMENT_DRIVES_PER_SET: usize = 2;

+    #[test]
+    fn replica_diagnostics_move_snapshot_buffers_and_preserve_json() {
+        let mut stable = durable_ledger(100);
+        stable.generation = 7;
+        stable.writer_epoch = 3;
+        let mut committed = stable.clone();
+        committed.generation = 11;
+        committed.writer_epoch = 5;
+        let members = vec![replica_id(0, 1), replica_id(1, 0)];
+        let members_buffer = members.as_ptr();
+        let revision = "native-revision".to_string();
+        let revision_buffer = revision.as_ptr();
+        let reason = "temporary read error".to_string();
+        let reason_buffer = reason.as_ptr();
+        let record =
+            ScannerPauseBacklogReplicaRecord::new(Some(stable), Some(ScannerPauseBacklogCommitRecord::new(committed, members)));
+        record
+            .validate()
+            .expect("the diagnostic fixture must satisfy the persisted record contract");
+        let diagnostics = scanner_pause_backlog_replica_diagnostics(vec![
+            ScannerPauseBacklogReplica {
+                id: replica_id(0, 1),
+                revision: Some(DataUsageCacheRevision::Etag(revision)),
+                state: ScannerPauseBacklogReplicaState::Valid(Box::new(record)),
+            },
+            ScannerPauseBacklogReplica {
+                id: replica_id(1, 0),
+                revision: None,
+                state: ScannerPauseBacklogReplicaState::Unavailable(reason),
+            },
+        ]);
+        assert_eq!(diagnostics[0].revision.as_ref().expect("revision").as_ptr(), revision_buffer);
+        assert_eq!(diagnostics[0].committed_replicas.as_ref().expect("membership").as_ptr(), members_buffer);
+        assert_eq!(diagnostics[1].error.as_ref().expect("read error").as_ptr(), reason_buffer);
+        assert_eq!(
+            serde_json::to_value(diagnostics).expect("diagnostics must serialize"),
+            serde_json::json!([
+                {
+                    "pool_index": 0, "set_index": 1, "state": "valid", "revision": "native-revision",
+                    "stable_generation": 7, "stable_writer_epoch": 3,
+                    "committed_generation": 11, "committed_writer_epoch": 5,
+                    "committed_replica_count": 2,
+                    "committed_replicas": [{"pool_index": 0, "set_index": 1}, {"pool_index": 1, "set_index": 0}],
+                    "error": null
+                },
+                {
+                    "pool_index": 1, "set_index": 0, "state": "unavailable", "revision": null,
+                    "stable_generation": null, "stable_writer_epoch": null,
+                    "committed_generation": null, "committed_writer_epoch": null,
+                    "committed_replica_count": null, "committed_replicas": null,
+                    "error": "temporary read error"
+                }
+            ])
+        );
+    }
+
    fn run_native_retirement_test<C, F>(case: C)
    where
        C: FnOnce() -> F + Send + 'static,
@@ -1984,13 +2188,17 @@ mod tests {
    }

    async fn native_retirement_store() -> (tempfile::TempDir, Arc<ECStore>) {
+        native_pause_backlog_store(3, 2).await
+    }
+
+    async fn native_pause_backlog_store(pool_count: usize, sets_per_pool: usize) -> (tempfile::TempDir, Arc<ECStore>) {
        register_scanner_pause_backlog_retirement();
        let root = tempfile::tempdir().expect("native retirement fixture directory");
        let store = super::super::tests::setup_scanner_cycle_store_at_path_with_layout_and_disk_preinit(
            root.path(),
            false,
-            3,
-            2,
+            pool_count,
+            sets_per_pool,
            NATIVE_RETIREMENT_DRIVES_PER_SET,
            false,
        )
@@ -1998,6 +2206,344 @@ mod tests {
        (root, store)
    }

+    #[test]
+    #[serial_test::serial]
+    fn native_single_pool_partial_commit_converges_across_24_sets() {
+        run_native_retirement_test(async || {
+            set_runtime_error(None);
+            let (_root, store) = native_pause_backlog_store(1, 24).await;
+            let now = unix_now();
+            let mut controller = ScannerPauseBacklogController::claim(Arc::clone(&store), now)
+                .await
+                .expect("bootstrap the single-pool ledger");
+            controller.observe(observation(now + 1, true, 7)).await;
+            controller.observe(observation(now + 2, false, 7)).await;
+            let stable = controller.loaded.ledger.clone();
+            assert!(stable.pending_full_scan);
+            let ids = scanner_pause_backlog_replica_ids(&controller.loaded.replicas);
+            assert_eq!(ids.len(), 24);
+            let old_commit = controller
+                .loaded
+                .authoritative_commit
+                .clone()
+                .expect("initial full membership proof");
+            drop(controller);
+            let candidate = claim_scanner_pause_backlog_writer(&stable, now + 3).expect("interrupted writer generation");
+            let new_commit = ScannerPauseBacklogCommitRecord::new(candidate, ids.clone());
+            for (index, id) in ids.iter().enumerate() {
+                let commit = if index < 12 { &new_commit } else { &old_commit };
+                replace_native_pause_backlog_replica(
+                    &store,
+                    *id,
+                    ScannerPauseBacklogReplicaRecord::new(Some(stable.clone()), Some(commit.clone())),
+                )
+                .await;
+            }
+
+            let metric_status = load_scanner_pause_backlog(Arc::clone(&store))
+                .await
+                .expect("the common rollback must be selectable")
+                .status(now + 3, None);
+            assert_eq!(metric_status.persistence_state, "rolled_back_partial_commit");
+            assert!(
+                metric_status.replica_diagnostics.is_none(),
+                "metrics must not allocate discarded replica diagnostics"
+            );
+            let status = scanner_pause_backlog_status(Arc::clone(&store)).await;
+            assert_eq!(status.persistence_state, "rolled_back_partial_commit");
+            assert_eq!(status.healthy_replicas, 0);
+            assert_eq!(status.replica_count, 24);
+            assert_eq!(status.generation, stable.generation);
+            let diagnostics = status.replica_diagnostics.expect("partial commit must expose every replica");
+            assert_eq!(diagnostics.len(), 24);
+            assert!(
+                diagnostics
+                    .iter()
+                    .all(|replica| replica.committed_replicas.as_ref() == Some(&ids))
+            );
+            let expected = claim_scanner_pause_backlog_writer(&stable, now + 4).expect("recovery writer generation");
+            let recovered = ScannerPauseBacklogController::claim(Arc::clone(&store), now + 4)
+                .await
+                .expect("the common stable rollback must converge without resetting the ledger");
+            assert_current_native_writer_ledger(&store, &expected).await;
+            assert!(recovered.loaded.ledger.pending_full_scan);
+            assert_eq!(recovered.loaded.ledger.dirty_usage_buckets, 7);
+            assert_eq!(recovered.loaded.ledger.writer_epoch, stable.writer_epoch + 1);
+            drop(recovered);
+            shutdown_native_retirement_store(store).await;
+            set_runtime_error(None);
+        });
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn native_single_pool_conflicting_stable_proofs_preserve_all_24_sets() {
+        run_native_retirement_test(async || {
+            set_runtime_error(None);
+            let (_root, store) = native_pause_backlog_store(1, 24).await;
+            let now = unix_now();
+            let controller = ScannerPauseBacklogController::claim(Arc::clone(&store), now)
+                .await
+                .expect("bootstrap the single-pool ledger");
+            let stable = controller.loaded.ledger.clone();
+            let ids = scanner_pause_backlog_replica_ids(&controller.loaded.replicas);
+            assert_eq!(ids.len(), 24);
+            drop(controller);
+            let first = claim_scanner_pause_backlog_writer(&stable, now + 1).expect("first valid fork");
+            let second = claim_scanner_pause_backlog_writer(&stable, now + 2).expect("second valid fork");
+            for (index, id) in ids.iter().enumerate() {
+                let ledger = if index < 12 { &first } else { &second };
+                let commit = ScannerPauseBacklogCommitRecord::new(ledger.clone(), ids.clone());
+                replace_native_pause_backlog_replica(
+                    &store,
+                    *id,
+                    ScannerPauseBacklogReplicaRecord::new(Some(ledger.clone()), Some(commit)),
+                )
+                .await;
+            }
+            let mut before = Vec::with_capacity(24);
+            for set in &store.pools[0].disk_set {
+                before.push(native_replica_bytes(set).await);
+            }
+            let error = match ScannerPauseBacklogController::claim(Arc::clone(&store), now + 3).await {
+                Ok(_) => panic!("conflicting rollback proofs must never be reset or arbitrarily selected"),
+                Err(error) => error,
+            };
+            assert!(
+                error.contains("neither a surviving membership commit nor a stable rollback point"),
+                "{error}"
+            );
+            let status = scanner_pause_backlog_status(Arc::clone(&store)).await;
+            assert_eq!(status.persistence_state, "unavailable");
+            assert_eq!(status.replica_count, 24);
+            assert_eq!(status.replica_diagnostics.as_ref().map(Vec::len), Some(24));
+            for (index, set) in store.pools[0].disk_set.iter().enumerate() {
+                assert_eq!(
+                    native_replica_bytes(set).await,
+                    before[index],
+                    "failed recovery must preserve bytes and CAS revisions"
+                );
+            }
+            shutdown_native_retirement_store(store).await;
+            set_runtime_error(None);
+        });
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn native_status_reports_missing_and_conflicting_pause_backlog_replicas() {
+        run_native_retirement_test(async || {
+            set_runtime_error(None);
+            let (_root, store) = native_retirement_store().await;
+
+            let empty = scanner_pause_backlog_status(Arc::clone(&store)).await;
+            assert_eq!(empty.persistence_state, "missing");
+            assert_eq!(empty.replica_count, 6);
+            assert_eq!(
+                empty.replica_diagnostics.as_ref().map(Vec::len),
+                Some(6),
+                "initial all-missing membership should be visible without being treated as an error"
+            );
+            assert!(
+                empty
+                    .replica_diagnostics
+                    .as_ref()
+                    .expect("missing replicas should be diagnosed")
+                    .iter()
+                    .all(|replica| replica.state == ScannerPauseBacklogReplicaStatusState::Missing)
+            );
+
+            let stable = durable_ledger(unix_now());
+            let members = (0..3)
+                .flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
+                .collect::<Vec<_>>();
+            let committed = ScannerPauseBacklogCommitRecord::new(stable.clone(), members);
+            let record = ScannerPauseBacklogReplicaRecord::new(Some(stable), Some(committed));
+            let bytes = encode_scanner_pause_backlog_record(&record).expect("stable replica should encode");
+            Arc::clone(&store)
+                .save_scanner_pause_backlog_replica(0, 0, bytes, DataUsageCacheRevision::Missing.preconditions())
+                .await
+                .expect("native fixture should publish its one stable replica");
+
+            let conflicted = scanner_pause_backlog_status(Arc::clone(&store)).await;
+            assert_eq!(conflicted.persistence_state, "unavailable");
+            assert_eq!(conflicted.replica_count, 6, "failed selection must preserve the actual replica count");
+            assert!(
+                conflicted
+                    .error
+                    .as_deref()
+                    .is_some_and(|error| error.contains("neither a surviving membership commit"))
+            );
+            let diagnostics = conflicted
+                .replica_diagnostics
+                .as_ref()
+                .expect("authority conflict should expose per-replica diagnostics");
+            assert_eq!(diagnostics.len(), 6);
+            assert!(diagnostics.iter().any(|replica| {
+                replica.pool_index == 0
+                    && replica.set_index == 0
+                    && replica.state == ScannerPauseBacklogReplicaStatusState::Valid
+                    && replica.stable_generation == Some(1)
+                    && replica.committed_generation == Some(1)
+                    && replica.committed_replica_count == Some(6)
+                    && replica.revision.is_some()
+            }));
+            assert_eq!(
+                diagnostics
+                    .iter()
+                    .filter(|replica| replica.state == ScannerPauseBacklogReplicaStatusState::Missing)
+                    .count(),
+                5
+            );
+            let conflicted_json = serde_json::to_value(&conflicted).expect("status diagnostics should encode as JSON");
+            let conflicted_diagnostics = conflicted_json["replica_diagnostics"]
+                .as_array()
+                .expect("replica diagnostics should serialize as an array");
+            assert!(
+                conflicted_diagnostics
+                    .iter()
+                    .any(|replica| { replica["pool_index"] == 0 && replica["set_index"] == 0 && replica["state"] == "valid" })
+            );
+
+            for (pool_index, set_index, bytes) in [
+                (0, 1, b"invalid backlog JSON".to_vec()),
+                (1, 0, br#"{"replica_schema_version":999}"#.to_vec()),
+            ] {
+                Arc::clone(&store)
+                    .save_scanner_pause_backlog_replica(
+                        pool_index,
+                        set_index,
+                        bytes,
+                        DataUsageCacheRevision::Missing.preconditions(),
+                    )
+                    .await
+                    .expect("native fixture should publish malformed and future-schema replicas");
+            }
+
+            let classified = scanner_pause_backlog_status(Arc::clone(&store)).await;
+            assert!(
+                classified
+                    .error
+                    .as_deref()
+                    .is_some_and(|error| error.contains("future schema 999"))
+            );
+            let classified_json = serde_json::to_value(&classified).expect("classified status should encode as JSON");
+            let classified_diagnostics = classified_json["replica_diagnostics"]
+                .as_array()
+                .expect("classified diagnostics should serialize as an array");
+            assert!(
+                classified_diagnostics
+                    .iter()
+                    .any(|replica| { replica["pool_index"] == 0 && replica["set_index"] == 1 && replica["state"] == "invalid" })
+            );
+            assert!(classified_diagnostics.iter().any(|replica| {
+                replica["pool_index"] == 1 && replica["set_index"] == 0 && replica["state"] == "future_schema"
+            }));
+            assert!(classified_diagnostics.iter().any(|replica| {
+                replica["pool_index"] == 0
+                    && replica["set_index"] == 1
+                    && replica["error"].as_str().is_some_and(|reason| !reason.is_empty())
+            }));
+
+            let unavailable = scanner_pause_backlog_replica_diagnostics(vec![ScannerPauseBacklogReplica {
+                id: replica_id(9, 9),
+                revision: None,
+                state: ScannerPauseBacklogReplicaState::Unavailable("read failed".to_string()),
+            }]);
+            assert_eq!(unavailable[0].state, ScannerPauseBacklogReplicaStatusState::Unavailable);
+            assert_eq!(unavailable[0].error.as_deref(), Some("read failed"));
+
+            shutdown_native_retirement_store(store).await;
+            set_runtime_error(None);
+        });
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn native_status_omits_replica_diagnostics_for_healthy_backlog() {
+        run_native_retirement_test(async || {
+            set_runtime_error(None);
+            let (_root, store) = native_retirement_store().await;
+            ScannerPauseBacklogController::claim(Arc::clone(&store), unix_now())
+                .await
+                .expect("all-missing native backlog should claim and stabilize");
+
+            let status = scanner_pause_backlog_status(Arc::clone(&store)).await;
+            assert_eq!(status.persistence_state, "healthy");
+            assert!(status.durable);
+            assert_eq!(status.replica_count, 6);
+            assert!(status.replica_diagnostics.is_none());
+            let status_json = serde_json::to_value(status).expect("healthy status should encode");
+            assert!(status_json.get("replica_diagnostics").is_none());
+
+            shutdown_native_retirement_store(store).await;
+            set_runtime_error(None);
+        });
+    }
+
+    #[test]
+    #[serial_test::serial]
+    fn native_status_reports_competing_maximum_membership_commits() {
+        run_native_retirement_test(async || {
+            set_runtime_error(None);
+            let (_root, store) = native_retirement_store().await;
+            let controller = ScannerPauseBacklogController::claim(Arc::clone(&store), unix_now())
+                .await
+                .expect("all-missing native backlog should claim and stabilize");
+            let stable = controller.loaded.ledger.clone();
+            let replica_ids = scanner_pause_backlog_replica_ids(&controller.loaded.replicas);
+            drop(controller);
+
+            let split = replica_ids.len() / 2;
+            let first_membership = replica_ids[..split].to_vec();
+            let second_membership = replica_ids[split..].to_vec();
+            let now = stable.last_updated_at_unix_secs.saturating_add(10);
+            let first_ledger = claim_scanner_pause_backlog_writer(&stable, now).expect("first proof should advance");
+            let second_ledger = claim_scanner_pause_backlog_writer(&stable, now + 1).expect("second proof should advance");
+            assert_ne!(first_ledger, second_ledger);
+            let first_commit = ScannerPauseBacklogCommitRecord::new(first_ledger, first_membership);
+            let second_commit = ScannerPauseBacklogCommitRecord::new(second_ledger, second_membership);
+
+            for (index, id) in replica_ids.iter().enumerate() {
+                let committed = if index < split { &first_commit } else { &second_commit };
+                replace_native_pause_backlog_replica(
+                    &store,
+                    *id,
+                    ScannerPauseBacklogReplicaRecord::new(Some(stable.clone()), Some(committed.clone())),
+                )
+                .await;
+            }
+
+            let status = scanner_pause_backlog_status(Arc::clone(&store)).await;
+            assert_eq!(status.persistence_state, "unavailable");
+            assert_eq!(status.replica_count, replica_ids.len());
+            assert!(
+                status
+                    .error
+                    .as_deref()
+                    .is_some_and(|error| error.contains("conflicting maximum-membership commit proofs"))
+            );
+            let diagnostics = status
+                .replica_diagnostics
+                .as_ref()
+                .expect("competing proofs should expose replica diagnostics");
+            assert_eq!(diagnostics.len(), replica_ids.len());
+            assert!(
+                diagnostics
+                    .iter()
+                    .all(|replica| replica.committed_replica_count == Some(split) && replica.revision.is_some())
+            );
+            let revisions = diagnostics
+                .iter()
+                .filter_map(|replica| replica.revision.as_ref())
+                .collect::<std::collections::BTreeSet<_>>();
+            assert_eq!(revisions.len(), 2, "each competing proof cohort should have its own record revision");
+
+            shutdown_native_retirement_store(store).await;
+            set_runtime_error(None);
+        });
+    }
+
    async fn shutdown_native_retirement_store(store: Arc<ECStore>) {
        if let Some(token) = store.background_cancel_token() {
            token.cancel();
@@ -2027,6 +2573,21 @@ mod tests {
        (bytes, revision)
    }

+    async fn replace_native_pause_backlog_replica(
+        store: &Arc<ECStore>,
+        id: ScannerPauseBacklogReplicaId,
+        record: ScannerPauseBacklogReplicaRecord,
+    ) {
+        let set = Arc::clone(&store.pools[id.pool_index].disk_set[id.set_index]);
+        let current = read_scanner_pause_backlog_replica(set).await;
+        let revision = current.revision.expect("native backlog record should have a CAS revision");
+        let bytes = encode_scanner_pause_backlog_record(&record).expect("native backlog record should encode");
+        Arc::clone(store)
+            .save_scanner_pause_backlog_replica(id.pool_index, id.set_index, bytes, revision.preconditions())
+            .await
+            .expect("native backlog replica should accept its current CAS revision");
+    }
+
    async fn assert_native_source_missing(store: &ECStore, set_index: usize) {
        let source = store.pools[0].disk_set[set_index]
            .get_object_reader(
@@ -3969,11 +4530,13 @@ mod tests {
                assert_eq!(ledger.phase, ScannerPauseBacklogPhase::CatchingUp);
                assert!(ledger.pending_full_scan);
                assert_eq!(ledger.next_attempt_at_unix_secs, 721);
-                let status = status_from_ledger(&ledger, 422, "healthy".to_string(), true, 1, 1, 0, false, false, None);
+                let status = status_from_ledger(&ledger, 422, "healthy".to_string(), true, 1, 1, 0, false, false, None, None);
                assert_eq!(status.phase, ScannerPauseBacklogPhase::CatchingUp);
                assert!(status.pending_full_scan);
                assert_eq!(status.pending_work_items, 1);
                assert!(status.rate_limited);
+                let status_json = serde_json::to_value(status).expect("healthy status should encode");
+                assert!(status_json.get("replica_diagnostics").is_none());
            }
        }

@@ -700,6 +700,143 @@ async fn cycle_budget_deadline_handler_fences_and_releases_guard() {
    global_metrics().set_cycle(None).await;
 }

+async fn checkpoint_runtime_handoff(store: Arc<MemoryConfigStore>, expected_cycle: u64, mut epoch: u64) -> (u64, u64) {
+    let (encoded, mut revision) = read_config_with_revision(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str())
+        .await
+        .expect("read the previous durable generation");
+    let (mut cycle, saved_epoch) =
+        decode_scanner_cycle_state(&encoded.expect("a preceding generation must exist")).expect("decode preceding cycle");
+    assert_eq!((cycle.next, saved_epoch), (expected_cycle, epoch));
+    cycle.current = cycle.next;
+    let ctx = CancellationToken::new();
+    let mut metrics = ScannerCycleMetricsGuard::new(cycle.clone()).await;
+    tokio::time::pause();
+    let budget = ScannerCycleBudget::new(
+        &ctx,
+        ScannerCycleBudgetConfig {
+            max_duration: Some(Duration::from_secs(5)),
+            ..Default::default()
+        },
+    );
+    let worker = async {
+        budget.token().cancelled().await;
+        assert!(
+            finalize_partial_scan_cycle_for_epoch(&ctx, store.clone(), &mut cycle, &mut revision, epoch, &mut metrics, Some(0),)
+                .await,
+            "runtime expiry must durably advance the partial cycle"
+        );
+        budget.mark_cycle_state_persisted();
+    };
+    let outcome = await_scanner_cycle_with_budget_fence(&ctx, &budget, worker, std::future::pending()).await;
+    tokio::time::resume();
+    assert_eq!(outcome, ScannerCycleWaitOutcome::Deadline { worker_stopped: true });
+    assert_eq!(budget.reason(), Some(ScannerCycleBudgetReason::Runtime));
+    assert!(budget.cycle_state_persisted());
+    assert!(
+        fence_scanner_epoch_after_cycle_timeout(
+            &ctx,
+            store.clone(),
+            &mut cycle,
+            &mut revision,
+            &mut epoch,
+            false,
+            std::future::pending(),
+        )
+        .await
+    );
+    assert!(
+        claim_scanner_leadership(
+            &ctx,
+            store.clone(),
+            &mut cycle,
+            &mut revision,
+            &mut epoch,
+            false,
+            ScannerCycleResetPolicy::None,
+        )
+        .await
+    );
+    let persisted = read_config(store, &DATA_USAGE_BLOOM_NAME_PATH)
+        .await
+        .expect("read durable takeover fence");
+    let (saved, saved_epoch) = decode_scanner_cycle_state(&persisted).expect("decode takeover fence");
+    assert_eq!((saved.next, saved_epoch), (cycle.next, epoch));
+    (cycle.next, epoch)
+}
+
+#[tokio::test]
+#[serial]
+async fn checkpoint_fixture_runtime_deadline_save_reload_resume() {
+    // One control store survives every handoff, as it does across real cycles.
+    let store = Arc::new(MemoryConfigStore::default());
+    let ctx = CancellationToken::new();
+    let mut usage = complete_usage_with_bucket_count(Some(std::time::SystemTime::UNIX_EPOCH), 0);
+    usage.scanner_epoch = Some(7);
+    save_config(
+        store.clone(),
+        DATA_USAGE_OBJ_NAME_PATH.as_str(),
+        serde_json::to_vec(&usage).expect("usage baseline"),
+    )
+    .await
+    .expect("seed usage fence once");
+    let mut cycle = CurrentCycle {
+        next: 11,
+        ..Default::default()
+    };
+    let mut revision = DataUsageCacheRevision::Missing;
+    assert!(persist_scanner_cycle_state(&ctx, store.clone(), &mut cycle, &mut revision, 7).await);
+    let checkpoint = crate::scanner_folder::run_checkpoint_fixture(false, |cycle, epoch| {
+        checkpoint_runtime_handoff(store.clone(), cycle, epoch)
+    })
+    .await;
+    let persisted = read_config(store.clone(), &DATA_USAGE_BLOOM_NAME_PATH)
+        .await
+        .expect("final durable control state");
+    let (saved, epoch) = decode_scanner_cycle_state(&persisted).expect("final control state");
+    assert_eq!((saved.next, epoch), (checkpoint.info.next_cycle, checkpoint.info.leader_epoch));
+    assert!(saved.next > 11 && epoch > 7);
+    let cache_name = "bucket/.usage-cache.bin";
+    let cache_revisions = crate::DataUsageCache::read_revisions(store.clone(), cache_name)
+        .await
+        .expect("initial cache revisions");
+    checkpoint
+        .save_with_revisions_for_epoch(store.clone(), cache_name, &cache_revisions, 0)
+        .await
+        .expect("publish the current checkpoint into the fenced store");
+    let cache_path = rustfs_utils::path::path_join_buf(&[crate::BUCKET_META_PREFIX, cache_name]);
+    let key = memory_config_key(RUSTFS_META_BUCKET, &cache_path);
+    let before = store.objects.lock().await.get(&key).cloned().expect("saved checkpoint bytes");
+    let mut revisions = crate::DataUsageCache::read_revisions(store.clone(), cache_name)
+        .await
+        .expect("current cache revisions");
+    let mut stale = checkpoint.clone();
+    stale.info.next_cycle = 11;
+    stale.info.leader_epoch = 7;
+    stale.info.snapshot_complete = false;
+    for late_cycle in [11, saved.next] {
+        let late = crate::scanner_io::persist_scanner_checkpoint(
+            store.clone(),
+            store.clone(),
+            crate::scanner_io::ScannerCheckpointPersistContext {
+                ctx: &ctx,
+                expected_publication_epoch: 0,
+                cycle: late_cycle,
+                leader_epoch: 7,
+            },
+            cache_name,
+            &stale,
+            &mut revisions,
+        )
+        .await;
+        assert!(matches!(late, crate::scanner_io::ScannerCheckpointPersistResult::FenceChanged));
+        assert_eq!(
+            store.objects.lock().await.get(&key),
+            Some(&before),
+            "late writes cannot overwrite the current checkpoint"
+        );
+    }
+}
+
 #[tokio::test]
 async fn scanner_cycle_recovery_wake_survives_wait_registration_race() {
    notify_scanner_cycle_recovery_wake();
@@ -3004,5 +3004,8 @@ use item_actions::*;
 pub use item_actions::{GetSizeFn, ScannerItem};
 use ledger::*;

+#[cfg(test)]
+pub(crate) use tests::checkpoint_fixture::run_checkpoint_fixture;
+
 #[cfg(test)]
 mod tests;
@@ -26,7 +26,7 @@ use std::io::Write;
 use std::os::unix::fs::{PermissionsExt, symlink};
 use std::sync::Mutex;

-mod checkpoint_fixture;
+pub(super) mod checkpoint_fixture;
 pub(super) mod enumeration_restart;
 mod incremental_enumeration;

@@ -870,13 +870,13 @@ async fn check_complete_sampling_resumption(resume_mode: HealScanMode) {
 #[tokio::test]
 #[serial]
 async fn checkpoint_fixture_save_reload_resume_across_cycles_and_leaders() {
-    run_checkpoint_fixture(false).await;
+    run_checkpoint_fixture(false, |cycle, epoch| std::future::ready((cycle + 1, epoch + 1))).await;
 }

 #[tokio::test]
 #[serial]
 async fn checkpoint_fixture_hot_digest_retains_partial_progress() {
-    run_checkpoint_fixture(true).await;
+    run_checkpoint_fixture(true, |cycle, epoch| std::future::ready((cycle + 1, epoch + 1))).await;
 }

 /// A bucket written between every cycle requests a new plan each round. Once
@@ -988,7 +988,98 @@ async fn write_checkpoint_object(root: &std::path::Path, object: &str, versions:
    write_test_object_metadata_bytes(root, "bucket", object, &metadata.marshal_msg().expect("fixture metadata")).await;
 }

-async fn run_checkpoint_fixture(change_digest: bool) {
+pub(super) async fn scan_budgeted_metadata_repair_tail() -> DataUsageCache {
+    let (scanner, root) = build_test_scanner().await;
+    let _guard = TestGuard {
+        temp_dir: Some(root.clone()),
+    };
+    for index in 0..8 {
+        write_checkpoint_object(&root, &format!("static/{index:04}"), &[(None, 1)]).await;
+    }
+    let tail = "static/zz-failed";
+    write_test_object_metadata_bytes(&root, "bucket", tail, b"not-valid-filemeta").await;
+    let identity = crate::DataUsageScanIdentity {
+        version: 1,
+        bucket_incarnation: Uuid::from_u128(1),
+        set_layout: DataUsageScanPlanDigest([41; 32]),
+        publication_epoch: 0,
+        tier_registry_generation: crate::runtime_tier_registry_for_cycle(11, 7).await.generation,
+        scan_mode: HealScanMode::Normal,
+    };
+    let store = FixtureStore::new();
+    let mut retained_prefix = 0;
+    for round in 0..12_u64 {
+        let mut cache = DataUsageCache::default();
+        let revisions = cache
+            .load_with_revisions(store.clone(), CACHE_NAME)
+            .await
+            .expect("load repair-tail checkpoint");
+        assert_eq!(
+            cache.prepare_bucket_checkpoint("bucket", 11 + round, 7 + round, SOURCE, PLAN, identity),
+            if round == 0 {
+                crate::DataUsageCachePrepareOutcome::Reset
+            } else {
+                crate::DataUsageCachePrepareOutcome::Reused
+            },
+            "a bounded repair scan must keep its verified prefix across the handoff"
+        );
+        cache.info.skip_healing = false;
+        let ctx = CancellationToken::new();
+        let budget = ScannerCycleBudget::new_with_progress_tracking(
+            &ctx,
+            ScannerCycleBudgetConfig {
+                max_objects: Some(4),
+                ..Default::default()
+            },
+        );
+        let outcome = scanner
+            .local_disk
+            .clone()
+            .nsscanner_disk(
+                budget.token(),
+                budget.clone(),
+                vec![scanner.local_disk.clone()],
+                cache,
+                None,
+                scan_options(HealScanMode::Normal),
+            )
+            .await
+            .expect("required metadata repair must produce a partial scan");
+        let ScannerDiskScanOutcome::Partial(cache) = outcome else {
+            panic!("unrepaired metadata cannot certify a complete bucket")
+        };
+        assert!(!cache.info.skip_healing);
+        assert!(!cache.info.snapshot_complete);
+        cache
+            .save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
+            .await
+            .expect("persist scanner repair obligation with CAS");
+        let saved = store.strict_load().await;
+        if let Some(pending) = saved
+            .info
+            .pending_heals
+            .iter()
+            .find(|entry| entry.object.as_deref() == Some(tail))
+        {
+            assert!(round > 0, "the repair target must be outside the first bounded prefix");
+            assert_eq!(pending.bucket, "bucket");
+            assert_eq!(pending.kind, PendingScannerHealKind::Object);
+            assert!(!saved.info.snapshot_complete);
+            assert!(retained_prefix >= 4, "the saved prefix must advance before discovering the tail");
+            return saved;
+        }
+        let retained_now = retained(&saved);
+        assert!(retained_now > retained_prefix, "the bounded scan must retain new prefix observations");
+        retained_prefix = retained_now;
+    }
+    panic!("required metadata repair at the budgeted tail was never discovered");
+}
+
+pub(crate) async fn run_checkpoint_fixture<Handoff, Step>(change_digest: bool, mut handoff: Handoff) -> DataUsageCache
+where
+    Handoff: FnMut(u64, u64) -> Step,
+    Step: std::future::Future<Output = (u64, u64)>,
+{
    let (scanner, root) = build_test_scanner().await;
    let _guard = TestGuard {
        temp_dir: Some(root.clone()),
@@ -1007,6 +1098,8 @@ async fn run_checkpoint_fixture(change_digest: bool) {
    let store = FixtureStore::new();
    let mut previous = 0;
    let mut visited = 0;
+    let mut cycle = 11;
+    let mut epoch = 7;
    for round in 0..3_u8 {
        write_checkpoint_object(&root, "hot/current", &[(None, 1)]).await;
        let mut cache = DataUsageCache::default();
@@ -1023,8 +1116,8 @@ async fn run_checkpoint_fixture(change_digest: bool) {
            &mut cache,
            "bucket",
            SOURCE,
-            11 + u64::from(round),
-            7 + u64::from(round),
+            cycle,
+            epoch,
            plan,
            crate::scanner_io::DataUsageCacheReuseOptions {
                require_source: true,
@@ -1091,6 +1184,7 @@ async fn run_checkpoint_fixture(change_digest: bool) {
        assert!(loaded.info.scan_plan_digest.is_none(), "old readers must rebuild an uncertified sweep");
        crate::remote_scanner::checkpoint_fixture_partial_return(budget.progress(), budget.entries_visited()).await;
        previous = reloaded;
+        (cycle, epoch) = handoff(loaded.info.next_cycle, loaded.info.leader_epoch).await;
    }
    assert!(visited > 0, "fixture must exercise the directory walk");
    assert!(previous > 0, "fixture must retain and enumerate static subtree entries");
@@ -1151,7 +1245,7 @@ async fn run_checkpoint_fixture(change_digest: bool) {
    write_checkpoint_object(&root, "hot/later", &[(None, 1)]).await;
    let final_plan = crate::scanner_io::checkpoint_fixture_bucket_digest(PLAN, Some(3));
    let mut saw_mixed_sweep_end = false;
-    for round in 0..32 {
+    for _ in 0..32 {
        let mut cache = DataUsageCache::default();
        let revisions = cache
            .load_with_revisions(store.clone(), CACHE_NAME)
@@ -1161,8 +1255,8 @@ async fn run_checkpoint_fixture(change_digest: bool) {
            &mut cache,
            "bucket",
            SOURCE,
-            14 + round,
-            10 + round,
+            cycle,
+            epoch,
            final_plan,
            crate::scanner_io::DataUsageCacheReuseOptions {
                require_source: true,
@@ -1215,13 +1309,14 @@ async fn run_checkpoint_fixture(change_digest: bool) {
            assert_eq!((total.objects, total.versions, total.size), (25, 2, 34));
            assert_eq!(saved.checked_flatten("bucket/static").expect("static subtree").objects, 23);
            assert_eq!(saved.checked_flatten("bucket/hot").expect("hot subtree").objects, 2);
-            return;
+            return saved;
        }
        assert!(!saved.info.snapshot_complete);
        assert!(saved.info.scan_plan_digest.is_none());
        if !budget.budget_elapsed() {
            saw_mixed_sweep_end = true;
        }
+        (cycle, epoch) = handoff(saved.info.next_cycle, saved.info.leader_epoch).await;
    }
    panic!("finite stable fixture must converge using the same four-object budget without an unbounded final sweep");
 }
@@ -219,6 +219,7 @@ async fn mrf_ownership_cancelled_batch_restores_sync_without_per_item_clones() {
 struct NoticeStorage {
    calls: std::sync::Mutex<HashMap<String, u32>>,
    retry_started: tokio::sync::Notify,
+    first_started: tokio::sync::Notify,
    bucket_incarnation_id: Uuid,
 }

@@ -272,6 +273,9 @@ impl HealStorageAPI for NoticeStorage {
            *count += 1;
            *count
        };
+        if call == 1 {
+            self.first_started.notify_one();
+        }
        if call > 1 {
            if call == 2 {
                self.retry_started.notify_one();
@@ -310,27 +314,106 @@ impl HealStorageAPI for NoticeStorage {
    }
 }

+/// Singleton channels need a fresh process. Reuse nextest isolation when its
+/// startup environment already satisfies the fixture; otherwise configure it
+/// before launching the single-test child, including the cargo-test fallback.
+fn run_isolated_mrf_fixture(test_name: &str, environment: &[(&str, &str)]) -> bool {
+    const CHILD: &str = "RUSTFS_MRF_FIXTURE_CHILD";
+    let child = std::env::var(CHILD).ok().as_deref() == Some(test_name);
+    let isolated = std::env::var("NEXTEST_EXECUTION_MODE").ok().as_deref() == Some("process-per-test")
+        && std::env::var("NEXTEST_TEST_NAME").ok().as_deref() == Some(test_name);
+    let enabled = rustfs_utils::get_env_bool(rustfs_config::ENV_HEAL_MRF_ENABLE, rustfs_config::DEFAULT_HEAL_MRF_ENABLE);
+    let configured = environment
+        .iter()
+        .all(|(key, value)| std::env::var(key).ok().as_deref() == Some(*value));
+    if child || (isolated && enabled && configured) {
+        return false;
+    }
+    let output = std::process::Command::new(std::env::current_exe().expect("test executable"))
+        .args(["--exact", test_name, "--nocapture"])
+        .env(CHILD, test_name)
+        .env(rustfs_config::ENV_HEAL_MRF_ENABLE, "true")
+        .envs(environment.iter().copied())
+        .output()
+        .expect("isolated MRF fixture process");
+    let stdout = String::from_utf8_lossy(&output.stdout);
+    assert!(
+        output.status.success() && stdout.contains("1 passed;"),
+        "{stdout}\n{}",
+        String::from_utf8_lossy(&output.stderr)
+    );
+    true
+}
+
+#[tokio::test]
+#[serial]
+async fn checkpoint_fixture_metadata_heal_reaches_budgeted_tail() {
+    // Required metadata repair is independent of probabilistic object sampling.
+    if run_isolated_mrf_fixture(
+        "scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail",
+        &[
+            ("RUSTFS_HEAL_OBJECT_SELECT_PROB", "0"),
+            ("NO_PROXY", "localhost,127.0.0.1,::1"),
+        ],
+    ) {
+        return;
+    }
+    let journal_root = tempfile::tempdir().expect("repair-tail MRF journal");
+    let _journal_env = rustfs_test_utils::TestECStoreEnv::builder()
+        .base_dir(journal_root.path())
+        .build()
+        .await;
+    let storage = Arc::new(NoticeStorage {
+        bucket_incarnation_id: Uuid::from_u128(1),
+        ..Default::default()
+    });
+    let manager = Arc::new(HealManager::new(
+        storage.clone(),
+        Some(HealConfig {
+            enable_auto_heal: false,
+            mainline_throttle_enable: false,
+            ..Default::default()
+        }),
+    ));
+    manager.start().await.expect("start real heal manager");
+    spawn_mrf_consumer(manager.clone());
+    let saved = super::checkpoint_fixture::scan_budgeted_metadata_repair_tail().await;
+    tokio::time::timeout(Duration::from_secs(30), async {
+        loop {
+            if storage
+                .calls
+                .lock()
+                .expect("repair calls")
+                .get("static/zz-failed")
+                .copied()
+                .unwrap_or(0)
+                > 0
+            {
+                break;
+            }
+            storage.first_started.notified().await;
+        }
+    })
+    .await
+    .expect("the real MRF consumer must dispatch the discovered tail repair");
+    assert!(
+        saved
+            .info
+            .pending_heals
+            .iter()
+            .any(|entry| { entry.bucket == "bucket" && entry.object.as_deref() == Some("static/zz-failed") }),
+        "the dispatched repair must already be recorded in the saved scanner ledger"
+    );
+    manager.stop().await.expect("stop repair-tail manager");
+}
+
 #[tokio::test]
 #[serial]
 async fn mrf_ownership_manager_completion_preserves_scanner_pending() {
-    const CHILD: &str = "RUSTFS_MRF_OWNERSHIP_TEST_CHILD";
-    if std::env::var_os(CHILD).is_none() {
-        let output = std::process::Command::new(std::env::current_exe().expect("test executable"))
-            .args([
-                "--exact",
-                "scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending",
-                "--nocapture",
-            ])
-            .env(CHILD, "1")
-            .env("RUSTFS_HEAL_MRF_ENABLE", "true")
-            .output()
-            .expect("isolated ingress test process");
-        let stdout = String::from_utf8_lossy(&output.stdout);
-        assert!(
-            output.status.success() && stdout.contains("1 passed;"),
-            "{stdout}\n{}",
-            String::from_utf8_lossy(&output.stderr)
-        );
+    if run_isolated_mrf_fixture(
+        "scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending",
+        &[],
+    ) {
        return;
    }
    // The production ingress channel is a process singleton; isolation keeps
@@ -69,10 +69,13 @@ async fn wait_for_checkpoint_foreground_quiet(ctx: &CancellationToken) -> bool {
 /// Persist one bounded checkpoint and refresh its CAS revisions.
 ///
 /// Local and remote workers share the same publication/leader fencing and
-/// revision-refresh contract; only their lock/cancellation handling remains
+/// revision-refresh contract. Cycle/leader state belongs to the global store,
+/// while cache revisions and publication admission belong to the set store.
+/// Only their lock/cancellation handling remains
 /// at the caller because those guards have different concrete types.
-pub(crate) async fn persist_scanner_checkpoint<S>(
+pub(crate) async fn persist_scanner_checkpoint<S, F>(
    store: Arc<S>,
+    fence_store: Arc<F>,
    context: ScannerCheckpointPersistContext<'_>,
    cache_name: &str,
    checkpoint: &DataUsageCache,
@@ -80,6 +83,7 @@ pub(crate) async fn persist_scanner_checkpoint<S>(
 ) -> ScannerCheckpointPersistResult
 where
    S: ScannerObjectIO + ScannerConfigObjectDelete,
+    F: ScannerObjectIO,
 {
    let foreground_quiet = wait_for_checkpoint_foreground_quiet(context.ctx).await;
    if !foreground_quiet && context.ctx.is_cancelled() {
@@ -97,9 +101,13 @@ where
        );
    }

-    if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, store.clone())
-        .await
-        .is_err()
+    if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(
+        context.cycle,
+        context.leader_epoch,
+        fence_store.clone(),
+    )
+    .await
+    .is_err()
    {
        return ScannerCheckpointPersistResult::FenceChanged;
    }
@@ -117,7 +125,7 @@ where
        return ScannerCheckpointPersistResult::Failed(error);
    }

-    if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, store.clone())
+    if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, fence_store)
        .await
        .is_err()
        || scanner_publication_admission_for_epoch(store.clone(), context.expected_publication_epoch)
@@ -1189,8 +1189,14 @@ impl ScannerIOCache for SetDisks {
                                    await_scanner_disk_shutdown(scan.as_mut()).await;
                                    break Err(Error::other("scanner leader fence changed before checkpoint save"));
                                }
+                                let Some(fence_store) = crate::resolve_scanner_object_store_handle() else {
+                                    scan_ctx.cancel();
+                                    await_scanner_disk_shutdown(scan.as_mut()).await;
+                                    break Err(Error::other("scanner global cycle fence store is unavailable before checkpoint save"));
+                                };
                                match persist_scanner_checkpoint(
                                    store_clone_clone.clone(),
+                                    fence_store,
                                    ScannerCheckpointPersistContext {
                                        ctx: &ctx_clone,
                                        expected_publication_epoch: expected_publication_epoch_clone,
@@ -27,9 +27,10 @@ use crate::storage_api::owner::{
 };
 use crate::storage_api::scan::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, ObjectIO as _};
 use crate::{
-    DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints, ScannerObjectOptions,
-    ScannerPutObjReader, UNKNOWN_TIER, init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests,
-    init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
+    DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints,
+    ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER, init_bucket_metadata_sys_for_scanner_tests,
+    init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
+    save_config,
 };
 use rustfs_concurrency::{
    AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider,
@@ -45,6 +46,7 @@ use uuid::Uuid;

 mod scoped_entry_fallback;
 mod service_cohort;
+mod usage_expiration;

 #[derive(Clone)]
 struct FixedWorkloadProvider {
@@ -521,12 +523,16 @@ fn complete_set_cache_with_segment_proof(
 }

 async fn setup_two_pool_scanner_store() -> (tempfile::TempDir, Arc<ECStore>) {
+    setup_local_scanner_store(2, 4).await
+}
+
+async fn setup_local_scanner_store(pool_count: usize, drives_per_set: usize) -> (tempfile::TempDir, Arc<ECStore>) {
    init_ecstore_config_for_scanner_tests();
    let temp_dir = tempfile::tempdir().expect("multi-pool scanner test directory should be created");
    let mut pools = Vec::new();
-    for pool_index in 0..2 {
+    for pool_index in 0..pool_count {
        let mut endpoints = Vec::new();
-        for disk_index in 0..4 {
+        for disk_index in 0..drives_per_set {
            let disk_path = temp_dir.path().join(format!("pool{pool_index}-disk{disk_index}"));
            tokio::fs::create_dir_all(&disk_path)
                .await
@@ -541,7 +547,7 @@ async fn setup_two_pool_scanner_store() -> (tempfile::TempDir, Arc<ECStore>) {
        pools.push(PoolEndpoints {
            legacy: false,
            set_count: 1,
-            drives_per_set: 4,
+            drives_per_set,
            endpoints: Endpoints::from(endpoints),
            cmd_line: format!("scanner-cycle-pool-{pool_index}"),
            platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
@@ -566,6 +572,303 @@ async fn setup_two_pool_scanner_store() -> (tempfile::TempDir, Arc<ECStore>) {
    (temp_dir, store)
 }

+#[tokio::test]
+#[serial]
+async fn checkpoint_uses_global_cycle_fence_with_set_scoped_cache_revisions() {
+    let (_temp_dir, store) = setup_two_pool_scanner_store().await;
+    let cycle = 41341;
+    let leader_epoch = 17;
+    save_config(
+        store.clone(),
+        DATA_USAGE_BLOOM_NAME_PATH.as_str(),
+        crate::scanner::encode_scanner_cycle_fence_for_test(cycle, leader_epoch),
+    )
+    .await
+    .expect("persist cycle fence through global scanner store");
+    crate::remote_scanner::validate_remote_scanner_request_fence_with_store(cycle, leader_epoch, store.clone())
+        .await
+        .expect("global scanner store must read the fence");
+    let ctx = CancellationToken::new();
+    let cache_name = "checkpoint-bucket/.usage-cache.bin";
+    let mut checkpoint = DataUsageCache::default();
+    checkpoint.info.name = "checkpoint-bucket".to_string();
+    checkpoint.info.next_cycle = cycle;
+    checkpoint.info.leader_epoch = leader_epoch;
+    let mut mismatched_sets = 0;
+    for (pool_index, pool) in store.pools.iter().enumerate() {
+        for (set_index, set) in pool.disk_set.iter().enumerate() {
+            let publication_epoch = scanner_publication_epoch(set.clone())
+                .await
+                .expect("set publication admission");
+            DataUsageCache::default()
+                .save(set.clone(), cache_name)
+                .await
+                .expect("seed set-scoped cache");
+            let mut revisions = DataUsageCache::read_revisions(set.clone(), cache_name)
+                .await
+                .expect("initial set-scoped revisions");
+            let initial_revisions = revisions.clone();
+            let make_context = || ScannerCheckpointPersistContext {
+                ctx: &ctx,
+                expected_publication_epoch: publication_epoch,
+                cycle,
+                leader_epoch,
+            };
+            if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(cycle, leader_epoch, set.clone())
+                .await
+                .is_err()
+            {
+                mismatched_sets += 1;
+                assert!(
+                    matches!(
+                        persist_scanner_checkpoint(
+                            set.clone(),
+                            set.clone(),
+                            make_context(),
+                            cache_name,
+                            &checkpoint,
+                            &mut revisions,
+                        )
+                        .await,
+                        ScannerCheckpointPersistResult::FenceChanged
+                    ),
+                    "set-scoped cycle lookup must reproduce the original failure for pool={pool_index}, set={set_index}"
+                );
+            }
+            assert_eq!(revisions, initial_revisions, "rejected checkpoint must not update revisions");
+            for skip_healing in [false, true] {
+                checkpoint.info.skip_healing = skip_healing;
+                let before_save = revisions.clone();
+                assert!(
+                    matches!(
+                        persist_scanner_checkpoint(
+                            set.clone(),
+                            store.clone(),
+                            make_context(),
+                            cache_name,
+                            &checkpoint,
+                            &mut revisions,
+                        )
+                        .await,
+                        ScannerCheckpointPersistResult::Saved
+                    ),
+                    "checkpoint helper must use the global cycle fence"
+                );
+                assert_ne!(revisions, before_save, "new checkpoint content must advance cache revisions");
+            }
+            let saved_revisions = revisions.clone();
+            for rejected_context in [
+                ScannerCheckpointPersistContext {
+                    cycle: cycle - 1,
+                    ..make_context()
+                },
+                ScannerCheckpointPersistContext {
+                    leader_epoch: leader_epoch - 1,
+                    ..make_context()
+                },
+                ScannerCheckpointPersistContext {
+                    expected_publication_epoch: publication_epoch + 1,
+                    ..make_context()
+                },
+            ] {
+                assert!(
+                    matches!(
+                        persist_scanner_checkpoint(
+                            set.clone(),
+                            store.clone(),
+                            rejected_context,
+                            cache_name,
+                            &checkpoint,
+                            &mut revisions,
+                        )
+                        .await,
+                        ScannerCheckpointPersistResult::FenceChanged
+                    ),
+                    "stale cycle, leader or publication epoch must reject checkpoint persistence"
+                );
+                assert_eq!(revisions, saved_revisions, "rejected writer must not refresh revisions");
+                assert_eq!(
+                    DataUsageCache::read_revisions(set.clone(), cache_name)
+                        .await
+                        .expect("read rejected checkpoint"),
+                    saved_revisions,
+                    "rejected writer must not mutate persisted caches"
+                );
+            }
+
+            let mut newer = checkpoint.clone();
+            newer.info.next_cycle = cycle + 1;
+            newer
+                .save(set.clone(), cache_name)
+                .await
+                .expect("simulate a newer cache writer");
+            let newer_revisions = DataUsageCache::read_revisions(set.clone(), cache_name)
+                .await
+                .expect("capture newer cache revisions");
+            assert_ne!(newer_revisions, saved_revisions);
+            assert!(
+                matches!(
+                    persist_scanner_checkpoint(
+                        set.clone(),
+                        store.clone(),
+                        make_context(),
+                        cache_name,
+                        &checkpoint,
+                        &mut revisions,
+                    )
+                    .await,
+                    ScannerCheckpointPersistResult::Failed(StorageError::PreconditionFailed)
+                ),
+                "matching global fence must not authorize stale cache CAS revisions"
+            );
+            assert_eq!(revisions, saved_revisions, "CAS failure must not refresh the stale writer");
+            assert_eq!(
+                DataUsageCache::read_revisions(set.clone(), cache_name)
+                    .await
+                    .expect("read cache after CAS failure"),
+                newer_revisions,
+                "CAS failure must preserve the newer persisted caches"
+            );
+        }
+    }
+    assert!(mismatched_sets > 0, "multi-pool fixture must expose a set-scoped cycle fence mismatch");
+}
+
+#[derive(Debug)]
+struct CheckpointFenceStore {
+    before: Vec<u8>,
+    after: Vec<u8>,
+    fail_on_read: Option<usize>,
+    reads: std::sync::atomic::AtomicUsize,
+}
+
+#[async_trait::async_trait]
+impl crate::storage_api::owner::ObjectIO for CheckpointFenceStore {
+    type Error = crate::EcstoreError;
+    type RangeSpec = crate::storage_api::owner::HTTPRangeSpec;
+    type HeaderMap = http::HeaderMap;
+    type ObjectOptions = crate::ScannerObjectOptions;
+    type ObjectInfo = crate::ScannerObjectInfo;
+    type GetObjectReader = crate::ScannerGetObjectReader;
+    type PutObjectReader = crate::ScannerPutObjReader;
+
+    async fn get_object_reader(
+        &self,
+        bucket: &str,
+        object: &str,
+        _range: Option<Self::RangeSpec>,
+        _headers: Self::HeaderMap,
+        _opts: &Self::ObjectOptions,
+    ) -> crate::EcstoreResult<Self::GetObjectReader> {
+        assert_eq!(bucket, RUSTFS_META_BUCKET);
+        assert_eq!(object, DATA_USAGE_BLOOM_NAME_PATH.as_str(), "global store must only read the cycle fence");
+        let read = self.reads.fetch_add(1, Ordering::SeqCst);
+        if self.fail_on_read == Some(read) {
+            return Err(crate::EcstoreError::other("injected global fence read failure"));
+        }
+        let state = if read == 0 { &self.before } else { &self.after };
+        Ok(crate::ScannerGetObjectReader {
+            stream: Box::new(std::io::Cursor::new(state.clone())),
+            object_info: crate::ScannerObjectInfo::default(),
+            buffered_body: None,
+            body_source: Default::default(),
+        })
+    }
+
+    async fn put_object(
+        &self,
+        _bucket: &str,
+        _object: &str,
+        _data: &mut Self::PutObjectReader,
+        _opts: &Self::ObjectOptions,
+    ) -> crate::EcstoreResult<Self::ObjectInfo> {
+        panic!("checkpoint persistence must never write through the global fence store");
+    }
+}
+
+#[tokio::test]
+#[serial]
+async fn checkpoint_rejects_unreadable_or_changed_global_fence() {
+    let (_temp_dir, store) = setup_two_pool_scanner_store().await;
+    let set = &store.pools[1].disk_set[0];
+    let publication_epoch = scanner_publication_epoch(set.clone()).await.expect("idle set admission");
+    let cycle = 41341;
+    let leader_epoch = 17;
+    let valid = crate::scanner::encode_scanner_cycle_fence_for_test(cycle, leader_epoch);
+    let ctx = CancellationToken::new();
+    let mut checkpoint = DataUsageCache::default();
+    checkpoint.info.name = "checkpoint-bucket".to_string();
+    checkpoint.info.next_cycle = cycle;
+    checkpoint.info.leader_epoch = leader_epoch;
+
+    for (case, before, after, fail_on_read, expected_reads) in [
+        (
+            "cycle_changed",
+            valid.clone(),
+            crate::scanner::encode_scanner_cycle_fence_for_test(cycle + 1, leader_epoch),
+            None,
+            2,
+        ),
+        (
+            "leader_changed",
+            valid.clone(),
+            crate::scanner::encode_scanner_cycle_fence_for_test(cycle, leader_epoch + 1),
+            None,
+            2,
+        ),
+        ("post_save_corrupt", valid.clone(), vec![0; 7], None, 2),
+        ("post_save_read_failure", valid.clone(), valid.clone(), Some(1), 2),
+        ("pre_save_corrupt", vec![0; 7], valid.clone(), None, 1),
+        ("pre_save_read_failure", valid.clone(), valid.clone(), Some(0), 1),
+    ] {
+        let cache_name = format!("checkpoint-bucket/{case}/.usage-cache.bin");
+        DataUsageCache::default()
+            .save(set.clone(), &cache_name)
+            .await
+            .expect("seed checkpoint cache");
+        let mut revisions = DataUsageCache::read_revisions(set.clone(), &cache_name)
+            .await
+            .expect("read seeded revisions");
+        let initial = revisions.clone();
+        let fence_store = Arc::new(CheckpointFenceStore {
+            before,
+            after,
+            fail_on_read,
+            reads: std::sync::atomic::AtomicUsize::new(0),
+        });
+        assert!(
+            matches!(
+                persist_scanner_checkpoint(
+                    set.clone(),
+                    fence_store.clone(),
+                    ScannerCheckpointPersistContext {
+                        ctx: &ctx,
+                        expected_publication_epoch: publication_epoch,
+                        cycle,
+                        leader_epoch
+                    },
+                    &cache_name,
+                    &checkpoint,
+                    &mut revisions,
+                )
+                .await,
+                ScannerCheckpointPersistResult::FenceChanged
+            ),
+            "unreadable or changed global fence must reject checkpoint success: {case}"
+        );
+        assert_eq!(fence_store.reads.load(Ordering::SeqCst), expected_reads, "global fence checks: {case}");
+        assert_eq!(revisions, initial, "rejected checkpoint must not refresh caller revisions: {case}");
+        let persisted = DataUsageCache::read_revisions(set.clone(), &cache_name)
+            .await
+            .expect("read persisted checkpoint");
+        if expected_reads == 1 {
+            assert_eq!(persisted, initial, "pre-save rejection must not mutate persisted caches: {case}");
+        } else {
+            assert_ne!(persisted, initial, "post-save fence check must run after a real cache write: {case}");
+        }
+    }
+}
+
 async fn wait_for_namespace_commit_tails(store: &ECStore) {
    tokio::time::timeout(Duration::from_secs(30), async {
        while store.scanner_data_usage_publication_blocked().await {
@@ -0,0 +1,264 @@
+// Copyright 2026 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+use super::*;
+use crate::storage_api::owner::ObjectOperations as _;
+use crate::storage_api::scanner_io::{
+    BUCKET_LIFECYCLE_CONFIG, apply_bucket_usage_memory_overlay, get_bucket_usage_memory, init_background_expiry,
+    load_admin_data_usage_from_backend_cached, load_data_usage_from_backend, record_bucket_object_delete_memory,
+    record_bucket_object_write_memory, replace_bucket_usage_memory_from_info, seed_persisted_usage, update_bucket_metadata,
+};
+
+// Lifecycle expiry goes through the process-global store, which the first
+// store created in the process claims. Nextest runs this test in its own
+// process; the lifecycle scenarios share the store that owns expiry processing.
+#[tokio::test]
+#[serial]
+async fn lifecycle_expiration_usage_converges_across_complete_scans() {
+    let (_temp_dir, store) = setup_two_pool_scanner_store().await;
+    let next_cycle = check_persisted_stale_usage_recovery(&store, false, 1).await;
+    let next_cycle = check_persisted_stale_usage_recovery(&store, true, next_cycle + 1).await;
+    let next_cycle = check_usage_after_lifecycle_expiration(&store, "idle", false, next_cycle).await;
+    check_usage_after_lifecycle_expiration(&store, "busy", true, next_cycle).await;
+}
+
+#[tokio::test]
+#[serial]
+async fn historical_usage_converges_with_writes_on_one_drive() {
+    let (_temp_dir, store) = setup_local_scanner_store(1, 1).await;
+    check_persisted_stale_usage_recovery(&store, true, 1).await;
+}
+
+async fn check_usage_after_lifecycle_expiration(
+    store: &Arc<ECStore>,
+    scenario: &str,
+    write_between_scans: bool,
+    first_cycle: u64,
+) -> u64 {
+    clear_dirty_usage_buckets_for_tests();
+    let cases = [
+        ("ilm-empty", true, false),
+        ("ilm-partial", true, true),
+        ("delete-empty", false, false),
+        ("delete-partial", false, true),
+    ]
+    .map(|(name, lifecycle, keep_one)| (format!("{scenario}-{name}"), lifecycle, keep_one));
+    let usage_matches = |snapshot: &DataUsageInfo, retained_writes: u64| {
+        cases.iter().all(|(bucket, _, keep_one)| {
+            snapshot.buckets_usage.get(bucket).is_some_and(|usage| {
+                usage.objects_count == u64::from(*keep_one) + retained_writes
+                    && usage.size == (u64::from(*keep_one) + retained_writes) * 42
+            })
+        })
+    };
+    let mut baseline = DataUsageInfo {
+        last_update: Some(SystemTime::now()),
+        usage_snapshot_complete: true,
+        ..Default::default()
+    };
+    for (bucket, _, _) in &cases {
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create usage bucket");
+        baseline.buckets_usage.insert(bucket.clone(), BucketUsageInfo::default());
+        baseline.bucket_sizes.insert(bucket.clone(), 0);
+    }
+    baseline.buckets_count = 4;
+    replace_bucket_usage_memory_from_info(&baseline).await;
+
+    for (bucket, lifecycle, keep_one) in &cases {
+        for object in ["expire/first", if *keep_one { "keep/second" } else { "expire/second" }] {
+            let mut reader = ScannerPutObjReader::from_vec(vec![0; 42]);
+            store
+                .put_object(bucket, object, &mut reader, &ScannerObjectOptions::default())
+                .await
+                .expect("write object before expiration");
+            record_bucket_object_write_memory(bucket, None, 42).await;
+        }
+        if *lifecycle {
+            let config = br#"<LifecycleConfiguration><Rule><ID>expire</ID><Status>Enabled</Status><Filter><Prefix>expire/</Prefix></Filter><Expiration><Days>0</Days></Expiration></Rule></LifecycleConfiguration>"#;
+            update_bucket_metadata(bucket, BUCKET_LIFECYCLE_CONFIG, config.to_vec())
+                .await
+                .expect("configure immediate expiration after writes");
+        } else {
+            for object in ["expire/first", "expire/second"]
+                .into_iter()
+                .take(if *keep_one { 1 } else { 2 })
+            {
+                store
+                    .delete_object(bucket, object, ScannerObjectOptions::default())
+                    .await
+                    .expect("ordinary deletion should succeed");
+                record_bucket_object_delete_memory(bucket, 42, true).await;
+            }
+        }
+    }
+    init_background_expiry(store.clone()).await;
+    wait_for_namespace_commit_tails(store.as_ref()).await;
+
+    let mut reconciled = false;
+    let mut observed_expiration = false;
+    let mut retained_writes = 0;
+    let mut reconciled_cycles = 0;
+    let mut cycle = first_cycle;
+    while cycle < first_cycle + 8 {
+        let ctx = CancellationToken::new();
+        let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
+        let (updates, mut receiver) = mpsc::channel(4);
+        let result = tokio::time::timeout(
+            Duration::from_secs(30),
+            ScannerIOCycle::nsscanner_with_status(store.as_ref(), ctx, budget, updates, cycle, 11, HealScanMode::Normal),
+        )
+        .await
+        .expect("scan should finish")
+        .expect("scan should succeed");
+        while let Some(mut snapshot) = receiver.recv().await {
+            if result.status != ScannerCycleStatus::Complete || !snapshot.is_complete_bucket_usage_snapshot() {
+                continue;
+            }
+            if usage_matches(&snapshot, retained_writes) {
+                observed_expiration = true;
+                if write_between_scans {
+                    for (bucket, _, _) in &cases {
+                        let mut reader = ScannerPutObjReader::from_vec(vec![0; 42]);
+                        store
+                            .put_object(bucket, &format!("keep/write-{cycle}"), &mut reader, &ScannerObjectOptions::default())
+                            .await
+                            .expect("write between scan observation and publication");
+                        record_bucket_object_write_memory(bucket, None, 42).await;
+                    }
+                    retained_writes += 1;
+                    wait_for_namespace_commit_tails(store.as_ref()).await;
+                }
+                replace_bucket_usage_memory_from_info(&snapshot).await;
+                apply_bucket_usage_memory_overlay(&mut snapshot).await;
+                if usage_matches(&snapshot, retained_writes) {
+                    reconciled_cycles += 1;
+                    reconciled = !write_between_scans || reconciled_cycles >= 3;
+                    break;
+                }
+            }
+        }
+        cycle += 1;
+        if reconciled {
+            break;
+        }
+        tokio::time::sleep(Duration::from_millis(100)).await;
+    }
+    assert!(
+        observed_expiration,
+        "{scenario}: complete scans must observe successful lifecycle deletions"
+    );
+    assert!(
+        reconciled,
+        "{scenario}: usage overlay must converge to the complete scanner counts and bytes"
+    );
+    cycle
+}
+
+async fn check_persisted_stale_usage_recovery(store: &Arc<ECStore>, continuing_writes: bool, first_cycle: u64) -> u64 {
+    let cases = if continuing_writes {
+        [("historical-busy-empty", 0_u64), ("historical-busy-retained", 1_u64)]
+    } else {
+        [("historical-empty", 0_u64), ("historical-retained", 1_u64)]
+    };
+    let mut stale = DataUsageInfo {
+        last_update: Some(SystemTime::now()),
+        usage_snapshot_complete: true,
+        usage_snapshot_converged: Some(true),
+        buckets_count: 2,
+        scanner_epoch: Some(11),
+        scanner_cycle: Some(first_cycle - 1),
+        ..Default::default()
+    };
+    for (bucket, retained) in cases {
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create historical usage bucket");
+        if retained > 0 {
+            let mut reader = ScannerPutObjReader::from_vec(vec![0; 42]);
+            store
+                .put_object(bucket, "retained", &mut reader, &ScannerObjectOptions::default())
+                .await
+                .expect("create the only live object before seeding stale usage");
+        }
+        stale.buckets_usage.insert(
+            bucket.to_owned(),
+            BucketUsageInfo {
+                objects_count: 436 + retained,
+                size: 4_587_688_512 + retained * 42,
+                ..Default::default()
+            },
+        );
+        stale.bucket_sizes.insert(bucket.to_owned(), 4_587_688_512 + retained * 42);
+    }
+    stale.calculate_totals();
+    wait_for_namespace_commit_tails(store.as_ref()).await;
+    stale.last_update = Some(SystemTime::now());
+    seed_persisted_usage(stale, store.clone())
+        .await
+        .expect("persist an incorrect complete aggregate");
+    let loaded = load_data_usage_from_backend(store.clone())
+        .await
+        .expect("load the incorrect durable baseline");
+    assert!(loaded.is_complete_bucket_usage_snapshot());
+    for (bucket, retained) in cases {
+        assert_eq!(loaded.buckets_usage[bucket].objects_count, 436 + retained);
+        assert_eq!(loaded.buckets_usage[bucket].size, 4_587_688_512 + retained * 42);
+    }
+    // Start from the incorrect durable aggregate; only the production scanner
+    // may publish a corrected snapshot. Per-set cache corruption is separate.
+    replace_bucket_usage_memory_from_info(&loaded).await;
+    let cycles = if continuing_writes { 6 } else { 2 };
+    for cycle in first_cycle..first_cycle + cycles {
+        let added_objects = if continuing_writes { cycle - first_cycle + 1 } else { 0 };
+        if continuing_writes {
+            for (bucket, _) in cases {
+                let mut reader = ScannerPutObjReader::from_vec(vec![0; 42]);
+                store
+                    .put_object(bucket, &format!("keep/write-{cycle}"), &mut reader, &ScannerObjectOptions::default())
+                    .await
+                    .expect("write before each historical recovery scan");
+                record_bucket_object_write_memory(bucket, None, 42).await;
+            }
+            wait_for_namespace_commit_tails(store.as_ref()).await;
+        }
+        let ctx = CancellationToken::new();
+        let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
+        let (updates, receiver) = mpsc::channel(4);
+        let persist = tokio::spawn(crate::scanner::store_data_usage_in_backend(ctx.clone(), store.clone(), receiver));
+        let result = tokio::time::timeout(
+            Duration::from_secs(30),
+            ScannerIOCycle::nsscanner_with_status(store.as_ref(), ctx, budget, updates, cycle, 11, HealScanMode::Normal),
+        )
+        .await
+        .expect("historical recovery scan should finish")
+        .expect("historical recovery scan should succeed");
+        assert_eq!(result.status, ScannerCycleStatus::Complete);
+        tokio::time::timeout(Duration::from_secs(30), persist)
+            .await
+            .expect("scanner publication should finish")
+            .expect("scanner publication task should not panic");
+
+        let durable = load_data_usage_from_backend(store.clone())
+            .await
+            .expect("reload scanner-produced durable usage");
+        assert!(durable.is_complete_bucket_usage_snapshot());
+        assert_eq!(durable.usage_snapshot_converged, Some(true));
+        assert_eq!(durable.scanner_cycle, Some(cycle));
+        let mut displayed = load_admin_data_usage_from_backend_cached(store.clone())
+            .await
+            .expect("load usage through the admin snapshot path");
+        apply_bucket_usage_memory_overlay(&mut displayed).await;
+        for (bucket, retained) in cases {
+            assert_eq!(get_bucket_usage_memory(bucket).await, Some((retained + added_objects) * 42));
+            for snapshot in [&durable, &displayed] {
+                assert_eq!(snapshot.buckets_usage[bucket].objects_count, retained + added_objects);
+                assert_eq!(snapshot.buckets_usage[bucket].size, (retained + added_objects) * 42);
+            }
+        }
+    }
+    first_cycle + cycles
+}
@@ -346,6 +346,18 @@ pub(crate) mod scanner_io {
    pub(crate) use super::storage_contracts::{BucketInfo, BucketOptions};
    #[cfg(test)]
    pub(crate) use super::storage_contracts::{HTTPRangeSpec, ObjectIO};
+    #[cfg(test)]
+    pub(crate) use rustfs_ecstore::api::{
+        bucket::{
+            lifecycle::bucket_lifecycle_ops::init_background_expiry, metadata::BUCKET_LIFECYCLE_CONFIG,
+            metadata_sys::update as update_bucket_metadata,
+        },
+        data_usage::{
+            apply_bucket_usage_memory_overlay, get_bucket_usage_memory, load_admin_data_usage_from_backend_cached,
+            load_data_usage_from_backend, record_bucket_object_delete_memory, record_bucket_object_write_memory,
+            replace_bucket_usage_memory_from_info, store_data_usage_in_backend as seed_persisted_usage,
+        },
+    };
 }

 pub(crate) type ScannerBucketListing = EcstoreScannerBucketListing;
@@ -61,7 +61,7 @@ pub struct ListPartsInfo {
    pub upload_id: String,
    pub storage_class: String,
    pub part_number_marker: usize,
-    pub next_part_number_marker: usize,
+    pub next_part_number_marker: Option<usize>,
    pub max_parts: usize,
    pub is_truncated: bool,
    pub parts: Vec<PartInfo>,
@@ -8,6 +8,8 @@ collection:

 ## Operations

+- [Multipart upload memory diagnosis](operations/multipart-memory.md) — allocator attribution, Docker reproduction and sustained upload checks.
+
 For the logical per-operation io_uring read cap, see
 [io_uring read chunk size](operations/io-uring-read-chunks.md).

@@ -31,6 +33,9 @@ expansion, rebalance, decommission, heal, drive replacement, restart
 recovery, and the `rc` CLI mapping), start with
 [Cluster and erasure-coding lifecycle operations](operations/cluster-lifecycle-operations.md).

+For concurrent bucket creation and deletion, see
+[Bucket operation admission](operations/bucket-operation-admission.md).
+
 For persisted administrator bucket tasks and bucket recreation, see
 [Bucket heal recovery](operations/bucket-heal-recovery.md).

@@ -12,7 +12,7 @@
 | Class | Provider (`impl WorkloadAdmissionSnapshotProvider`) | `active` / `queued` / `limit` source | Reports `Unknown` when |
 |---|---|---|---|
 | `ForegroundRead` | `ConcurrencyManager` in `rustfs/src/storage/concurrency/manager.rs` (source of truth); re-exposed unchanged by the RustFS runtime provider | disk-read permits in use / `None` (the semaphore exposes no waiter count) / configured max concurrent disk reads | the storage registry has no entry |
-| `ForegroundWrite` | `ConcurrencyManager` in `rustfs/src/storage/concurrency/manager.rs` (source of truth); re-exposed unchanged by the RustFS runtime provider | allocated/reserved permit units under automatic sizing, request-count permits for explicit/strict limits, or the legacy active-write counter / multipart requests waiting in the bounded admission queue (`None` for the strict and legacy policies) / matching unit budget or request-count limit | the storage registry has no entry |
+| `ForegroundWrite` | `ConcurrencyManager` in `rustfs/src/storage/concurrency/manager.rs` (source of truth); re-exposed unchanged by the RustFS runtime provider | allocated/reserved permit units under automatic sizing, request-count permits for explicit/strict limits, or the legacy active-write counter / multipart `UploadPart` and `UploadPartCopy` requests waiting in the bounded admission queue (`None` for the strict and legacy policies) / matching unit budget or request-count limit | the storage registry has no entry |
 | `Metadata` | `RustFsWorkloadAdmissionSnapshotProvider` in `rustfs/src/workload_admission.rs` | `Open` once the bucket metadata runtime handle exists; no counts | bucket metadata runtime not initialized |
 | `Scanner` | same | scanner active work-unit counter / none / configured set-scan limit when nonzero | scanner runtime not initialized |
 | `Repair` | same | heal active tasks / heal queue length / `None` (limits live behind the async heal manager state) | heal manager not initialized |
@@ -0,0 +1,16 @@
+# Bucket creation and deletion admission
+
+`CreateBucket` and `DeleteBucket` share a process-wide execution budget of eight active transactions. Short bursts wait in a FIFO queue with space for 128 waiting requests. Each request can wait up to 30 seconds for an execution slot. These limits protect storage and namespace-lock resources while allowing parallel test workers to create isolated buckets.
+
+A request that finds the queue full or exceeds the queue wait returns HTTP 503 with the S3 error code `SlowDown` and `Retry-After: 1`. This is an overload response, not a bucket-name or filesystem error. The active and waiting limits are shared by creation and deletion and are currently fixed.
+
+Waiting requests do not start storage mutations. Disconnecting while waiting releases the queue slot. Once admitted, the transaction owns its execution slot until the mutation and post-commit hooks finish, including when its caller disconnects.
+
+For automated tests and applications:
+
+- Keep SDK retries enabled and use bounded exponential backoff with jitter for `SlowDown`. Honor `Retry-After` as a minimum delay when implementing your own retry policy.
+- Set the client request timeout above the queue wait plus the expected operation time. The 30-second limit covers admission waiting only; it does not limit transaction execution.
+- Reduce the client concurrency if bursts exceed the waiting capacity or repeatedly time out. A bucket per test worker remains a supported isolation pattern.
+- After an ambiguous network timeout, check the bucket state before assuming the operation failed: an admitted transaction may finish after the client stops waiting.
+
+The serving implementation is `BucketOperationAdmission` in `rustfs/src/app/bucket_usecase.rs`. This budget does not change object PUT/GET admission, erasure quorum, or persistence rules.
@@ -217,9 +217,18 @@ If a set has lost more drives than its parity, heal cannot rebuild; recover from
 - **Ordinary restart.** No topology change in flight, identical arguments on every node, one node at a time per [rolling-restart.md](rolling-restart.md). Startup reloads pool metadata, `rebalance.bin`, the decommission queue, and heal recovery records, and resumes workers by leader and owner rules.
 - **Rebalance.** Read `/rebalance/status` first. Handle active or stopping pools, `lastError`, cleanup warnings, and stop propagation before deciding to stop or restart. Do not delete `rebalance.bin`.
 - **Decommission.** Non-terminal entries resume; failed and canceled entries survive restarts until cleared or retried through the allowed path (§9).
+- **Scanner pause backlog.** An unreadable or unprovable `.scanner-pause-backlog.json` gates scanner cycles, including a pending usage full rebuild. Follow [the diagnostic and recovery procedure](#111-scanner-pause-backlog-recovery) below; keep the original records intact.
 - **Root heal.** A graceful shutdown persists an unfinished cluster heal as `root-heal-<task-id>.json` in `.rustfs.sys` (`RootHealRecovery` in `crates/heal/src/heal/manager/root_recovery.rs`, 64 KiB bound) and replays it with the same task id. Terminal records use `terminal-root-heal-`; invalid, oversized, or unsupported records move to `quarantined-root-heal-intent-` or `quarantined-root-heal-terminal-` and are skipped with a log line until an operator repairs them. Do not delete quarantine markers or reuse a task id.
 - **Unclean shutdown.** When the previous run left its marker behind, the heal manager's `process_unclean_shutdown` (`crates/heal/src/heal/manager/unclean_shutdown.rs`) enqueues a full erasure-set heal for every local set; graceful shutdown clears the marker. See [durability-modes.md](durability-modes.md) for which sync tier can leave what behind.

+### 11.1 Scanner pause backlog recovery
+
+1. Record the running version and build commit on every node. Confirm that every potential scanner leader includes serialized replica publication and cancellation-safe publication ownership; a version label or successful rolling restart alone does not prove this. An older writer can still use the earlier concurrent publication path during a mixed-version rollout.
+2. Query `/rustfs/admin/v3/scanner/status` on each node, identify the leader, and retain complete, timestamped snapshots before and after its next persistence retry. For an unavailable or degraded backlog, `pause_backlog.replica_diagnostics` lists every currently writable `(pool_index, set_index)`, its `missing`, `valid`, `invalid`, `future_schema`, or `unavailable` state, its object revision (CAS token), stable and committed generations/epochs, and read/decode errors. `committed_replicas` names the membership claimed by that record, while `committed_replica_count` gives its size. Compare this list with the actual configured topology on every node; a reported set count differing from `replica_count` needs investigation before recovery.
+3. Restore access to unavailable members and resolve object-lock or storage failures without editing the ledger. Publication is serialized to avoid competition within one replica cohort; external contention can still make a write fail. The leader retries failed claims on the five-minute persistence schedule; read/write duration can consume that delay, so a slow attempt does not guarantee another five-minute wait after it returns.
+4. A surviving, complete membership commit or identical stable rollback values across all current replicas permit automatic recovery. The writer stabilizes that proven ledger using each member's current CAS revision, verifies all members by readback, and advances its writer epoch/generation. `rolled_back_partial_commit` can therefore recover without an operator reset. Its `healthy_replicas: 0` means no record currently matches the selected stable/commit combination; it does not mean every object is missing. Status reads can also straddle publication phases, so a single transient snapshot is not a completed recovery proof. Confirm repeated `healthy` status with no persistence error, a successfully advanced writer epoch/generation, and then scanner cycle progress; pending full-scan work must remain until it completes.
+5. If the writer cannot prove either a complete membership commit or a common stable rollback, preserve the original objects and revisions and escalate with the snapshots, build commits, topology, and lock/storage errors. Invalid or newer-schema records require a compatible reader or supported repair; conflicting records must not be resolved by choosing the largest generation. There is no supported force-reset for lost authority. Do not delete, copy over, or manually edit `.scanner-pause-backlog.json`; doing so may discard pending data-movement work. `usage-state/reset` requests a usage rebuild and does not repair or bypass this ledger gate.
+
 ## 12. Mutual exclusion and prohibited actions

 | Operation | With rebalance | With decommission | Notes |
@@ -0,0 +1,46 @@
+# Multipart upload memory diagnosis
+
+**Use this when:** sustained multipart uploads increase process memory or end in an OOM kill, and you need to distinguish request buffers, background work, and allocator retention.
+
+## Record the actual workload
+
+Record the image digest, source revision, architecture, allocator version, I/O backend, data filesystem and mounts, container memory/swap limits, and the client version. Docker directories sharing one device are not equivalent to separate physical disks. Keep any local-test disk-check bypass out of production configuration.
+
+Count active UploadPart requests independently from file/upload concurrency. Record part size, parts per upload, retries, completed uploads, aborted uploads, TLS, throughput, and request latency. A distributed client's concurrency applies per client. Warp multipart-put also performs create and complete operations; its PUTPART summary alone does not count successful completions.
+
+Run a short baseline followed by a sustained isolated run long enough to cross the reported growth window. Keep a defined idle observation window after uploads, completion, abort and cleanup. Bound disk use as well as memory: a successful long upload test can fill the test filesystem.
+
+## Separate memory measurements
+
+Collect these measurements on the same time axis:
+
+- Process RssAnon/Pss_Anon, file RSS, swap and total RSS from Linux procfs.
+- Cgroup memory.current, memory.stat anon/file and memory.events. Docker stats is not a substitute for this split.
+- Allocator reserved and committed bytes, and requested bytes only when their live statistics are available.
+- Active requests, EC producer/queue/writer bytes, GET buffered bytes, tasks, threads, file descriptors, and scanner/heal/replication activity.
+
+The EC queue budget controls queued encoded blocks, not complete per-request or process memory. Admission permits constrain concurrent operations; they do not establish an RSS limit or release allocations retained by completed work.
+
+### Allocator statistics
+
+Mimalloc JSON count statistics have direct current, peak and total fields. Scalar counters contain only cumulative totals. In particular, a scalar malloc_requested must never be interpreted as current live bytes or a peak. The malloc_normal/huge statistics are not a reliable replacement for requested live bytes in release builds.
+
+Use rustfs_memory_allocator_malloc_requested_bytes only when rustfs_memory_allocator_malloc_requested_bytes_available equals 1. A value of 0 for availability means the live statistic could not be established; it does not mean the heap is empty. An all-zero requested count may mean accounting was compiled out. A supported count whose current becomes zero while peak/total remain nonzero is a valid drained heap observation.
+
+A previously published gauge can remain visible after a statistic becomes unavailable. Filter it by the availability metric. The malloc_requested_total_bytes metric is cumulative allocation activity and can grow indefinitely in a healthy process. Do not use its slope as evidence of retained memory.
+
+When live requested bytes are unavailable, capture a supported heap profile or allocation/free trace. Preserve attribution to allocation stacks and lifetime; committed bytes and RSS alone cannot establish a leak. Other raw allocator statistics, including page accounting, depend on the allocator build and are not interchangeable with live application bytes.
+
+### Idle reclaim
+
+The reclaim loop waits for request, delete-tail, scanner, heal, EC and GET-buffer activity to become idle. Continuous traffic can skip reclaim. Check the activity gauges, skipped reasons and idle streak rather than assuming that waiting a fixed period guarantees collection.
+
+A successful MiMalloc collect call collects the executing thread's heap. The reclaim ok counter does not prove that all Tokio worker or blocking-thread heaps have been collected. It cannot by itself rule out allocator retention. Avoid adding forced collection to each request without a measured cause and performance evidence.
+
+## Isolate the retaining path
+
+Compare the same upload ID and part number overwritten repeatedly with new upload IDs and parts. Test UploadPart without completion, complete cycles, abort, disconnect, timeout and slow storage separately. Compare TLS and plaintext only at matched workload and throughput. Correlate background activity rather than disabling durability or checksums.
+
+A live-heap leak requires evidence that allocations stay reachable or unfreed after their owner should finish. Stable live allocations with growing committed/RSS point toward allocator retention or fragmentation. Growth tracking metadata cardinality or background queues needs a bounded-state investigation. Accept stable caches and explain their bounds; do not require every byte of RSS to disappear after idle.
+
+Verify any fix with the same workload, failure paths and idle window. Preserve S3 part/checksum semantics, cancellation cleanup, commit fencing, write quorum and durability. Raising memory limits or reducing concurrency can be a diagnostic control, but does not prove a leak is fixed.
@@ -118,7 +118,7 @@ Post-merge and tag-driven; not a substitute for a PR gate.
 |---|---|---|
 | Push to `main`, weekly schedule, dispatch | `build.yml` `build-rustfs` (a development build on a main push restricts the matrix to the Linux targets) | build artifacts; no release publication |
 | Valid release or preview tag | `build.yml` `build-rustfs`, `create-release`, `upload-release-assets`, `publish-release` | draft release, checksummed assets, publish |
-| Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `build-docker`, `scan-docker-image`, `sync-dockerhub-description` | multi-architecture images, vulnerability report, and the Docker Hub overview republished from `README.md` |
+| Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `build-docker`, `scan-docker-image` | multi-architecture images and vulnerability report |
 | Successful release-tag build (`workflow_run`) | `package.yml` `package` | DEB/RPM packages and checksums uploaded to the release |
 | Successful non-preview release-tag build (`workflow_run`) | `helm-package.yml` `build-helm-package`, `publish-helm-package` | versioned chart and repository index |
 | Final tag's release published | `build.yml` `cleanup-preview-releases` | deletes every `<target>-preview.<N>` Release for that target; the tags are kept |
@@ -144,17 +144,6 @@ dependencies = [
 "pin-project-lite",
 ]

-[[package]]
-name = "async-recursion"
-version = "1.1.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 2.0.118",
-]
-
 [[package]]
 name = "async-trait"
 version = "0.1.91"
@@ -4135,7 +4124,6 @@ dependencies = [
 "aes-gcm",
 "arc-swap",
 "async-channel",
- "async-recursion",
 "async-trait",
 "aws-config",
 "aws-credential-types",
@@ -131,15 +131,67 @@ use std::{
    future::Future,
    io::Write,
    sync::{Arc, LazyLock},
+    time::Duration,
 };
-use tokio::sync::{Semaphore, TryAcquireError};
+use tokio::sync::{OwnedSemaphorePermit, Semaphore, TryAcquireError};
 use tracing::{Instrument as _, debug, error, info, instrument, warn};

 const LOG_COMPONENT_APP: &str = "app";
 const LOG_SUBSYSTEM_BUCKET: &str = "bucket";
 const BUCKET_OPERATION_CONCURRENCY: usize = 8;
-static BUCKET_OPERATION_ADMISSION: LazyLock<Arc<Semaphore>> =
-    LazyLock::new(|| Arc::new(Semaphore::new(BUCKET_OPERATION_CONCURRENCY)));
+const BUCKET_OPERATION_QUEUE_CAPACITY: usize = 128;
+const BUCKET_OPERATION_QUEUE_TIMEOUT: Duration = Duration::from_secs(30);
+static BUCKET_OPERATION_ADMISSION: LazyLock<Arc<BucketOperationAdmission>> = LazyLock::new(|| {
+    Arc::new(BucketOperationAdmission::new(
+        BUCKET_OPERATION_CONCURRENCY,
+        BUCKET_OPERATION_QUEUE_CAPACITY,
+    ))
+});
+
+struct BucketOperationAdmission {
+    active: Arc<Semaphore>,
+    waiting: Semaphore,
+}
+
+impl BucketOperationAdmission {
+    fn new(concurrency: usize, queue_capacity: usize) -> Self {
+        Self {
+            active: Arc::new(Semaphore::new(concurrency)),
+            waiting: Semaphore::new(queue_capacity),
+        }
+    }
+
+    async fn acquire(&self, operation: &'static str) -> S3Result<OwnedSemaphorePermit> {
+        match self.active.clone().try_acquire_owned() {
+            Ok(permit) => return Ok(permit),
+            Err(TryAcquireError::Closed) => return Err(bucket_admission_closed(operation)),
+            Err(TryAcquireError::NoPermits) => {}
+        }
+
+        // Wait on the request task; only admitted mutations become detached tasks.
+        // The queue permit bounds waiting requests and is released on cancellation.
+        let _waiting = self.waiting.try_acquire().map_err(|err| match err {
+            TryAcquireError::NoPermits => bucket_admission_slow_down(operation, "queue is full"),
+            TryAcquireError::Closed => bucket_admission_closed(operation),
+        })?;
+        tokio::time::timeout(BUCKET_OPERATION_QUEUE_TIMEOUT, self.active.clone().acquire_owned())
+            .await
+            .map_err(|_| bucket_admission_slow_down(operation, "queue wait timed out"))?
+            .map_err(|_| bucket_admission_closed(operation))
+    }
+}
+
+fn bucket_admission_closed(operation: &'static str) -> S3Error {
+    S3Error::with_message(S3ErrorCode::InternalError, format!("{operation} admission closed"))
+}
+
+fn bucket_admission_slow_down(operation: &'static str, reason: &'static str) -> S3Error {
+    let mut error = S3Error::with_message(S3ErrorCode::SlowDown, format!("{operation} {reason}; retry later"));
+    let mut headers = http::HeaderMap::new();
+    headers.insert(http::header::RETRY_AFTER, http::HeaderValue::from_static("1"));
+    error.set_headers(headers);
+    error
+}
 use urlencoding::encode;

 type ListObjectVersionsInfo = StorageListObjectVersionsInfo<ObjectInfo>;
@@ -1307,19 +1359,14 @@ where

 async fn await_bucket_usecase_on_fresh_task_with_admission<T, F>(
    operation: &'static str,
-    admission: Arc<Semaphore>,
+    admission: Arc<BucketOperationAdmission>,
    future: F,
 ) -> S3Result<T>
 where
    T: Send + 'static,
    F: Future<Output = S3Result<T>> + Send + 'static,
 {
-    let permit = admission.try_acquire_owned().map_err(|err| match err {
-        TryAcquireError::NoPermits => {
-            S3Error::with_message(S3ErrorCode::SlowDown, format!("{operation} concurrency limit reached; retry later"))
-        }
-        TryAcquireError::Closed => S3Error::with_message(S3ErrorCode::InternalError, format!("{operation} admission closed")),
-    })?;
+    let permit = admission.acquire(operation).await?;
    tokio::spawn(
        async move {
            let _permit = permit;
@@ -3239,7 +3286,7 @@ mod tests {

    #[tokio::test]
    async fn bucket_usecase_task_finishes_post_commit_hooks_after_parent_cancellation() {
-        let admission = Arc::new(Semaphore::new(1));
+        let admission = Arc::new(BucketOperationAdmission::new(1, 1));
        let committed = Arc::new(Notify::new());
        let committed_wait = committed.notified();
        let release_hook = Arc::new(Notify::new());
@@ -3274,45 +3321,39 @@ mod tests {
            hook_ran.load(Ordering::SeqCst),
            "the fresh task must own both the storage mutation and its post-commit hooks"
        );
-        assert_eq!(admission.available_permits(), 1);
+        assert_eq!(admission.active.available_permits(), 1);
    }

-    #[tokio::test]
-    async fn saturated_bucket_usecase_admission_does_not_start_work() {
-        let admission = Arc::new(Semaphore::new(1));
-        let held = admission
-            .clone()
-            .acquire_owned()
-            .await
-            .expect("test admission should remain open");
+    #[tokio::test(start_paused = true)]
+    async fn bucket_usecase_admission_times_out_without_starting_work() {
+        let admission = Arc::new(BucketOperationAdmission::new(1, 1));
+        let held = admission.active.clone().acquire_owned().await.expect("hold active slot");
        let started = Arc::new(AtomicBool::new(false));
        let started_for_task = started.clone();
-        let result = tokio::time::timeout(
-            Duration::from_secs(1),
-            await_bucket_usecase_on_fresh_task_with_admission("test bucket operation", admission, async move {
-                started_for_task.store(true, Ordering::SeqCst);
-                Ok(())
-            }),
-        )
-        .await
-        .expect("saturated admission must fail within the bounded timeout");
+        let result = await_bucket_usecase_on_fresh_task_with_admission("test bucket operation", admission.clone(), async move {
+            started_for_task.store(true, Ordering::SeqCst);
+            Ok(())
+        })
+        .await;
+        let error = result.expect_err("queued request must time out");
+        assert_eq!(error.code(), &S3ErrorCode::SlowDown);
+        assert_eq!(error.headers().expect("retry headers")[http::header::RETRY_AFTER], "1");
+        assert!(!started.load(Ordering::SeqCst), "timed-out work must not start");
+        assert_eq!(admission.waiting.available_permits(), 1);
        drop(held);
-
-        assert!(
-            !started.load(Ordering::SeqCst),
-            "saturated admission must not start a detached bucket operation"
-        );
-        assert_eq!(result.expect_err("saturated admission must fail fast").code(), &S3ErrorCode::SlowDown);
+        assert_eq!(admission.active.available_permits(), 1);
    }

    #[tokio::test]
-    async fn bucket_usecase_admission_rejects_excess_detached_tasks() {
-        let admission = Arc::new(Semaphore::new(BUCKET_OPERATION_CONCURRENCY));
+    async fn bucket_usecase_admission_queues_bursts_with_bounded_execution() {
+        let admission = Arc::new(BucketOperationAdmission::new(
+            BUCKET_OPERATION_CONCURRENCY,
+            BUCKET_OPERATION_QUEUE_CAPACITY,
+        ));
        let release = Arc::new(Semaphore::new(0));
        let started = Arc::new(AtomicUsize::new(0));
-        let mut tasks = Vec::with_capacity(BUCKET_OPERATION_CONCURRENCY);
-
-        for _ in 0..BUCKET_OPERATION_CONCURRENCY {
+        let mut tasks = Vec::with_capacity(100);
+        for _ in 0..100 {
            let admission_for_task = admission.clone();
            let release_for_task = release.clone();
            let started_for_task = started.clone();
@@ -3321,53 +3362,82 @@ mod tests {
                admission_for_task,
                async move {
                    started_for_task.fetch_add(1, Ordering::SeqCst);
-                    let _release = release_for_task
-                        .acquire()
-                        .await
-                        .expect("test release gate should remain open");
+                    release_for_task.acquire().await.expect("release gate").forget();
                    Ok(())
                },
            )));
        }
-
-        tokio::time::timeout(Duration::from_secs(1), async {
-            while started.load(Ordering::SeqCst) != BUCKET_OPERATION_CONCURRENCY {
+        tokio::time::timeout(Duration::from_secs(5), async {
+            while admission.waiting.available_permits() != BUCKET_OPERATION_QUEUE_CAPACITY - 92
+                || started.load(Ordering::SeqCst) != BUCKET_OPERATION_CONCURRENCY
+            {
                tokio::task::yield_now().await;
            }
        })
        .await
-        .expect("all admitted operations should start");
-
-        let ninth_started = Arc::new(AtomicBool::new(false));
-        let ninth_started_for_task = ninth_started.clone();
-        let ninth_result = tokio::time::timeout(
-            Duration::from_secs(1),
-            await_bucket_usecase_on_fresh_task_with_admission("test bucket operation", admission.clone(), async move {
-                ninth_started_for_task.store(true, Ordering::SeqCst);
-                Ok(())
-            }),
-        )
-        .await
-        .expect("an excess bucket transaction must fail within the bounded timeout");
-        assert!(
-            !ninth_started.load(Ordering::SeqCst),
-            "the ninth bucket transaction must not start when admission is saturated"
-        );
-        assert_eq!(
-            ninth_result.expect_err("the ninth bucket transaction must fail fast").code(),
-            &S3ErrorCode::SlowDown
-        );
-
-        release.add_permits(BUCKET_OPERATION_CONCURRENCY);
+        .expect("all excess operations should queue");
+        assert_eq!(started.load(Ordering::SeqCst), BUCKET_OPERATION_CONCURRENCY);
+        release.add_permits(100);
        for task in tasks {
-            task.await
-                .expect("bucket transaction parent should join")
-                .expect("bucket transaction should succeed");
+            task.await.expect("request task joins").expect("burst operation succeeds");
        }
+        assert_eq!(started.load(Ordering::SeqCst), 100);
+        assert_eq!(admission.active.available_permits(), BUCKET_OPERATION_CONCURRENCY);
+        assert_eq!(admission.waiting.available_permits(), BUCKET_OPERATION_QUEUE_CAPACITY);
+    }

+    #[tokio::test]
+    async fn bucket_usecase_admission_bounds_queue_and_cancels_waiters() {
+        let admission = Arc::new(BucketOperationAdmission::new(1, 1));
+        let held = admission.active.clone().acquire_owned().await.expect("hold active slot");
+        let started = Arc::new(AtomicBool::new(false));
+        let started_for_task = started.clone();
+        let queued = tokio::spawn(await_bucket_usecase_on_fresh_task_with_admission(
+            "test bucket operation",
+            admission.clone(),
+            async move {
+                started_for_task.store(true, Ordering::SeqCst);
+                Ok(())
+            },
+        ));
+        tokio::time::timeout(Duration::from_secs(5), async {
+            while admission.waiting.available_permits() != 0 {
+                tokio::task::yield_now().await;
+            }
+        })
+        .await
+        .expect("request should queue");
+        let error = admission
+            .acquire("test bucket operation")
+            .await
+            .expect_err("full queue rejects immediately");
+        assert_eq!(error.code(), &S3ErrorCode::SlowDown);
+        assert_eq!(error.headers().expect("retry headers")[http::header::RETRY_AFTER], "1");
+        let response = error.to_http_response().expect("serialize admission error");
+        assert_eq!(response.status(), http::StatusCode::SERVICE_UNAVAILABLE);
+        assert_eq!(response.headers()[http::header::RETRY_AFTER], "1");
+        use http_body_util::BodyExt as _;
+        let body = response.into_body().collect().await.expect("read error XML").to_bytes();
+        assert!(String::from_utf8_lossy(&body).contains("<Code>SlowDown</Code>"));
+        queued.abort();
+        assert!(queued.await.expect_err("queued request was cancelled").is_cancelled());
+        assert_eq!(admission.waiting.available_permits(), 1);
+        drop(held);
+        assert!(!started.load(Ordering::SeqCst), "cancelled waiter must not mutate storage");
        await_bucket_usecase_on_fresh_task_with_admission("test bucket operation", admission, async { Ok(()) })
            .await
-            .expect("admission must recover after active transactions finish");
+            .expect("admission recovers after cancellation");
+    }
+
+    #[tokio::test]
+    async fn bucket_usecase_admission_maps_closed_semaphore_to_internal_error() {
+        let admission = BucketOperationAdmission::new(1, 1);
+        admission.active.close();
+        let error = admission
+            .acquire("test bucket operation")
+            .await
+            .expect_err("closed admission must fail");
+        assert_eq!(error.code(), &S3ErrorCode::InternalError);
    }

    #[tokio::test]
@@ -1685,6 +1685,28 @@ impl DefaultMultipartUsecase {
            return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
        };

+        // Reserve capacity before holding lifecycle locks or opening source streams.
+        // The source metadata has not yet established the authoritative copy length.
+        let _copy_admission = match self
+            .concurrency_manager()
+            .admit_multipart_copy()
+            .await
+            .map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "foreground write admission closed"))?
+        {
+            ForegroundWriteAdmission::Disabled => None,
+            ForegroundWriteAdmission::Admitted(permit) => {
+                counter!("rustfs.upload_part_copy.foreground_admission.total", "result" => "admitted").increment(1);
+                Some(permit)
+            }
+            ForegroundWriteAdmission::Rejected => {
+                counter!("rustfs.upload_part_copy.foreground_admission.total", "result" => "rejected").increment(1);
+                return Err(S3Error::with_message(
+                    S3ErrorCode::SlowDown,
+                    "foreground write concurrency limit reached, please reduce your request rate",
+                ));
+            }
+        };
+
        let (source_bucket_lifecycle_guard, destination_bucket_lifecycle_guard_storage) =
            acquire_copy_bucket_lifecycle_locks(store.as_ref(), &src_bucket, &bucket).await?;
        let current_source_incarnation_id = store
@@ -3377,6 +3399,189 @@ mod tests {
        }
    }

+    fn copy_admission_request(bucket: &str, upload_id: &str, range: Option<&str>) -> S3Request<UploadPartCopyInput> {
+        build_request(
+            UploadPartCopyInput::builder()
+                .bucket(bucket.to_string())
+                .key("destination".to_string())
+                .copy_source(CopySource::Bucket {
+                    bucket: bucket.into(),
+                    key: "source".into(),
+                    version_id: None,
+                })
+                .copy_source_range(range.map(str::to_owned))
+                .part_number(1)
+                .upload_id(upload_id.to_string())
+                .build()
+                .expect("copy input should build"),
+            Method::PUT,
+        )
+    }
+
+    async fn copy_admission_fixture(manager: Arc<ConcurrencyManager>) -> (DefaultMultipartUsecase, String, String) {
+        use crate::app::storage_api::test::contract::bucket::{BucketOperations, MakeBucketOptions};
+        let store = crate::app::gating_test_env::shared_gating_ecstore().await;
+        let ambient = crate::app::gating_test_env::shared_gating_ambient().await;
+        let context = Arc::new(AppContext::new(Arc::clone(&store), ambient.iam(), ambient.kms()));
+        let bucket = format!("copy-admission-{}", Uuid::new_v4().simple());
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create copy test bucket");
+        let upload = store
+            .new_multipart_upload(&bucket, "destination", &ObjectOptions::default())
+            .await
+            .expect("create copy destination session");
+        (
+            DefaultMultipartUsecase::with_context_and_concurrency_manager(Some(context), manager),
+            bucket,
+            upload.upload_id,
+        )
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn execute_upload_part_copy_shares_admission_before_source_lookup() {
+        let manager = Arc::new(ConcurrencyManager::with_large_put_admission_for_test(
+            true,
+            1,
+            rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
+            Duration::ZERO,
+        ));
+        let (usecase, bucket, upload_id) = copy_admission_fixture(Arc::clone(&manager)).await;
+        let held = manager
+            .admit_multipart_part(1024)
+            .await
+            .expect("hold the shared upload permit");
+        for range in [None, Some("bytes=0-33554431")] {
+            let err = Box::pin(usecase.execute_upload_part_copy(copy_admission_request(&bucket, &upload_id, range)))
+                .await
+                .expect_err("saturated gate must reject before the missing source is accessed");
+            assert_eq!(err.code(), &S3ErrorCode::SlowDown);
+            assert_eq!(manager.put_object_admission_snapshot().active, Some(1));
+        }
+        drop(held);
+        let err = Box::pin(usecase.execute_upload_part_copy(copy_admission_request(&bucket, &upload_id, None)))
+            .await
+            .expect_err("released capacity must reach the missing-source check");
+        assert_eq!(err.code(), &S3ErrorCode::NoSuchKey);
+        assert_eq!(manager.put_object_admission_snapshot().active, Some(0));
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn execute_upload_part_copy_rejects_before_waiting_for_bucket_lock() {
+        use crate::app::storage_api::test::contract::namespace::NamespaceLocking;
+        let manager = Arc::new(ConcurrencyManager::with_large_put_admission_for_test(
+            true,
+            1,
+            rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
+            Duration::ZERO,
+        ));
+        let (usecase, bucket, upload_id) = copy_admission_fixture(Arc::clone(&manager)).await;
+        let store = crate::app::gating_test_env::shared_gating_ecstore().await;
+        let lock = store
+            .new_ns_lock(&bucket, crate::app::storage_api::object_usecase::BUCKET_LIFECYCLE_LOCK_OBJECT)
+            .await
+            .expect("create bucket lifecycle lock");
+        let blocker = lock
+            .get_write_lock(Duration::from_secs(5))
+            .await
+            .expect("hold lifecycle write lock");
+        let held = manager.admit_multipart_part(1024).await.expect("fill foreground capacity");
+        let err = tokio::time::timeout(
+            Duration::from_secs(5),
+            Box::pin(usecase.execute_upload_part_copy(copy_admission_request(&bucket, &upload_id, None))),
+        )
+        .await
+        .expect("capacity rejection must not wait on the bucket lock")
+        .expect_err("copy must reject at capacity");
+        assert_eq!(err.code(), &S3ErrorCode::SlowDown);
+        drop(held);
+        drop(blocker);
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn execute_upload_part_copy_wait_cancellation_releases_queue_slot() {
+        let manager = Arc::new(ConcurrencyManager::with_multipart_admission_queue_for_test(1, Duration::from_secs(60), 1));
+        let (usecase, bucket, upload_id) = copy_admission_fixture(Arc::clone(&manager)).await;
+        let held = manager
+            .admit_multipart_part(1024)
+            .await
+            .expect("hold the only foreground permit");
+        let request = copy_admission_request(&bucket, &upload_id, None);
+        let waiter = tokio::spawn(async move { Box::pin(usecase.execute_upload_part_copy(request)).await });
+        tokio::time::timeout(Duration::from_secs(5), async {
+            while manager.put_object_admission_snapshot().queued != Some(1) {
+                tokio::task::yield_now().await;
+            }
+        })
+        .await
+        .expect("copy must wait in the bounded queue before accessing its source");
+        waiter.abort();
+        assert!(waiter.await.expect_err("waiting copy must be cancelled").is_cancelled());
+        assert_eq!(manager.put_object_admission_snapshot().queued, Some(0));
+        assert_eq!(manager.put_object_admission_snapshot().active, Some(1));
+        drop(held);
+        assert_eq!(manager.put_object_admission_snapshot().active, Some(0));
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn execute_upload_part_copy_cancellation_after_admission_releases_permit() {
+        use crate::app::storage_api::test::contract::namespace::NamespaceLocking;
+        let manager = Arc::new(ConcurrencyManager::with_large_put_admission_for_test(
+            true,
+            1,
+            rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
+            Duration::ZERO,
+        ));
+        let (usecase, bucket, upload_id) = copy_admission_fixture(Arc::clone(&manager)).await;
+        let store = crate::app::gating_test_env::shared_gating_ecstore().await;
+        let lock = store
+            .new_ns_lock(&bucket, crate::app::storage_api::object_usecase::BUCKET_LIFECYCLE_LOCK_OBJECT)
+            .await
+            .expect("create lifecycle lock");
+        let blocker = lock
+            .get_write_lock(Duration::from_secs(5))
+            .await
+            .expect("hold lifecycle lock");
+        let request = copy_admission_request(&bucket, &upload_id, None);
+        let task = tokio::spawn(async move { Box::pin(usecase.execute_upload_part_copy(request)).await });
+        tokio::time::timeout(Duration::from_secs(5), async {
+            while manager.put_object_admission_snapshot().active != Some(1) {
+                tokio::task::yield_now().await;
+            }
+        })
+        .await
+        .expect("copy must reserve capacity before waiting on its lifecycle lock");
+        assert!(matches!(
+            manager.admit_multipart_part(1024).await.expect("gate should remain open"),
+            ForegroundWriteAdmission::Rejected
+        ));
+        task.abort();
+        assert!(task.await.expect_err("blocked copy must be cancelled").is_cancelled());
+        assert_eq!(manager.put_object_admission_snapshot().active, Some(0));
+        drop(blocker);
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn execute_upload_part_copy_disabled_admission_keeps_source_error() {
+        let manager = Arc::new(ConcurrencyManager::with_large_put_admission_for_test(
+            false,
+            1,
+            rustfs_config::DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES,
+            Duration::ZERO,
+        ));
+        let (usecase, bucket, upload_id) = copy_admission_fixture(manager).await;
+        let err = Box::pin(usecase.execute_upload_part_copy(copy_admission_request(&bucket, &upload_id, None)))
+            .await
+            .expect_err("disabled admission must retain the source error");
+        assert_eq!(err.code(), &S3ErrorCode::NoSuchKey);
+    }
+
    #[test]
    fn test_validate_copy_source_range_not_exceeds_returns_invalid_range_when_range_exceeds() {
        use super::validate_copy_source_range_not_exceeds;
@@ -21,6 +21,10 @@ use rustfs_object_data_cache::{
 use std::{sync::Arc, time::Duration};
 use tracing::warn;

+const IGNORED_OBJECT_CACHE_ENABLE_ENV: &str = "RUSTFS_OBJECT_CACHE_ENABLE";
+const IGNORED_OBJECT_CACHE_TTL_SECS_ENV: &str = "RUSTFS_OBJECT_CACHE_TTL_SECS";
+const IGNORED_OBJECT_CACHE_ENV_KEYS: [&str; 2] = [IGNORED_OBJECT_CACHE_ENABLE_ENV, IGNORED_OBJECT_CACHE_TTL_SECS_ENV];
+
 #[derive(Debug, Default)]
 struct ObjectDataCacheEnvValues {
    enabled: Option<bool>,
@@ -64,6 +68,7 @@ impl ObjectDataCacheAdapter {
    /// or an unparseable numeric override) disables the whole cache rather than
    /// silently starting with defaults the operator never chose.
    pub(crate) fn from_env_or_disabled() -> Arc<Self> {
+        warn_ignored_object_cache_env_keys();
        match object_data_cache_config_from_env() {
            Ok(config) => Self::from_config_or_disabled(config, "env"),
            Err(invalid_keys) => {
@@ -282,6 +287,26 @@ where
    }
 }

+fn ignored_object_cache_env_keys() -> Vec<&'static str> {
+    IGNORED_OBJECT_CACHE_ENV_KEYS
+        .iter()
+        .copied()
+        .filter(|key| std::env::var_os(key).is_some())
+        .collect()
+}
+
+fn warn_ignored_object_cache_env_keys() {
+    let ignored_keys = ignored_object_cache_env_keys();
+    if ignored_keys.is_empty() {
+        return;
+    }
+    warn!(
+        ignored_env_keys = %ignored_keys.join(","),
+        replacement_prefix = "RUSTFS_OBJECT_DATA_CACHE_",
+        "ignored unrecognized object cache environment variables"
+    );
+}
+
 fn object_data_cache_config_from_env() -> Result<ObjectDataCacheConfig, Vec<&'static str>> {
    let mut invalid_keys: Vec<&'static str> = Vec::new();

@@ -386,8 +411,9 @@ fn parse_object_data_cache_mode(value: &str) -> Option<ObjectDataCacheMode> {
 #[cfg(test)]
 mod tests {
    use super::{
-        ObjectDataCacheAdapter, ObjectDataCacheEnvValues, object_data_cache_config_from_env,
-        object_data_cache_config_from_values, parse_object_data_cache_mode,
+        IGNORED_OBJECT_CACHE_ENABLE_ENV, IGNORED_OBJECT_CACHE_TTL_SECS_ENV, ObjectDataCacheAdapter, ObjectDataCacheEnvValues,
+        ignored_object_cache_env_keys, object_data_cache_config_from_env, object_data_cache_config_from_values,
+        parse_object_data_cache_mode,
    };
    use rustfs_object_data_cache::{ObjectDataCacheConfig, ObjectDataCacheMode};
    use std::time::Duration;
@@ -408,6 +434,8 @@ mod tests {
            (rustfs_config::ENV_OBJECT_DATA_CACHE_FILL_CONCURRENCY_PER_CPU, None),
            (rustfs_config::ENV_OBJECT_DATA_CACHE_FILL_CONCURRENCY_MAX, None),
            (rustfs_config::ENV_OBJECT_DATA_CACHE_IDENTITY_KEYS_MAX, None),
+            (IGNORED_OBJECT_CACHE_ENABLE_ENV, None),
+            (IGNORED_OBJECT_CACHE_TTL_SECS_ENV, None),
        ]
    }

@@ -665,6 +693,21 @@ mod tests {
        });
    }

+    #[test]
+    fn unrecognized_object_cache_env_vars_are_reported_but_do_not_enable_data_cache() {
+        let vars = with_env_overrides(&[
+            (IGNORED_OBJECT_CACHE_ENABLE_ENV, "true"),
+            (IGNORED_OBJECT_CACHE_TTL_SECS_ENV, "300"),
+        ]);
+        temp_env::with_vars(vars, || {
+            let ignored_keys = ignored_object_cache_env_keys();
+            let config = object_data_cache_config_from_env().expect("ignored cache env should not invalidate config");
+
+            assert_eq!(ignored_keys, vec![IGNORED_OBJECT_CACHE_ENABLE_ENV, IGNORED_OBJECT_CACHE_TTL_SECS_ENV]);
+            assert_eq!(config, ObjectDataCacheConfig::default());
+        });
+    }
+
    #[test]
    fn object_data_cache_env_valid_numeric_value_is_applied() {
        let vars = with_env_overrides(&[
@@ -1679,6 +1679,7 @@ mod tests {
    }

    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+    #[serial_test::serial]
    async fn executes_thread_profile_jobs_against_the_service_process() {
        for scope in [ThreadProfileScope::NativeThreads, ThreadProfileScope::TokioRuntime] {
            let now = Utc::now();
@@ -44,6 +44,9 @@ const MAX_STATE_BYTES: u64 = 4 * 1024 * 1024;
 const MAX_ARTIFACT_BYTES: usize = 524_288;
 const MAX_STATE_FILES: usize = 65_536;

+// The executable identity is stable for this process; avoid hashing the full binary for every profile request.
+static EXECUTABLE_PROVENANCE: tokio::sync::OnceCell<ProfileProvenance> = tokio::sync::OnceCell::const_new();
+
 #[derive(Clone)]
 pub(crate) struct DiagnosticJobRuntime {
    config: HeartbeatConfig,
@@ -386,16 +389,21 @@ fn target_and_identity(config: &HeartbeatConfig) -> Result<(DiagnosticJobTarget,
 }

 pub(super) async fn executable_provenance() -> Result<ProfileProvenance, &'static str> {
-    let digest = tokio::task::spawn_blocking(hash_current_executable)
-        .await
-        .map_err(|_| "PROVENANCE_FAILED")?
-        .map_err(|_| "PROVENANCE_FAILED")?;
-    Ok(ProfileProvenance::new(
-        crate::version::build::COMMIT_HASH,
-        digest,
-        env!("CARGO_PKG_VERSION"),
-        enabled_build_features(),
-    ))
+    let provenance = EXECUTABLE_PROVENANCE
+        .get_or_try_init(|| async {
+            let digest = tokio::task::spawn_blocking(hash_current_executable)
+                .await
+                .map_err(|_| "PROVENANCE_FAILED")?
+                .map_err(|_| "PROVENANCE_FAILED")?;
+            Ok::<_, &'static str>(ProfileProvenance::new(
+                crate::version::build::COMMIT_HASH,
+                digest,
+                env!("CARGO_PKG_VERSION"),
+                enabled_build_features(),
+            ))
+        })
+        .await?;
+    Ok(provenance.clone())
 }

 fn hash_current_executable() -> Result<String, ()> {
@@ -683,13 +683,13 @@ mod tests {
    }

    async fn wait_running(status: &mut watch::Receiver<DiagnosticScheduleStatus>) {
-        tokio::time::timeout(Duration::from_secs(2), async {
+        tokio::time::timeout(Duration::from_secs(15), async {
            while !matches!(&*status.borrow(), DiagnosticScheduleStatus::Running { .. }) {
                status.changed().await.expect("schedule remains active");
            }
        })
        .await
-        .expect("running");
+        .unwrap_or_else(|_| panic!("timed out waiting for schedule to run; current status: {:?}", status.borrow()));
    }

    fn blocking_runner() -> Runner {
@@ -2460,9 +2460,11 @@ mod tests {
                signed_request
                    .headers_mut()
                    .extend(crate::storage_api::server::http::gen_signature_headers(&uri, &Method::GET).unwrap());
+                sender.ready().await.unwrap();
                let success = sender.send_request(signed_request).await.unwrap();
                assert_eq!(success.status(), StatusCode::OK);
                success.into_body().collect().await.unwrap();
+                sender.ready().await.unwrap();
                let rejected = sender
                    .send_request(
                        Request::builder()
@@ -2476,6 +2478,7 @@ mod tests {
                    .unwrap();
                assert!(rejected.status().is_client_error());
                rejected.into_body().collect().await.unwrap();
+                sender.ready().await.unwrap();
                let unrelated = sender
                    .send_request(Request::builder().uri("/not-rpc").body(Empty::<Bytes>::new()).unwrap())
                    .await
@@ -3144,18 +3147,47 @@ mod tests {
    async fn local_runtime_profile_cancellation_waits_for_lease_release() {
        let state = tempfile::tempdir().unwrap();
        std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).unwrap();
+        let provenance = super::super::job_delivery::executable_provenance()
+            .await
+            .expect("test executable provenance should be available");
+        assert!(provenance.is_valid(), "profile capture requires valid executable provenance");
        let (mut request, _) = runtime_request(state.path());
        request.duration_millis = 5_000;
-        let runtime = spawn_local_trace_capture_runtime(state.path(), &CancellationToken::new()).unwrap();
+        let server_state_root = state.path().to_path_buf();
+        let (ready, wait) = tokio::sync::oneshot::channel();
+        let (stop, stopped) = tokio::sync::oneshot::channel();
+        let server = std::thread::spawn(move || {
+            tokio::runtime::Builder::new_multi_thread()
+                .worker_threads(2)
+                .enable_all()
+                .build()
+                .unwrap()
+                .block_on(async {
+                    let server_shutdown = CancellationToken::new();
+                    let runtime = spawn_local_trace_capture_runtime(&server_state_root, &server_shutdown).unwrap();
+                    ready.send(()).unwrap();
+                    let _ = stopped.await;
+                    runtime.shutdown().await;
+                });
+        });
+        wait.await.unwrap();
        let cancel = CancellationToken::new();
        let task_cancel = cancel.clone();
        let state_root = state.path().to_path_buf();
-        let task = tokio::spawn(async move { super::request_local_runtime_profile(&state_root, request, &task_cancel).await });
+        let mut task =
+            tokio::spawn(async move { super::request_local_runtime_profile(&state_root, request, &task_cancel).await });
        tokio::time::timeout(Duration::from_secs(30), async {
            loop {
                if super::super::profile_cpu::CollectorLease::acquire().is_err() {
                    break;
                }
+                if task.is_finished() {
+                    let detail = match (&mut task).await.expect("request task should not panic") {
+                        Ok(_) => "request completed successfully".to_owned(),
+                        Err(error) => format!("request failed: {error}"),
+                    };
+                    panic!("runtime profile finished before acquiring its collector lease: {detail}");
+                }
                tokio::time::sleep(Duration::from_millis(5)).await;
            }
        })
@@ -3165,7 +3197,8 @@ mod tests {
        assert!(matches!(task.await.unwrap(), Err(LocalTraceCaptureError::RuntimeProfile(code)) if code == "CANCELLED"));
        let lease = super::super::profile_cpu::CollectorLease::acquire().expect("client cancellation joins collector");
        drop(lease);
-        runtime.shutdown().await;
+        stop.send(()).unwrap();
+        server.join().unwrap();
    }
    #[tokio::test]
    #[serial]
@@ -253,67 +253,52 @@ fn numeric_json_value(value: &Value) -> Option<u64> {
 }

 #[cfg(any(not(target_os = "windows"), test))]
-fn numeric_json_field(value: &Value, field: &str) -> Option<u64> {
+fn find_mimalloc_stat<'a>(value: &'a Value, metric: &str) -> Option<&'a Value> {
    match value {
        Value::Object(fields) => fields
-            .get(field)
-            .and_then(numeric_json_value)
-            .or_else(|| fields.values().find_map(|value| numeric_json_field(value, field))),
-        Value::Array(values) => values.iter().find_map(|value| numeric_json_field(value, field)),
-        _ => None,
-    }
-}
-
-#[cfg(any(not(target_os = "windows"), test))]
-fn mimalloc_stat_field(value: &Value, metric: &str, field: &str) -> Option<u64> {
-    match value {
-        Value::Object(fields) => {
-            if let Some(metric_value) = fields.get(metric)
-                && let Some(value) = numeric_json_value(metric_value).or_else(|| numeric_json_field(metric_value, field))
-            {
-                return Some(value);
-            }
-
-            fields.values().find_map(|value| mimalloc_stat_field(value, metric, field))
-        }
-        Value::Array(values) => values.iter().find_map(|value| mimalloc_stat_field(value, metric, field)),
+            .get(metric)
+            .or_else(|| fields.values().find_map(|value| find_mimalloc_stat(value, metric))),
+        Value::Array(values) => values.iter().find_map(|value| find_mimalloc_stat(value, metric)),
        _ => None,
    }
 }

 #[cfg(any(not(target_os = "windows"), test))]
 fn mimalloc_stat_current(value: &Value, metric: &str) -> Option<u64> {
-    mimalloc_stat_field(value, metric, "current")
-}
-
-#[cfg(any(not(target_os = "windows"), test))]
-fn mimalloc_stat_sum(value: &Value, metrics: &[&str], field: &str) -> Option<u64> {
-    metrics
-        .iter()
-        .map(|metric| mimalloc_stat_field(value, metric, field))
-        .try_fold(0_u64, |sum, value| value.map(|value| sum.saturating_add(value)))
-        .filter(|value| *value > 0)
+    find_mimalloc_stat(value, metric)?
+        .as_object()?
+        .get("current")
+        .and_then(numeric_json_value)
 }

 #[cfg(any(not(target_os = "windows"), test))]
 fn parse_mimalloc_stats_json(stats_json: &str) -> Option<AllocatorMemoryObservation> {
    let value = serde_json::from_str::<Value>(stats_json).ok()?;
-    let malloc_metrics = ["malloc_normal", "malloc_huge"];
-    let observation = AllocatorMemoryObservation {
+    let mut observation = AllocatorMemoryObservation {
        reserved_bytes: mimalloc_stat_current(&value, "reserved"),
        committed_bytes: mimalloc_stat_current(&value, "committed"),
        page_committed_bytes: mimalloc_stat_current(&value, "page_committed"),
-        malloc_requested_bytes: mimalloc_stat_current(&value, "malloc_requested")
-            .filter(|value| *value > 0)
-            .or_else(|| mimalloc_stat_sum(&value, &malloc_metrics, "current")),
-        malloc_requested_peak_bytes: mimalloc_stat_field(&value, "malloc_requested", "peak")
-            .filter(|value| *value > 0)
-            .or_else(|| mimalloc_stat_sum(&value, &malloc_metrics, "peak")),
-        malloc_requested_total_bytes: mimalloc_stat_field(&value, "malloc_requested", "total")
-            .filter(|value| *value > 0)
-            .or_else(|| mimalloc_stat_sum(&value, &malloc_metrics, "total")),
        heap_count: mimalloc_stat_current(&value, "heaps").or_else(|| mimalloc_stat_current(&value, "heap_count")),
+        ..AllocatorMemoryObservation::default()
    };
+    if let Some(requested) = find_mimalloc_stat(&value, "malloc_requested") {
+        // Count fields belong to one statistic; scalar counters only have a total.
+        let (current, peak, total) = match requested {
+            Value::Object(fields) => (
+                fields.get("current").and_then(numeric_json_value),
+                fields.get("peak").and_then(numeric_json_value),
+                fields.get("total").and_then(numeric_json_value),
+            ),
+            _ => (None, None, numeric_json_value(requested)),
+        };
+        // All-zero stats can mean accounting was compiled out. Normal/huge
+        // allocation stats are not a substitute for live requested bytes.
+        if [current, peak, total].into_iter().flatten().any(|value| value > 0) {
+            observation.malloc_requested_bytes = current;
+            observation.malloc_requested_peak_bytes = peak;
+            observation.malloc_requested_total_bytes = total;
+        }
+    }

    if observation == AllocatorMemoryObservation::default() {
        None
@@ -405,6 +390,13 @@ fn record_container_resource_detection() {
    metrics::gauge!("rustfs_container_overridden").set(if res.overridden { 1.0 } else { 0.0 });
 }

+fn allocator_snapshot_or_unavailable(snapshot: Option<AllocatorMemorySnapshot>) -> AllocatorMemorySnapshot {
+    snapshot.unwrap_or_else(|| AllocatorMemorySnapshot {
+        backend: crate::allocator_reclaim::allocator_backend(),
+        observation: AllocatorMemoryObservation::default(),
+    })
+}
+
 async fn record_memory_snapshot(process_sampler: Arc<Mutex<ProcessSampler>>) {
    match tokio::task::spawn_blocking(move || {
        let mut sampler = process_sampler.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
@@ -433,11 +425,12 @@ async fn record_memory_snapshot(process_sampler: Arc<Mutex<ProcessSampler>>) {
                );
            }

-            if let Some(allocator) = allocator {
-                record_allocator_memory_observation(allocator.backend, allocator.observation);
-            }
+            let allocator = allocator_snapshot_or_unavailable(allocator);
+            record_allocator_memory_observation(allocator.backend, allocator.observation);
        }
        Err(err) => {
+            let allocator = allocator_snapshot_or_unavailable(None);
+            record_allocator_memory_observation(allocator.backend, allocator.observation);
            debug!(error = ?err, "memory observability sampler task failed");
        }
    }
@@ -541,7 +534,7 @@ mod tests {
    }

    #[test]
-    fn parse_mimalloc_stats_json_falls_back_to_allocated_bytes_when_requested_is_zero() {
+    fn parse_mimalloc_stats_json_does_not_infer_live_bytes_from_allocation_totals() {
        let parsed = parse_mimalloc_stats_json(
            r#"{
                "stat_version": 1,
@@ -558,17 +551,107 @@ mod tests {

        assert_eq!(parsed.reserved_bytes, Some(1_048_576));
        assert_eq!(parsed.committed_bytes, Some(524_288));
-        assert_eq!(parsed.malloc_requested_bytes, Some(262_144));
-        assert_eq!(parsed.malloc_requested_peak_bytes, Some(393_216));
-        assert_eq!(parsed.malloc_requested_total_bytes, Some(10_485_760));
+        assert_eq!(parsed.malloc_requested_bytes, None);
+        assert_eq!(parsed.malloc_requested_peak_bytes, None);
+        assert_eq!(parsed.malloc_requested_total_bytes, None);
        assert_eq!(parsed.heap_count, Some(1));
    }

+    #[test]
+    fn parse_mimalloc_stats_json_scalar_requested_is_only_a_cumulative_total() {
+        for requested in ["311856733152", "\"311856733152\""] {
+            let payload = format!(r#"{{ "stat_version": 5, "malloc_requested": {requested} }}"#);
+            let parsed = parse_mimalloc_stats_json(&payload).expect("counter stats should parse");
+            assert_eq!(parsed.malloc_requested_bytes, None);
+            assert_eq!(parsed.malloc_requested_peak_bytes, None);
+            assert_eq!(parsed.malloc_requested_total_bytes, Some(311_856_733_152));
+        }
+    }
+
+    #[test]
+    fn parse_mimalloc_stats_json_preserves_zero_after_live_allocations_are_freed() {
+        let parsed =
+            parse_mimalloc_stats_json(r#"{ "malloc_requested": { "current": 0, "peak": 10485760, "total": 20971520 } }"#)
+                .expect("supported drained allocation stats should parse");
+        assert_eq!(parsed.malloc_requested_bytes, Some(0));
+        assert_eq!(parsed.malloc_requested_peak_bytes, Some(10_485_760));
+        assert_eq!(parsed.malloc_requested_total_bytes, Some(20_971_520));
+    }
+
+    #[test]
+    fn parse_mimalloc_stats_json_does_not_fabricate_missing_or_invalid_requested_fields() {
+        let parsed = parse_mimalloc_stats_json(r#"{ "malloc_requested": { "total": 10485760 } }"#)
+            .expect("partial counter stats should parse");
+        assert_eq!(parsed.malloc_requested_bytes, None);
+        assert_eq!(parsed.malloc_requested_peak_bytes, None);
+        assert_eq!(parsed.malloc_requested_total_bytes, Some(10_485_760));
+        for payload in [
+            r#"{ "malloc_requested": -1 }"#,
+            r#"{ "malloc_requested": "invalid" }"#,
+            r#"{ "malloc_requested": { "current": -1 } }"#,
+            r#"{ "malloc_requested": { "unexpected": { "current": 123 } } }"#,
+        ] {
+            assert_eq!(parse_mimalloc_stats_json(payload), None, "invalid schema: {payload}");
+        }
+    }
+
+    #[test]
+    fn parse_mimalloc_stats_json_does_not_mix_duplicate_statistic_nodes() {
+        let parsed = parse_mimalloc_stats_json(
+            r#"{
+                "malloc_requested": 10,
+                "wrapper": { "malloc_requested": { "current": 20, "peak": 30, "total": 40 } }
+            }"#,
+        )
+        .expect("top-level counter should parse");
+        assert_eq!(parsed.malloc_requested_bytes, None);
+        assert_eq!(parsed.malloc_requested_peak_bytes, None);
+        assert_eq!(parsed.malloc_requested_total_bytes, Some(10));
+
+        let parsed = parse_mimalloc_stats_json(
+            r#"{
+                "a": { "malloc_requested": { "current": 7 } },
+                "b": { "malloc_requested": { "peak": 9, "total": 20 } }
+            }"#,
+        )
+        .expect("first wrapped statistic should parse");
+        assert_eq!(parsed.malloc_requested_bytes, Some(7));
+        assert_eq!(parsed.malloc_requested_peak_bytes, None);
+        assert_eq!(parsed.malloc_requested_total_bytes, None);
+    }
+
+    #[test]
+    fn parse_mimalloc_stats_json_keeps_all_zero_requested_stats_unavailable() {
+        for requested in ["0", "\"0\"", r#"{ "current": 0, "peak": 0, "total": 0 }"#] {
+            let payload = format!(r#"{{ "reserved": {{ "current": 1 }}, "malloc_requested": {requested} }}"#);
+            let parsed = parse_mimalloc_stats_json(&payload).expect("reserved count should parse");
+            assert_eq!(parsed.reserved_bytes, Some(1));
+            assert_eq!(parsed.malloc_requested_bytes, None);
+            assert_eq!(parsed.malloc_requested_peak_bytes, None);
+            assert_eq!(parsed.malloc_requested_total_bytes, None);
+        }
+    }
+
    #[test]
    fn parse_mimalloc_stats_json_rejects_unrecognized_payload() {
        assert_eq!(parse_mimalloc_stats_json(r#"{ "allocator": "unknown" }"#), None);
    }

+    #[test]
+    fn missing_allocator_snapshot_marks_requested_bytes_as_unavailable() {
+        let supported = super::AllocatorMemorySnapshot {
+            backend: crate::allocator_reclaim::allocator_backend(),
+            observation: rustfs_io_metrics::AllocatorMemoryObservation {
+                malloc_requested_bytes: Some(128),
+                ..Default::default()
+            },
+        };
+        assert_eq!(super::allocator_snapshot_or_unavailable(Some(supported)), supported);
+        let unavailable = super::allocator_snapshot_or_unavailable(None);
+        assert_eq!(unavailable.backend, supported.backend);
+        assert_eq!(unavailable.observation, rustfs_io_metrics::AllocatorMemoryObservation::default());
+    }
+
    #[test]
    fn read_allocator_memory_snapshot_uses_mimalloc_stats_json() {
        let snapshot = super::read_allocator_memory_snapshot();
@@ -787,6 +787,15 @@ impl ConcurrencyManager {
            .await
    }

+    /// Admit a server-side multipart copy through the shared write pool.
+    ///
+    /// Source length is not authoritative until metadata is validated, but
+    /// admission must precede source readers and lifecycle locks. Use the
+    /// existing unknown-length weight and bounded multipart wait queue.
+    pub async fn admit_multipart_copy(&self) -> Result<ForegroundWriteAdmission, tokio::sync::AcquireError> {
+        self.admit_multipart_part(-1).await
+    }
+
    // ============================================
    // Adaptive I/O Strategy Methods
    // ============================================
@@ -1847,6 +1856,24 @@ mod integration_tests {
        )
    }

+    #[tokio::test]
+    #[serial]
+    async fn multipart_copy_admission_uses_unknown_size_weight_and_shared_pool() {
+        for (configured_limit, expected_weight) in [(None, 8), (Some("1"), 1)] {
+            let mut manager = ConcurrencyManager::new();
+            manager.foreground_write_admission_policy = automatic_write_policy(configured_limit, 64);
+            let copy = manager.admit_multipart_copy().await.expect("copy admission should succeed");
+            assert!(matches!(copy, ForegroundWriteAdmission::Admitted(_)));
+            assert_eq!(manager.put_object_admission_snapshot().active, Some(expected_weight));
+            if configured_limit.is_some() {
+                let part = manager.admit_put_object(-1).await.expect("shared gate remains open");
+                assert!(matches!(part, ForegroundWriteAdmission::Rejected));
+            }
+            drop(copy);
+            assert_eq!(manager.put_object_admission_snapshot().active, Some(0));
+        }
+    }
+
    #[tokio::test(start_paused = true)]
    #[serial]
    async fn test_concurrency_manager_auto_admission_completes_bounded_multipart_bursts() {
@@ -71,7 +71,7 @@ pub(crate) fn build_list_parts_output(res: ListPartsInfo) -> ListPartsOutput {
        owner: Some(owner),
        initiator: Some(initiator),
        is_truncated: Some(res.is_truncated),
-        next_part_number_marker: res.next_part_number_marker.try_into().ok(),
+        next_part_number_marker: res.next_part_number_marker.and_then(|v| v.try_into().ok()),
        max_parts: res.max_parts.try_into().ok(),
        part_number_marker: res.part_number_marker.try_into().ok(),
        storage_class: Some(
@@ -228,7 +228,7 @@ mod tests {
            upload_id: "upload-a".to_string(),
            storage_class: "STANDARD".to_string(),
            part_number_marker: 1,
-            next_part_number_marker: 2,
+            next_part_number_marker: Some(2),
            max_parts: 1000,
            is_truncated: true,
            parts: vec![PartInfo {
@@ -259,6 +259,55 @@ mod tests {
        assert_eq!(output.initiator, Some(rustfs_initiator()));
    }

+    #[test]
+    fn test_list_parts_output_xml_omits_terminal_marker() {
+        for (marker, part_numbers) in [(0, vec![]), (3, vec![10]), (0, vec![1, 3])] {
+            let output = build_list_parts_output(ListPartsInfo {
+                part_number_marker: marker,
+                max_parts: 2,
+                parts: part_numbers
+                    .into_iter()
+                    .map(|part_num| PartInfo {
+                        part_num,
+                        ..Default::default()
+                    })
+                    .collect(),
+                ..Default::default()
+            });
+
+            assert_eq!(output.is_truncated, Some(false));
+            assert_eq!(output.next_part_number_marker, None);
+            let xml = String::from_utf8(crate::storage::storage_api::serialize(&output).expect("output should serialize"))
+                .expect("XML should be UTF-8");
+            assert!(xml.contains("<IsTruncated>false</IsTruncated>"));
+            assert!(xml.contains(&format!("<PartNumberMarker>{marker}</PartNumberMarker>")));
+            assert!(!xml.contains("NextPartNumberMarker"), "terminal page must omit the token: {xml}");
+        }
+    }
+
+    #[test]
+    fn test_list_parts_output_xml_preserves_truncated_marker() {
+        let output = build_list_parts_output(ListPartsInfo {
+            is_truncated: true,
+            next_part_number_marker: Some(3),
+            max_parts: 2,
+            parts: [1, 3]
+                .into_iter()
+                .map(|part_num| PartInfo {
+                    part_num,
+                    ..Default::default()
+                })
+                .collect(),
+            ..Default::default()
+        });
+
+        let xml = String::from_utf8(crate::storage::storage_api::serialize(&output).expect("output should serialize"))
+            .expect("XML should be UTF-8");
+        assert!(xml.contains("<IsTruncated>true</IsTruncated>"));
+        assert!(xml.contains("<NextPartNumberMarker>3</NextPartNumberMarker>"));
+        assert_eq!(output.parts.as_ref().expect("parts should be present").len(), 2);
+    }
+
    #[test]
    fn test_list_parts_output_reports_logical_size_for_compressed_parts() {
        let input = ListPartsInfo {
@@ -374,7 +423,7 @@ mod tests {
        let input = ListPartsInfo {
            storage_class: "STANDARD_IA".to_string(),
            part_number_marker: usize::MAX,
-            next_part_number_marker: usize::MAX,
+            next_part_number_marker: Some(usize::MAX),
            max_parts: usize::MAX,
            parts: vec![PartInfo {
                part_num: usize::MAX,
@@ -80,9 +80,20 @@ async fn cpu_without_local_sampler_is_explicitly_unsupported() {
 #[cfg(feature = "pyroscope")]
 #[tokio::test]
 async fn cpu_with_local_sampler_returns_bounded_redacted_samples() {
-    let result = capture_cpu_profile(&request(), &CancellationToken::new())
-        .await
-        .expect("local CPU capture should complete");
+    // A CPU-time sampler needs CPU work during its capture window.
+    let workload_cancel = CancellationToken::new();
+    let _workload_cancel_on_drop = workload_cancel.clone().drop_guard();
+    let worker_cancel = workload_cancel.clone();
+    let worker = std::thread::spawn(move || {
+        let mut value = 1_u64;
+        while !worker_cancel.is_cancelled() {
+            value = std::hint::black_box(value.wrapping_mul(6364136223846793005).wrapping_add(1));
+        }
+    });
+    let capture = capture_cpu_profile(&request(), &CancellationToken::new()).await;
+    workload_cancel.cancel();
+    worker.join().expect("CPU workload should stop after capture");
+    let result = capture.expect("local CPU capture should complete");
    assert!(matches!(result.outcome(), ProfileOutcome::Succeeded | ProfileOutcome::Partial));
    let json = serde_json::to_string(&result).expect("result JSON");
    assert!(json.contains("sha256:"));
@@ -113,12 +113,27 @@ def fault_tolerance_counts(text):
    return counts


+def auto_testing_dir():
+    """Locate the auto-testing checkout. Lanes that run this script from a
+    subdirectory checkout (security keeps its rustfs clone in rustfs-repo/)
+    still check auto-testing out at the workspace root, so ROOT alone is not
+    always the right base."""
+    candidates = [ROOT / "auto-testing"]
+    workspace = os.environ.get("GITHUB_WORKSPACE")
+    if workspace:
+        candidates.append(Path(workspace) / "auto-testing")
+    for candidate in candidates:
+        if (candidate / ".git").exists():
+            return candidate
+    return candidates[0]
+
+
 def record(chain, suite, report, output):
    require(suite in SUITES, "unknown suite")
    result = {"schema": 1, "suite": suite, "chain": chain, "valid": False, "counts": {}, "report_sha256": None}
    error = None
    try:
-        private_head = subprocess.check_output(["git", "-C", "auto-testing", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
+        private_head = subprocess.check_output(["git", "-C", str(auto_testing_dir()), "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
        require(private_head == chain["testing_sha"], "suite used a different private script revision")
        require(report.is_file() and 0 < report.stat().st_size <= MAX_REPORT, "missing, empty or oversized report")
        data = report.read_bytes()
@@ -5,6 +5,7 @@ import hashlib
 import io
 import json
 from pathlib import Path
+import subprocess
 import tempfile
 import unittest
 from unittest import mock
@@ -271,6 +272,34 @@ class EnvelopeTests(unittest.TestCase):
                evidence.record(self.chain, "kms", report, self.root / "new" / "kms.json")
            self.assertFalse(Path(self.env["GITHUB_OUTPUT"]).exists())

+    def test_record_uses_workspace_testing_checkout_for_nested_lane(self):
+        lane_checkout = self.root / "rustfs-repo"
+        lane_checkout.mkdir()
+        private_checkout = self.root / "auto-testing"
+        private_checkout.mkdir()
+        subprocess.run(["git", "init", "--quiet", str(private_checkout)], check=True)
+        subprocess.run(["git", "-c", "user.name=Fixture", "-c", "user.email=fixture@example.invalid",
+                        "-c", "commit.gpgSign=false", "commit", "--allow-empty", "-qm", "fixture"],
+                       cwd=private_checkout, check=True)
+        private_head = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=private_checkout, text=True).strip()
+        chain = {**self.chain, "testing_sha": private_head}
+        report = self.root / "security.md"
+        report.write_text("| Case | Status |\n| --- | --- |\n| SEC-1 | PASS |\n")
+        env = {**self.env, "GITHUB_WORKSPACE": str(self.root)}
+        output = self.root / "nested-success" / "security.json"
+        with mock.patch.object(evidence, "ROOT", lane_checkout), mock.patch.dict(evidence.os.environ, env):
+            evidence.record(chain, "security", report, output)
+            self.assertTrue(json.loads(output.read_text())["valid"])
+            wrong_pin_output = self.root / "nested-wrong-pin" / "security.json"
+            with self.assertRaisesRegex(ValueError, "suite used a different private script revision"):
+                evidence.record({**chain, "testing_sha": "f" * 40}, "security", report, wrong_pin_output)
+            self.assertFalse(json.loads(wrong_pin_output.read_text())["valid"])
+        fallback_output = self.root / "local-success" / "security.json"
+        with mock.patch.object(evidence, "ROOT", self.root), mock.patch.dict(evidence.os.environ, self.env):
+            evidence.os.environ.pop("GITHUB_WORKSPACE", None)
+            evidence.record(chain, "security", report, fallback_output)
+            self.assertTrue(json.loads(fallback_output.read_text())["valid"])
+
    def test_unknown_status_cannot_hide_among_passing_cases(self):
        text = "| Case | Name | Status |\n| --- | --- | --- |\n| KMS-1 | fixture | PASS |\n| KMS-2 | fixture | NOT RUN |\n"
        with self.assertRaises(ValueError):
@@ -0,0 +1,660 @@
+#!/usr/bin/env bash
+
+# Manual Docker validation for rustfs/backlog#2704.
+#
+# The script recreates the low-concurrency multipart UploadPart shape reported in
+# rustfs/rustfs#8271: one single-node RustFS container with four data paths and a
+# Warp multipart workload whose PUTPART concurrency is five by default. It is a
+# diagnostic harness, not a CI gate. Without --run and --ack-isolated it reports
+# SKIP and exits successfully.
+
+set -euo pipefail
+
+RUN=0
+ACK_ISOLATED=0
+STRICT=0
+KEEP_CONTAINERS=0
+UNSAFE_BYPASS_DISK_CHECK=0
+IMAGE="rustfs/rustfs:1.0.0"
+CASES="reported"
+OUT_DIR=""
+CONTAINER_PREFIX="rustfs-2704"
+PORT=19000
+CONSOLE_PORT=19001
+DURATION="5m"
+CLIENTS=5
+CONCURRENT=1
+PART_CONCURRENT=1
+PARTS=100
+PART_SIZE="5MiB"
+BUCKET="rustfs-2704-uploadpart"
+WARP_BIN="warp"
+SAMPLE_INTERVAL="1"
+TLS_MODE="auto"
+TLS_DIR=""
+DATA_ROOT=""
+ACCESS_KEY="rustfsadmin"
+SECRET_KEY="rustfsadmin"
+MEMORY_LIMIT=""
+CPU_LIMIT=""
+DOCKER_PLATFORM=""
+EXTRA_RUSTFS_ENV=()
+CURRENT_CASE_DIR=""
+CURRENT_CONTAINER_NAME=""
+
+usage() {
+  cat <<'USAGE'
+Usage:
+  scripts/validate_issue2704_uploadpart_memory.sh --run --ack-isolated [options]
+
+Default workload:
+  * Docker RustFS image rustfs/rustfs:1.0.0
+  * one node, four bind-mounted data directories
+  * TLS when a TLS bundle can be provided or generated
+  * Warp multipart-put with five client streams, --concurrent=1 shape approximated
+    as local PUTPART concurrency 5, --part.concurrent=1, --part.size=5MiB
+
+Options:
+  --run                         Execute the workload. Without this, prints SKIP.
+  --ack-isolated                Confirm the Docker host is dedicated to this run.
+  --strict                      Missing prerequisites are FAIL instead of SKIP.
+  --image IMAGE                 RustFS image (default: rustfs/rustfs:1.0.0).
+  --case LIST                   Comma-separated cases to run (default: reported).
+                                Cases: reported,no-cache-env,data-cache-disabled,
+                                write-reclaim,direct-write,tls-off,metrics-off.
+  --out-dir PATH                Artifact directory (default: temporary).
+  --data-root PATH              Use existing/prepared data root; creates rustfs0..3.
+  --container-prefix NAME       Docker container name prefix (default: rustfs-2704).
+  --port PORT                   Host S3 port for the first case (default: 19000).
+  --console-port PORT           Host console port for the first case (default: 19001).
+  --duration DURATION           Warp duration (default: 5m).
+  --clients N                   Local client streams / PUTPART concurrency (default: 5).
+  --concurrent N                Warp --concurrent per stream shape (default: 1).
+  --part.concurrent N           Warp --part.concurrent (default: 1).
+  --parts N                     Multipart parts per object (default: 100).
+  --part.size SIZE              Multipart part size (default: 5MiB).
+  --bucket NAME                 Warp bucket (default: rustfs-2704-uploadpart).
+  --warp-bin PATH               Warp binary (default: warp on PATH).
+  --sample-interval SECONDS     Sampler interval (default: 1).
+  --tls auto|on|off             TLS behavior (default: auto).
+  --tls-dir PATH                Existing RustFS TLS bundle with rustfs_cert.pem/rustfs_key.pem.
+  --memory-limit BYTES          Optional Docker --memory value; omitted by default.
+  --cpu-limit VALUE             Optional Docker --cpus value; omitted by default.
+  --docker-platform PLATFORM    Optional Docker --platform value.
+  --env KEY=VALUE               Extra RustFS env, repeatable.
+  --keep-containers             Leave containers for manual inspection.
+  --unsafe-bypass-disk-check    Set RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true for
+                                local same-device approximations only. Do not use
+                                this for the real RHEL/XFS four-disk reproduction.
+  -h, --help                    Show this help.
+
+Artifacts per case:
+  * summary.env                 command, image, host, fs, cgroup and case metadata
+  * sampler.csv                 process, cgroup, dirty/page-cache and Docker memory samples
+  * smaps_rollup/ and status/   periodic /proc snapshots on Linux
+  * cgroup/                     periodic memory.stat and memory.events on Linux cgroup v2
+  * rustfs.log                  Docker logs after the run
+  * warp.log and warp*.zst      Warp output and bench data
+
+The script never writes credentials to artifacts beyond the local test access key
+name. Use only disposable local credentials.
+USAGE
+}
+
+fail() {
+  printf 'FAIL: %s\n' "$1" >&2
+  exit 1
+}
+
+skip_or_fail() {
+  local message=$1
+  if ((STRICT)); then
+    fail "$message"
+  fi
+  printf 'SKIP: %s\n' "$message"
+  exit 0
+}
+
+need_value() {
+  (($# >= 2)) || fail "option $1 requires a value"
+}
+
+log() {
+  printf '[%s] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*"
+}
+
+split_cases() {
+  local input=$1
+  local IFS=','
+  read -r -a CASE_LIST <<<"$input"
+}
+
+is_positive_int() {
+  [[ ${1:-} =~ ^[0-9]+$ ]] && (($1 > 0))
+}
+
+is_nonnegative_int() {
+  [[ ${1:-} =~ ^[0-9]+$ ]]
+}
+
+normalize_case_name() {
+  local case_name=$1
+  case "$case_name" in
+    reported|no-cache-env|data-cache-disabled|write-reclaim|direct-write|tls-off|metrics-off)
+      printf '%s\n' "$case_name"
+      ;;
+    *)
+      fail "unknown case '$case_name'"
+      ;;
+  esac
+}
+
+parse_args() {
+  while (($#)); do
+    case "$1" in
+      --run) RUN=1; shift ;;
+      --ack-isolated) ACK_ISOLATED=1; shift ;;
+      --strict) STRICT=1; shift ;;
+      --keep-containers) KEEP_CONTAINERS=1; shift ;;
+      --unsafe-bypass-disk-check) UNSAFE_BYPASS_DISK_CHECK=1; shift ;;
+      --image) need_value "$@"; IMAGE=$2; shift 2 ;;
+      --case|--cases) need_value "$@"; CASES=$2; shift 2 ;;
+      --out-dir) need_value "$@"; OUT_DIR=$2; shift 2 ;;
+      --data-root) need_value "$@"; DATA_ROOT=$2; shift 2 ;;
+      --container-prefix) need_value "$@"; CONTAINER_PREFIX=$2; shift 2 ;;
+      --port) need_value "$@"; PORT=$2; shift 2 ;;
+      --console-port) need_value "$@"; CONSOLE_PORT=$2; shift 2 ;;
+      --duration) need_value "$@"; DURATION=$2; shift 2 ;;
+      --clients) need_value "$@"; CLIENTS=$2; shift 2 ;;
+      --concurrent) need_value "$@"; CONCURRENT=$2; shift 2 ;;
+      --part.concurrent|--part-concurrent) need_value "$@"; PART_CONCURRENT=$2; shift 2 ;;
+      --parts) need_value "$@"; PARTS=$2; shift 2 ;;
+      --part.size|--part-size) need_value "$@"; PART_SIZE=$2; shift 2 ;;
+      --bucket) need_value "$@"; BUCKET=$2; shift 2 ;;
+      --warp-bin) need_value "$@"; WARP_BIN=$2; shift 2 ;;
+      --sample-interval) need_value "$@"; SAMPLE_INTERVAL=$2; shift 2 ;;
+      --tls) need_value "$@"; TLS_MODE=$2; shift 2 ;;
+      --tls-dir) need_value "$@"; TLS_DIR=$2; shift 2 ;;
+      --memory-limit) need_value "$@"; MEMORY_LIMIT=$2; shift 2 ;;
+      --cpu-limit) need_value "$@"; CPU_LIMIT=$2; shift 2 ;;
+      --docker-platform) need_value "$@"; DOCKER_PLATFORM=$2; shift 2 ;;
+      --env) need_value "$@"; EXTRA_RUSTFS_ENV+=("$2"); shift 2 ;;
+      -h|--help) usage; exit 0 ;;
+      *) fail "unknown option: $1" ;;
+    esac
+  done
+}
+
+require_prerequisites() {
+  ((RUN)) || skip_or_fail "pass --run to execute the #2704 Docker workload"
+  ((ACK_ISOLATED)) || skip_or_fail "pass --ack-isolated after confirming the Docker host is dedicated to this memory run"
+  command -v docker >/dev/null 2>&1 || skip_or_fail "docker is required"
+  command -v "$WARP_BIN" >/dev/null 2>&1 || skip_or_fail "warp is required; install it or pass --warp-bin"
+  command -v curl >/dev/null 2>&1 || skip_or_fail "curl is required"
+  is_positive_int "$CLIENTS" || fail "--clients must be a positive integer"
+  is_positive_int "$CONCURRENT" || fail "--concurrent must be a positive integer"
+  is_positive_int "$PART_CONCURRENT" || fail "--part.concurrent must be a positive integer"
+  is_positive_int "$PARTS" || fail "--parts must be a positive integer"
+  is_positive_int "$PORT" || fail "--port must be a positive integer"
+  is_positive_int "$CONSOLE_PORT" || fail "--console-port must be a positive integer"
+  python3 - "$SAMPLE_INTERVAL" <<'PY' || fail "--sample-interval must be a finite positive number"
+import math, sys
+value = float(sys.argv[1])
+raise SystemExit(0 if math.isfinite(value) and value > 0 else 1)
+PY
+}
+
+prepare_out_dir() {
+  if [[ -z $OUT_DIR ]]; then
+    OUT_DIR=$(mktemp -d "${TMPDIR:-/tmp}/rustfs-2704-uploadpart.XXXXXX")
+  else
+    mkdir -p "$OUT_DIR"
+  fi
+  OUT_DIR=$(cd "$OUT_DIR" && pwd)
+}
+
+generate_tls_bundle() {
+  local target_dir=$1
+  mkdir -p "$target_dir"
+  if [[ -s $target_dir/rustfs_cert.pem && -s $target_dir/rustfs_key.pem ]]; then
+    return 0
+  fi
+  command -v openssl >/dev/null 2>&1 || return 1
+  openssl req -x509 -newkey rsa:2048 -sha256 -days 2 -nodes \
+    -subj "/CN=localhost" \
+    -addext "subjectAltName=DNS:localhost,IP:127.0.0.1" \
+    -keyout "$target_dir/rustfs_key.pem" \
+    -out "$target_dir/rustfs_cert.pem" >/dev/null 2>&1 || return 1
+  cp "$target_dir/rustfs_cert.pem" "$target_dir/ca.crt"
+  chmod a+r "$target_dir/rustfs_cert.pem" "$target_dir/rustfs_key.pem" "$target_dir/ca.crt"
+}
+
+resolve_tls_for_case() {
+  local case_name=$1
+  local case_dir=$2
+  CASE_TLS_ENABLED=0
+  CASE_TLS_DIR=""
+  if [[ $case_name == tls-off || $TLS_MODE == off ]]; then
+    return 0
+  fi
+  if [[ -n $TLS_DIR ]]; then
+    [[ -s $TLS_DIR/rustfs_cert.pem && -s $TLS_DIR/rustfs_key.pem ]] || fail "--tls-dir must contain rustfs_cert.pem and rustfs_key.pem"
+    CASE_TLS_ENABLED=1
+    CASE_TLS_DIR=$(cd "$TLS_DIR" && pwd)
+    return 0
+  fi
+  if [[ $TLS_MODE == on || $TLS_MODE == auto ]]; then
+    local generated_dir="$case_dir/tls"
+    if generate_tls_bundle "$generated_dir"; then
+      CASE_TLS_ENABLED=1
+      CASE_TLS_DIR=$generated_dir
+      return 0
+    fi
+    [[ $TLS_MODE == auto ]] || fail "TLS requested but openssl is unavailable and --tls-dir was not supplied"
+  fi
+}
+
+case_env_args() {
+  local case_name=$1
+  local metrics_export_enabled=true
+  [[ $case_name == metrics-off ]] && metrics_export_enabled=false
+  CASE_ENV=(
+    -e "RUSTFS_ACCESS_KEY=$ACCESS_KEY"
+    -e "RUSTFS_SECRET_KEY=$SECRET_KEY"
+    -e "RUSTFS_ADDRESS=0.0.0.0:9000"
+    -e "RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001"
+    -e "RUSTFS_CONSOLE_ENABLE=true"
+    -e "RUSTFS_CORS_ALLOWED_ORIGINS=*"
+    -e "RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*"
+    -e "RUSTFS_VOLUMES=/data/rustfs{0...3}"
+    -e "RUSTFS_OBS_LOGS_EXPORT_ENABLED=false"
+    -e "RUSTFS_OBS_TRACES_EXPORT_ENABLED=false"
+    -e "RUSTFS_OBS_METRICS_EXPORT_ENABLED=$metrics_export_enabled"
+    -e "RUSTFS_OBS_SERVICE_NAME=rustfs-2704-$case_name"
+    -e "OTEL_RESOURCE_ATTRIBUTES=service.instance.id=rustfs-2704-$case_name"
+  )
+
+  if [[ $case_name != no-cache-env && $case_name != data-cache-disabled ]]; then
+    CASE_ENV+=(
+      -e "RUSTFS_OBJECT_CACHE_ENABLE=true"
+      -e "RUSTFS_OBJECT_CACHE_TTL_SECS=300"
+    )
+  fi
+  if [[ $case_name == data-cache-disabled ]]; then
+    CASE_ENV+=(
+      -e "RUSTFS_OBJECT_DATA_CACHE_ENABLE=false"
+      -e "RUSTFS_OBJECT_DATA_CACHE_TTL_SECS=300"
+    )
+  fi
+  if [[ $case_name == write-reclaim ]]; then
+    CASE_ENV+=(
+      -e "RUSTFS_OBJECT_FILE_CACHE_RECLAIM_WRITE_ENABLE=true"
+    )
+  fi
+  if [[ $case_name == direct-write ]]; then
+    CASE_ENV+=(
+      -e "RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=true"
+    )
+  fi
+  if ((UNSAFE_BYPASS_DISK_CHECK)); then
+    CASE_ENV+=(
+      -e "RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true"
+    )
+  fi
+  if ((CASE_TLS_ENABLED)); then
+    CASE_ENV+=(
+      -e "RUSTFS_TLS_PATH=/opt/tls"
+      -e "SSL_CERT_FILE=/opt/tls/ca.crt"
+    )
+  fi
+  if ((${#EXTRA_RUSTFS_ENV[@]})); then
+    for item in "${EXTRA_RUSTFS_ENV[@]}"; do
+      [[ $item == *=* ]] || fail "--env must be KEY=VALUE, got '$item'"
+      CASE_ENV+=(-e "$item")
+    done
+  fi
+}
+
+prepare_case_data_dirs() {
+  local case_dir=$1
+  if [[ -n $DATA_ROOT ]]; then
+    CASE_DATA_ROOT=$DATA_ROOT
+    mkdir -p "$CASE_DATA_ROOT"
+  else
+    CASE_DATA_ROOT="$case_dir/data"
+    mkdir -p "$CASE_DATA_ROOT"
+  fi
+  local disk_index
+  for disk_index in 0 1 2 3; do
+    mkdir -p "$CASE_DATA_ROOT/rustfs$disk_index"
+    chmod 0777 "$CASE_DATA_ROOT/rustfs$disk_index" || true
+  done
+}
+
+container_pid() {
+  local name=$1
+  docker inspect -f '{{.State.Pid}}' "$name" 2>/dev/null || true
+}
+
+container_cgroup_path() {
+  local pid=$1
+  [[ $(uname -s) == Linux ]] || return 1
+  [[ -r /proc/$pid/cgroup ]] || return 1
+  local rel
+  rel=$(awk -F: '$1 == "0" { print $3; exit }' "/proc/$pid/cgroup") || return 1
+  [[ -n $rel ]] || return 1
+  printf '/sys/fs/cgroup%s\n' "$rel"
+}
+
+read_status_field_kib() {
+  local status_file=$1
+  local field=$2
+  awk -v field="$field:" '$1 == field { print $2; found = 1; exit } END { if (!found) print "" }' "$status_file" 2>/dev/null
+}
+
+read_status_field_raw() {
+  local status_file=$1
+  local field=$2
+  awk -v field="$field:" '$1 == field { print $2; found = 1; exit } END { if (!found) print "" }' "$status_file" 2>/dev/null
+}
+
+meminfo_field_kib() {
+  local field=$1
+  awk -v field="$field:" '$1 == field { print $2; found = 1; exit } END { if (!found) print "" }' /proc/meminfo 2>/dev/null || true
+}
+
+cgroup_stat_field() {
+  local cgroup_path=$1
+  local field=$2
+  [[ -r $cgroup_path/memory.stat ]] || return 0
+  awk -v field="$field" '$1 == field { print $2; found = 1; exit } END { if (!found) print "" }' "$cgroup_path/memory.stat" 2>/dev/null
+}
+
+cgroup_event_field() {
+  local cgroup_path=$1
+  local field=$2
+  [[ -r $cgroup_path/memory.events ]] || return 0
+  awk -v field="$field" '$1 == field { print $2; found = 1; exit } END { if (!found) print "" }' "$cgroup_path/memory.events" 2>/dev/null
+}
+
+start_sampler() {
+  local case_dir=$1
+  local container_name=$2
+  local pid=$3
+  local cgroup_path=${4:-}
+  local csv="$case_dir/sampler.csv"
+  mkdir -p "$case_dir/status" "$case_dir/smaps_rollup" "$case_dir/cgroup"
+  printf 'epoch,pid,container_running,vmrss_kib,rssanon_kib,rssfile_kib,vmswap_kib,threads,fd_count,mem_total_kib,mem_available_kib,dirty_kib,writeback_kib,cgroup_current_bytes,cgroup_max,cgroup_anon_bytes,cgroup_file_bytes,cgroup_inactive_file_bytes,cgroup_active_file_bytes,cgroup_oom,cgroup_oom_kill,docker_mem_usage\n' >"$csv"
+
+  (
+    while :; do
+      local epoch running status_file smaps_file cgroup_current cgroup_max docker_mem_usage
+      epoch=$(date +%s)
+      running=$(docker inspect -f '{{.State.Running}}' "$container_name" 2>/dev/null || printf 'false')
+      status_file="/proc/$pid/status"
+      smaps_file="/proc/$pid/smaps_rollup"
+      local vmrss rssanon rssfile vmswap threads fd_count
+      vmrss=""; rssanon=""; rssfile=""; vmswap=""; threads=""; fd_count=""
+      if [[ -r $status_file ]]; then
+        cp "$status_file" "$case_dir/status/$epoch.status" 2>/dev/null || true
+        vmrss=$(read_status_field_kib "$status_file" VmRSS)
+        rssanon=$(read_status_field_kib "$status_file" RssAnon)
+        rssfile=$(read_status_field_kib "$status_file" RssFile)
+        vmswap=$(read_status_field_kib "$status_file" VmSwap)
+        threads=$(read_status_field_raw "$status_file" Threads)
+        fd_count=$(find "/proc/$pid/fd" -maxdepth 1 -type l 2>/dev/null | wc -l | tr -d ' ')
+      fi
+      if [[ -r $smaps_file ]]; then
+        cp "$smaps_file" "$case_dir/smaps_rollup/$epoch.smaps_rollup" 2>/dev/null || true
+      fi
+      local mem_total mem_available dirty writeback
+      mem_total=$(meminfo_field_kib MemTotal)
+      mem_available=$(meminfo_field_kib MemAvailable)
+      dirty=$(meminfo_field_kib Dirty)
+      writeback=$(meminfo_field_kib Writeback)
+      cgroup_current=""; cgroup_max=""
+      local cg_anon cg_file cg_inactive_file cg_active_file cg_oom cg_oom_kill
+      cg_anon=""; cg_file=""; cg_inactive_file=""; cg_active_file=""; cg_oom=""; cg_oom_kill=""
+      if [[ -n $cgroup_path && -d $cgroup_path ]]; then
+        [[ -r $cgroup_path/memory.current ]] && cgroup_current=$(<"$cgroup_path/memory.current")
+        [[ -r $cgroup_path/memory.max ]] && cgroup_max=$(<"$cgroup_path/memory.max")
+        [[ -r $cgroup_path/memory.stat ]] && cp "$cgroup_path/memory.stat" "$case_dir/cgroup/$epoch.memory.stat" 2>/dev/null || true
+        [[ -r $cgroup_path/memory.events ]] && cp "$cgroup_path/memory.events" "$case_dir/cgroup/$epoch.memory.events" 2>/dev/null || true
+        cg_anon=$(cgroup_stat_field "$cgroup_path" anon)
+        cg_file=$(cgroup_stat_field "$cgroup_path" file)
+        cg_inactive_file=$(cgroup_stat_field "$cgroup_path" inactive_file)
+        cg_active_file=$(cgroup_stat_field "$cgroup_path" active_file)
+        cg_oom=$(cgroup_event_field "$cgroup_path" oom)
+        cg_oom_kill=$(cgroup_event_field "$cgroup_path" oom_kill)
+      fi
+      docker_mem_usage=$(docker stats --no-stream --format '{{.MemUsage}}' "$container_name" 2>/dev/null | tr ',' ';' || true)
+      printf '%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n' \
+        "$epoch" "$pid" "$running" "$vmrss" "$rssanon" "$rssfile" "$vmswap" "$threads" "$fd_count" \
+        "$mem_total" "$mem_available" "$dirty" "$writeback" "$cgroup_current" "$cgroup_max" \
+        "$cg_anon" "$cg_file" "$cg_inactive_file" "$cg_active_file" "$cg_oom" "$cg_oom_kill" "$docker_mem_usage" >>"$csv"
+      [[ $running == true ]] || break
+      sleep "$SAMPLE_INTERVAL"
+    done
+  ) &
+  SAMPLER_PID=$!
+}
+
+stop_sampler() {
+  if [[ -n ${SAMPLER_PID:-} ]] && kill -0 "$SAMPLER_PID" 2>/dev/null; then
+    kill "$SAMPLER_PID" 2>/dev/null || true
+    wait "$SAMPLER_PID" 2>/dev/null || true
+  fi
+  SAMPLER_PID=""
+}
+
+write_summary_header() {
+  local summary_file=$1
+  local case_name=$2
+  local container_name=$3
+  local case_port=$4
+  local case_console_port=$5
+  local cgroup_path=${6:-}
+  {
+    printf 'issue=rustfs/backlog#2704\n'
+    printf 'case=%s\n' "$case_name"
+    printf 'image=%s\n' "$IMAGE"
+    printf 'container=%s\n' "$container_name"
+    printf 'endpoint_port=%s\n' "$case_port"
+    printf 'console_port=%s\n' "$case_console_port"
+    printf 'tls_enabled=%s\n' "$CASE_TLS_ENABLED"
+    printf 'tls_dir=%s\n' "${CASE_TLS_DIR:-}"
+    printf 'duration=%s\n' "$DURATION"
+    printf 'clients=%s\n' "$CLIENTS"
+    printf 'warp_concurrent=%s\n' "$CONCURRENT"
+    printf 'part_concurrent=%s\n' "$PART_CONCURRENT"
+    printf 'parts=%s\n' "$PARTS"
+    printf 'part_size=%s\n' "$PART_SIZE"
+    printf 'bucket=%s\n' "$BUCKET"
+    printf 'sample_interval=%s\n' "$SAMPLE_INTERVAL"
+    printf 'unsafe_bypass_disk_check=%s\n' "$UNSAFE_BYPASS_DISK_CHECK"
+    if ((UNSAFE_BYPASS_DISK_CHECK)); then
+      printf 'validation_scope=%s\n' "local-same-device-approximation"
+    else
+      printf 'validation_scope=%s\n' "distinct-disk-or-prepared-data-root"
+    fi
+    printf 'uname=%s\n' "$(uname -a)"
+    printf 'docker_version=%s\n' "$(docker version --format '{{.Server.Version}}' 2>/dev/null || true)"
+    printf 'docker_info_os=%s\n' "$(docker info --format '{{.OperatingSystem}}' 2>/dev/null || true)"
+    printf 'docker_info_cgroup_driver=%s\n' "$(docker info --format '{{.CgroupDriver}}' 2>/dev/null || true)"
+    printf 'docker_info_cgroup_version=%s\n' "$(docker info --format '{{.CgroupVersion}}' 2>/dev/null || true)"
+    printf 'image_id=%s\n' "$(docker image inspect "$IMAGE" --format '{{.Id}}' 2>/dev/null || true)"
+    printf 'image_repo_digests=%s\n' "$(docker image inspect "$IMAGE" --format '{{json .RepoDigests}}' 2>/dev/null || true)"
+    printf 'cgroup_path=%s\n' "$cgroup_path"
+    printf 'data_root=%s\n' "$CASE_DATA_ROOT"
+    if command -v df >/dev/null 2>&1; then
+      printf 'data_filesystems<<EOF\n'
+      df -T "$CASE_DATA_ROOT"/rustfs{0,1,2,3} 2>/dev/null || df "$CASE_DATA_ROOT"/rustfs{0,1,2,3} 2>/dev/null || true
+      printf 'EOF\n'
+    fi
+    printf 'case_env<<EOF\n'
+    printf '%q ' "${CASE_ENV[@]}"
+    printf '\nEOF\n'
+  } >"$summary_file"
+}
+
+wait_for_health() {
+  local endpoint=$1
+  local attempts=120
+  local i
+  for ((i = 1; i <= attempts; i++)); do
+    if curl -fsSk "$endpoint/health" >/dev/null 2>&1; then
+      return 0
+    fi
+    if [[ -n ${CURRENT_CONTAINER_NAME:-} ]]; then
+      local running
+      running=$(docker inspect -f '{{.State.Running}}' "$CURRENT_CONTAINER_NAME" 2>/dev/null || printf 'false')
+      [[ $running == true ]] || return 1
+    fi
+    sleep 1
+  done
+  return 1
+}
+
+run_warp() {
+  local case_dir=$1
+  local host=$2
+  local cmd=(
+    "$WARP_BIN" multipart-put
+    --host "$host"
+    --access-key "$ACCESS_KEY"
+    --secret-key "$SECRET_KEY"
+  )
+  if ((CASE_TLS_ENABLED)); then
+    cmd+=(--tls --insecure)
+  fi
+  cmd+=(
+    --bucket "$BUCKET"
+    --concurrent "$((CLIENTS * CONCURRENT))" \
+    --part.concurrent "$PART_CONCURRENT" \
+    --parts "$PARTS" \
+    --part.size "$PART_SIZE" \
+    --duration "$DURATION" \
+    --benchdata "$case_dir/warp" \
+    --noclear \
+    --no-color
+  )
+  "${cmd[@]}" >"$case_dir/warp.log" 2>&1
+}
+
+cleanup_current_container_on_exit() {
+  if [[ -n ${CURRENT_CONTAINER_NAME:-} && ${KEEP_CONTAINERS:-0} -eq 0 ]]; then
+    docker rm -f "$CURRENT_CONTAINER_NAME" >/dev/null 2>&1 || true
+  fi
+}
+
+cleanup_container() {
+  local container_name=$1
+  if docker ps -a --format '{{.Names}}' | grep -Fxq "$container_name"; then
+    docker logs "$container_name" >"${CURRENT_CASE_DIR:-$OUT_DIR}/rustfs.log" 2>&1 || true
+    if ((KEEP_CONTAINERS)); then
+      log "keeping container $container_name"
+    else
+      docker rm -f "$container_name" >/dev/null 2>&1 || true
+    fi
+  fi
+}
+
+run_case() {
+  local raw_case=$1
+  local case_index=$2
+  local case_name
+  case_name=$(normalize_case_name "$raw_case")
+  local case_dir="$OUT_DIR/$case_name"
+  mkdir -p "$case_dir"
+  CURRENT_CASE_DIR=$case_dir
+  prepare_case_data_dirs "$case_dir"
+  resolve_tls_for_case "$case_name" "$case_dir"
+  case_env_args "$case_name"
+
+  local case_port=$((PORT + case_index))
+  local case_console_port=$((CONSOLE_PORT + case_index))
+  local container_name="${CONTAINER_PREFIX}-${case_name}"
+  CURRENT_CONTAINER_NAME=$container_name
+  cleanup_container "$container_name"
+
+  local volume_args=(
+    -v "$CASE_DATA_ROOT/rustfs0:/data/rustfs0"
+    -v "$CASE_DATA_ROOT/rustfs1:/data/rustfs1"
+    -v "$CASE_DATA_ROOT/rustfs2:/data/rustfs2"
+    -v "$CASE_DATA_ROOT/rustfs3:/data/rustfs3"
+  )
+  if ((CASE_TLS_ENABLED)); then
+    volume_args+=(-v "$CASE_TLS_DIR:/opt/tls:ro")
+  fi
+
+  local docker_args=(run -d --name "$container_name" --security-opt no-new-privileges:true)
+  [[ -n $DOCKER_PLATFORM ]] && docker_args+=(--platform "$DOCKER_PLATFORM")
+  [[ -n $MEMORY_LIMIT ]] && docker_args+=(--memory "$MEMORY_LIMIT")
+  [[ -n $CPU_LIMIT ]] && docker_args+=(--cpus "$CPU_LIMIT")
+  docker_args+=(
+    -p "127.0.0.1:${case_port}:9000"
+    -p "127.0.0.1:${case_console_port}:9001"
+    "${CASE_ENV[@]}"
+    "${volume_args[@]}"
+    "$IMAGE"
+  )
+
+  log "starting $container_name for case $case_name"
+  docker "${docker_args[@]}" >"$case_dir/container.id"
+  local pid cgroup_path endpoint
+  pid=$(container_pid "$container_name")
+  cgroup_path=$(container_cgroup_path "$pid" 2>/dev/null || true)
+  write_summary_header "$case_dir/summary.env" "$case_name" "$container_name" "$case_port" "$case_console_port" "$cgroup_path"
+
+  if ((CASE_TLS_ENABLED)); then
+    endpoint="https://127.0.0.1:${case_port}"
+  else
+    endpoint="http://127.0.0.1:${case_port}"
+  fi
+  if ! wait_for_health "$endpoint"; then
+    docker logs "$container_name" >"$case_dir/rustfs-startup-failed.log" 2>&1 || true
+    cleanup_container "$container_name"
+    fail "RustFS did not become healthy for case $case_name"
+  fi
+
+  start_sampler "$case_dir" "$container_name" "$pid" "$cgroup_path"
+  local warp_status=0
+  log "running warp for case $case_name"
+  run_warp "$case_dir" "127.0.0.1:${case_port}" || warp_status=$?
+  stop_sampler
+
+  docker inspect "$container_name" >"$case_dir/container.inspect.json" 2>/dev/null || true
+  docker logs "$container_name" >"$case_dir/rustfs.log" 2>&1 || true
+  local exit_code oom_killed running
+  exit_code=$(docker inspect -f '{{.State.ExitCode}}' "$container_name" 2>/dev/null || true)
+  oom_killed=$(docker inspect -f '{{.State.OOMKilled}}' "$container_name" 2>/dev/null || true)
+  running=$(docker inspect -f '{{.State.Running}}' "$container_name" 2>/dev/null || true)
+  {
+    printf 'warp_exit_code=%s\n' "$warp_status"
+    printf 'container_running_after_warp=%s\n' "$running"
+    printf 'container_exit_code=%s\n' "$exit_code"
+    printf 'container_oom_killed=%s\n' "$oom_killed"
+  } >>"$case_dir/summary.env"
+
+  cleanup_container "$container_name"
+  if ((warp_status != 0)); then
+    fail "warp failed for case $case_name; see $case_dir/warp.log"
+  fi
+}
+
+main() {
+  parse_args "$@"
+  require_prerequisites
+  prepare_out_dir
+  split_cases "$CASES"
+  log "artifacts: $OUT_DIR"
+  local case_index=0
+  for case_name in "${CASE_LIST[@]}"; do
+    run_case "$case_name" "$case_index"
+    case_index=$((case_index + 1))
+  done
+  log "completed #2704 UploadPart memory validation: $OUT_DIR"
+}
+
+trap cleanup_current_container_on_exit EXIT
+main "$@"
Author	SHA1	Message	Date
overtrue	6de965ae3c	test(e2e): register paginator regression in Darwin inventory	2026-10-03 07:07:41 +08:00
overtrue	7c56853067	fix(storage): publish delete rollback backups atomically Stage rollback metadata outside the rollback directory and publish it only after the full write succeeds. A short write must not leave a backup that quorum rollback can rename over acknowledged version history. Add an isolated real short-write regression and register the backported ListParts SDK test in the smoke and Linux full inventories.	2026-10-03 06:43:25 +08:00
Hauser	6f61761c7b	fix(s3): honor sparse ListParts markers and verify termination Resume part listings at the first part above the numeric marker, even when that marker is absent. Use binary search over the sorted part numbers and retain the existing exact-tail empty-slice path. Add storage, XML, and real AWS SDK paginator regressions for empty and terminal pages, sparse markers, and multipart completion integrity. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> (cherry picked from commit `673031eea1`)	2026-10-03 06:32:24 +08:00
Chapman	1bae764530	fix: emit NextPartNumberMarker only when ListParts is truncated ListPartsInfo.next_part_number_marker was a non-optional usize that defaulted to 0 and was only assigned when the response was truncated. The S3 serializer then emitted it unconditionally as Some(0), causing AWS SDK paginators to loop infinitely on part_number_marker=0 instead of terminating. Change the field to Option<usize> (None by default) and set it only inside the is_truncated branch. The S3 output layer now uses .and_then() so NextPartNumberMarker is absent when IsTruncated=false, matching AWS S3 behavior. Fixes #8208 (cherry picked from commit `44de803a38`)	2026-10-03 06:32:24 +08:00
overtrue	7150ed2066	ci: remove Docker Hub description sync	2026-10-02 19:59:28 +08:00
overtrue	c04d566b26	fix(heal): wait for held legacy responsibility in replay test	2026-10-02 03:13:15 +08:00
Hauser	4dd267940c	fix(scanner): diagnose and verify pause backlog recovery (#8293 ) * fix(scanner): diagnose and verify pause backlog recovery Expose the retained replica snapshot and claimed membership in abnormal admin status responses. Keep diagnostics off metrics updates and verify single-pool recovery and conflicting-proof preservation across 24 sets. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(scanner): move replica snapshots into diagnostics Consume the terminal admin read snapshot in a single state match and move membership, revision, and error buffers into the response. Verify buffer handoff and the unchanged JSON contract without altering ledger authority. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-10-02 01:09:00 +08:00
Hauser	2aad57c984	chore: merge upstream release branch Integrate the updated release/1.0.1 branch before pushing the dependency and atomic API maintenance changes. Preserve the upstream removal of async-recursion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com>	2026-10-02 01:00:04 +08:00
Hauser	dc79582e40	improve	2026-10-02 00:54:32 +08:00
Hauser	25fb3d2516	chore: refresh dependencies and atomic update APIs Update workspace dependencies and the lockfile. Replace deprecated atomic fetch_update aliases with try_update while preserving closures and memory ordering. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com>	2026-10-02 00:46:57 +08:00
Hauser	5c39ed1495	fix(s3): queue bucket operations and restore strict Clippy checks (#8290 ) * fix(s3): queue concurrent bucket creation and deletion Keep eight active bucket transactions and bound admission waiting to 128 requests and 30 seconds. Preserve detached transaction ownership and return Retry-After with overload responses. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): restore strict Clippy compatibility on Rust 1.99 Use try_update without changing atomic ordering or overflow behavior. Keep recursive storage futures boxed once at each frame and remove the redundant async-recursion macro, including its non-recursive SQL planner use. Remove needless closure borrows and orphaned dependency entries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-10-02 00:41:05 +08:00
overtrue	da31b53dd6	fix(scanner): pass fence store to checkpoint fixture	2026-10-01 19:52:48 +08:00
Hauser	a6bee557e9	fix: add UploadPart OOM validation guardrails (#8287 ) * docs(release): validate candidates on release branch * fix(scanner): validate checkpoints against global cycle fence (#8278) ## Related Issues Related to rustfs/backlog#2701. ## Summary of Changes Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission. ## Verification Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance. ## Impact Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired. ## Additional Notes Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior. * fix(ci): restore E2E membership and pagination timeouts (#8281) * ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279) ## Related Issues Follow-up to #8229. ## Summary of Changes Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence. ## Verification The exact PR head `b66129ab9f` passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate. ## Impact Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates. ## Additional Notes Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup. * fix: add UploadPart OOM validation guardrails Add a Docker validation harness for backlog#2704 so the ordinary UploadPart low-concurrency memory workload can be reproduced with comparable case metadata, process/cgroup sampling, TLS and metrics toggles, cache-env controls, and write reclaim/direct-write experiments. Warn when operators set the unrecognized RUSTFS_OBJECT_CACHE_* variables that appeared in the reporter compose file. The variables are reported but remain ignored, so startup does not silently change object data cache behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: overtrue <anzhengchao@gmail.com> Co-authored-by: AL <allan.bednarowski@gmail.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-10-01 19:32:52 +08:00
Hauser	7b76c655a9	Gate multipart copy through write admission (#8283 ) Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-10-01 19:27:07 +08:00
Hauser	09a5c7b372	[release/1.0.1] Gate multipart copy through write admission (#8284 ) Gate multipart copy through write admission Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-10-01 16:06:03 +08:00
overtrue	f37df93df5	fix(test): await HTTP sender readiness in Top RPC fixture	2026-10-01 09:51:54 +08:00
overtrue	4816453cd5	fix(usage): combine identical snapshot retention branches	2026-10-01 08:06:04 +08:00
Hauser	f715e88f8a	fix(obs): distinguish allocator counters from live memory (#8274 ) * fix(obs): distinguish allocator counters from live memory Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(obs): parse allocator statistics from one node Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate late parity recovery from metadata hedges Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-10-01 07:57:22 +08:00
Hauser	6c06b4fe16	test(scanner): verify checkpoint takeover and repair dispatch (#8275 ) * test(scanner): cover checkpoint handoff and repair dispatch Drive runtime budget expiry, partial-cycle persistence, leadership claims, and stale checkpoint rejection between real disk-backed fixture scans. Verify that a metadata repair beyond the first bounded prefix is saved in the scanner ledger and dispatched by the MRF consumer. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: isolate scanner fixtures and refresh full e2e membership Reserve nextest capacity for the real-disk scanner publication and MRF admission fixtures. Bind both platform membership checks to the reviewed pagination deadline test added on main. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(scanner): consolidate checkpoint fixture lifecycle Keep one durable control store across timeout and leadership transitions, and inject generation advancement into the shared checkpoint scenario. Check the actual saved metadata path so late-write rejection also proves that existing checkpoint bytes remain intact. Centralize MRF fixture isolation and reuse nextest process isolation when the startup environment already satisfies the test contract. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-10-01 07:28:37 +08:00
Hauser	49462ae5d4	fix(usage): recover historical counters with generation fencing (#8273 ) * fix(usage): recover historical counters during continued writes Use newer converged scanner snapshots to reconcile stale absolute usage baselines while preserving concurrent mutation and expiry receipt fences. Cover durable publication, admin and quota reads, and legacy generations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(usage): fence snapshots and move preserved cache entries Apply the cached scanner generation floor before every reconciliation path and retain it even when an older snapshot happens to match core counts. Move preserved usage entries instead of cloning their histogram maps under the cache lock, retaining expiry receipt identity and cancellation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-10-01 06:32:34 +08:00
overtrue	4b50b98d22	fix(ci): locate security chain scripts in the workspace	2026-10-01 06:12:54 +08:00
overtrue	bcf83d4e41	fix(test): supply CPU workload for sampler regression	2026-10-01 04:59:54 +08:00
overtrue	65ce936131	test(e2e): retain startup and shutdown failure diagnostics	2026-10-01 01:46:41 +08:00
Peder Bergan	e4d1ade702	fix(usage): reconcile stale counters after lifecycle expiration (#8108 ) * fix(usage): reconcile stale counters after lifecycle expiration * fix(usage): account lifecycle expiry during continuous writes * test(usage): run lifecycle usage scenarios on one scanner store * test(usage): use a Windows-representable pre-mutation offset * fix(usage): harden expiry accounting recovery and quota checks Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(usage): recover stale persisted counts through the scanner Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test. Refs rustfs/backlog#2689 Refs rustfs/backlog#2691 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): bound delimiter pagination fixture concurrency The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(upgrade): establish a persisted previous-release baseline Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions. Refs rustfs/backlog#2689 Refs rustfs/backlog#2384 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): bound cancellation metadata persistence waits Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions. Refs rustfs/backlog#2697 Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-09-30 22:33:00 +08:00
Hauser	86fc35b49f	fix(scanner): expose pause backlog replica diagnostics (#8258 ) * fix(scanner): expose pause backlog replica diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(connect): stabilize runtime profile lease cancellation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): tolerate delayed schedule startup in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retry quota reads during usage warmup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid meta-bucket incarnation self-deadlock Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(test): use persisted incarnation in heal fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-09-30 22:25:31 +08:00
overtrue	b54b32b8a9	fix(deps): replace yanked yoke-derive release	2026-09-30 22:24:45 +08:00
overtrue	aff1c025af	fix(ci): include pagination regression in full E2E selection	2026-09-30 22:03:46 +08:00