test(e2e): rename stall timing variable flagged by typos

Add ListObjectsV2 paging, Range GET/HEAD, unversioned buckets, standard
and user metadata replay, ResponseStatus/TruncateBodyAt/Stall fault
actions, Range/User-Agent/prefix/continuation-token journal fields,
count_requests, direct seeding, and a configurable object cap to the
programmable fake S3 target, and add the on_demand_migration e2e
harness (OdmTestEnv, admin wrappers, source seeding, local-state
assertions, second RustFS source) with its self-test.

.agents/skills/arch-checks/SKILL.md

@@ -50,10 +50,11 @@ consider adding it to the script's `checked_files` list.
 
 ## `check_doc_paths.sh`
 
-Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
-`docs/architecture/*.md`) must not reference repo file paths that no longer
-exist. If your refactor moved code, update the docs that point at it — the
-error message lists `doc -> stale-path` pairs.
+Instruction docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`) and every
+Markdown file under `docs/` (architecture, operations, testing, index) must not
+reference repo file paths that no longer exist. If your refactor moved code,
+update the docs that point at it — the error message lists `doc -> stale-path`
+pairs. Cite paths plus symbol names, never line numbers (see `docs/README.md`).
 
 ## `check_no_planning_docs.sh`
 

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=ef914ec0b8daa9c2c5e52f501d339914662f42d6f6ed9d33877d56b97adf16f9
+sha256-darwin=9dccb0cd537cf79ae70c1c20e8281d36d03f2f09f81142a5341e26e3dc18709d
 sha256-linux=a8a816d7bb0e7cb5632b1863b33794bcb9fc7e765f150aa5e1bf16518e28dfb4

.config/e2e-nightly-selection.txt

@@ -1 +1 @@
-sha256=51da41c54167602f2bd6c45921b39a44562bf3cfcdf468d992bb992c62cad7fd
+sha256=d06524b44de97ed8f62b0fd8cf9fa504e3cd520ffcaacc32691d6f890ebe7f20

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=dbebfbab9b9efd4eff31211e69dd32235dc00e207f2ab0dd919a1b2ac9e724c2
+sha256=db9bd8cdcb0abe43461aa6b36499b17cabd4098e5b34e300b1a0f0d0f34d9884

.config/nextest.toml

@@ -355,7 +355,8 @@ test-group = 'ecstore-serial-flaky'
 # allowlist", so any new replication test lands in nightly by default (never
 # silently unrun) until it is explicitly blessed as fast here. Keep the two
 # regexes byte-identical. The committed profile selection digests make changes
-# visible in CI; current counts live in docs/testing/e2e-suite-inventory.md.
+# visible in CI; list current membership with `cargo nextest list -p e2e_test
+# --profile <profile>` (platform-dependent; see docs/testing/README.md).
 # HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
 # (#4724) because they set a loopback (127.0.0.1) replication target that the
 # SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
@@ -508,7 +509,7 @@ path = "junit.xml"
 # quota, checksum, encryption,
 # security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
 # skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
-# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
+# --profile e2e-full -p e2e_test` (platform-dependent; see docs/testing/README.md).
 #
 # The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
 #   * protocols:: — FTPS/SFTP/WebDAV, run from the dedicated protocol profile

.github/workflows/build.yml

@@ -1065,7 +1065,7 @@ jobs:
           while IFS= read -r preview_tag; do
             [[ -n "$preview_tag" ]] || continue
             echo "🧹 Deleting preview release $preview_tag (tag kept)"
-            gh release delete "$preview_tag" --yes
+            gh release delete "$preview_tag" --repo "${GITHUB_REPOSITORY}" --yes
             DELETED=$((DELETED + 1))
           done < <(
             jq -r --arg tag "$TAG" '

.github/workflows/rustfs-functional-chain.yml

@@ -12,10 +12,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Functional chain driver: runs the nine functional suites in a fixed order
-# (upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security, with
-# performance on its own runner in parallel) and guarantees the chain keeps
-# moving even when individual suites fail.
+# Functional chain driver: runs the ten functional suites in a fixed order
+# (upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security ->
+# replication, with performance on its own runner in parallel) and guarantees
+# the chain keeps moving even when individual suites fail.
 #
 # Each suite workflow can still be dispatched standalone (workflow_dispatch);
 # only chain-triggered runs forward to the next suite via repository_dispatch,

.github/workflows/rustfs-heal-test.yml

@@ -293,7 +293,7 @@ jobs:
           GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
         run: |
           set -uo pipefail
-          if [ -z "${{GH_TOKEN:-}}" ]; then
+          if [ -z "${GH_TOKEN:-}" ]; then
             echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
             exit 1
           fi
@@ -302,31 +302,31 @@ jobs:
             if gh api --method POST repos/rustfs/rustfs/dispatches \
               -f event_type='rustfs-chain-pool' \
               -F 'client_payload[from_suite]=heal'; then
-              echo "dispatched next suite Pool expansion (attempt ${{attempt}})"
+              echo "dispatched next suite Pool expansion (attempt ${attempt})"
               DISPATCHED=1
               break
             fi
-            echo "dispatch attempt ${{attempt}} failed; retrying in ${{attempt}}0s" >&2
-            sleep "${{attempt}}0"
+            echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
+            sleep "${attempt}0"
           done
-          if [ "${{DISPATCHED:-0}}" -ne 1 ]; then
+          if [ "${DISPATCHED:-0}" -ne 1 ]; then
             echo "ERROR: functional chain stalled: could not dispatch Pool expansion after 3 attempts" >&2
-            TITLE="[functional][chain] stalled after heal (run ${{GITHUB_RUN_ID}})"
+            TITLE="[functional][chain] stalled after heal (run ${GITHUB_RUN_ID})"
             BODY_FILE="$(mktemp)"
             {
               echo "The functional chain could not hand off from **heal** to **Pool expansion** after 3 attempts."
               echo ""
-              echo "- Failed suite job: ${{GITHUB_SERVER_URL}}/${{GITHUB_REPOSITORY}}/actions/runs/${{GITHUB_RUN_ID}}"
-              echo "- Expected next event: `rustfs-chain-pool`"
+              echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+              echo "- Expected next event: 'rustfs-chain-pool'"
               echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
               echo "- Recovery: re-dispatch manually with"
-              echo "  ```"
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
               echo "  gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-pool'"
-              echo "  ```"
-            } > "${{BODY_FILE}}"
-            gh issue create -R rustfs/backlog --title "${{TITLE}}" \
-              --body-file "${{BODY_FILE}}" --label functional-test \
-              || gh issue create -R rustfs/backlog --title "${{TITLE}}" --body-file "${{BODY_FILE}}" \
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
+            } > "${BODY_FILE}"
+            gh issue create -R rustfs/backlog --title "${TITLE}" \
+              --body-file "${BODY_FILE}" --label functional-test \
+              || gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
               || echo "could not file the stall alert issue either; check the token" >&2
             exit 1
           fi

.github/workflows/rustfs-kms-test.yml

@@ -349,7 +349,7 @@ jobs:
           GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
         run: |
           set -uo pipefail
-          if [ -z "${{GH_TOKEN:-}}" ]; then
+          if [ -z "${GH_TOKEN:-}" ]; then
             echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
             exit 1
           fi
@@ -358,31 +358,31 @@ jobs:
             if gh api --method POST repos/rustfs/rustfs/dispatches \
               -f event_type='rustfs-chain-tier' \
               -F 'client_payload[from_suite]=kms'; then
-              echo "dispatched next suite Tier (attempt ${{attempt}})"
+              echo "dispatched next suite Tier (attempt ${attempt})"
               DISPATCHED=1
               break
             fi
-            echo "dispatch attempt ${{attempt}} failed; retrying in ${{attempt}}0s" >&2
-            sleep "${{attempt}}0"
+            echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
+            sleep "${attempt}0"
           done
-          if [ "${{DISPATCHED:-0}}" -ne 1 ]; then
+          if [ "${DISPATCHED:-0}" -ne 1 ]; then
             echo "ERROR: functional chain stalled: could not dispatch Tier after 3 attempts" >&2
-            TITLE="[functional][chain] stalled after kms (run ${{GITHUB_RUN_ID}})"
+            TITLE="[functional][chain] stalled after kms (run ${GITHUB_RUN_ID})"
             BODY_FILE="$(mktemp)"
             {
               echo "The functional chain could not hand off from **kms** to **Tier** after 3 attempts."
               echo ""
-              echo "- Failed suite job: ${{GITHUB_SERVER_URL}}/${{GITHUB_REPOSITORY}}/actions/runs/${{GITHUB_RUN_ID}}"
-              echo "- Expected next event: `rustfs-chain-tier`"
+              echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+              echo "- Expected next event: 'rustfs-chain-tier'"
               echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
               echo "- Recovery: re-dispatch manually with"
-              echo "  ```"
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
               echo "  gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-tier'"
-              echo "  ```"
-            } > "${{BODY_FILE}}"
-            gh issue create -R rustfs/backlog --title "${{TITLE}}" \
-              --body-file "${{BODY_FILE}}" --label functional-test \
-              || gh issue create -R rustfs/backlog --title "${{TITLE}}" --body-file "${{BODY_FILE}}" \
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
+            } > "${BODY_FILE}"
+            gh issue create -R rustfs/backlog --title "${TITLE}" \
+              --body-file "${BODY_FILE}" --label functional-test \
+              || gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
               || echo "could not file the stall alert issue either; check the token" >&2
             exit 1
           fi

.github/workflows/rustfs-pool-expand-test.yml

@@ -605,7 +605,7 @@ jobs:
           GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
         run: |
           set -uo pipefail
-          if [ -z "${{GH_TOKEN:-}}" ]; then
+          if [ -z "${GH_TOKEN:-}" ]; then
             echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
             exit 1
           fi
@@ -614,31 +614,31 @@ jobs:
             if gh api --method POST repos/rustfs/rustfs/dispatches \
               -f event_type='rustfs-chain-security' \
               -F 'client_payload[from_suite]=pool'; then
-              echo "dispatched next suite Security (attempt ${{attempt}})"
+              echo "dispatched next suite Security (attempt ${attempt})"
               DISPATCHED=1
               break
             fi
-            echo "dispatch attempt ${{attempt}} failed; retrying in ${{attempt}}0s" >&2
-            sleep "${{attempt}}0"
+            echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
+            sleep "${attempt}0"
           done
-          if [ "${{DISPATCHED:-0}}" -ne 1 ]; then
+          if [ "${DISPATCHED:-0}" -ne 1 ]; then
             echo "ERROR: functional chain stalled: could not dispatch Security after 3 attempts" >&2
-            TITLE="[functional][chain] stalled after pool (run ${{GITHUB_RUN_ID}})"
+            TITLE="[functional][chain] stalled after pool (run ${GITHUB_RUN_ID})"
             BODY_FILE="$(mktemp)"
             {
               echo "The functional chain could not hand off from **pool** to **Security** after 3 attempts."
               echo ""
-              echo "- Failed suite job: ${{GITHUB_SERVER_URL}}/${{GITHUB_REPOSITORY}}/actions/runs/${{GITHUB_RUN_ID}}"
-              echo "- Expected next event: `rustfs-chain-security`"
+              echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+              echo "- Expected next event: 'rustfs-chain-security'"
               echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
               echo "- Recovery: re-dispatch manually with"
-              echo "  ```"
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
               echo "  gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-security'"
-              echo "  ```"
-            } > "${{BODY_FILE}}"
-            gh issue create -R rustfs/backlog --title "${{TITLE}}" \
-              --body-file "${{BODY_FILE}}" --label functional-test \
-              || gh issue create -R rustfs/backlog --title "${{TITLE}}" --body-file "${{BODY_FILE}}" \
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
+            } > "${BODY_FILE}"
+            gh issue create -R rustfs/backlog --title "${TITLE}" \
+              --body-file "${BODY_FILE}" --label functional-test \
+              || gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
               || echo "could not file the stall alert issue either; check the token" >&2
             exit 1
           fi

.github/workflows/rustfs-replication-test.yml

@@ -0,0 +1,365 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: RustFS Replication Test
+
+on:
+  workflow_dispatch:
+    inputs:
+      rustfs_version:
+        description: 'RustFS release tag to test (e.g. 1.0.0-rc.4-preview.1)'
+        required: false
+        default: '1.0.0-rc.4-preview.1'
+      package_url:
+        description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.'
+        required: false
+        type: string
+      suite:
+        description: 'Suite to run (all = bucket REP-* then site SITE-*)'
+        type: choice
+        options:
+          - all
+          - bucket
+          - site
+        default: all
+  repository_dispatch:
+    # Chain handoff: dispatched when the security suite finishes. This is the
+    # last link of the functional chain.
+    types: [rustfs-chain-replication]
+
+permissions:
+  contents: read
+
+# The replication suite uses the same shared VMs as the other functional
+# tests, so it must serialize with them instead of running in parallel.
+concurrency:
+  group: rustfs-shared-functional-tests
+  cancel-in-progress: false
+
+defaults:
+  run:
+    shell: bash
+
+env:
+  RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
+  RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
+  RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
+  RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
+  RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
+  PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+
+jobs:
+  replication-test:
+    runs-on: smoke-testing
+    # A failed replication run must not break the chain or the workflow: the
+    # failure is reported to rustfs/backlog instead (see the issue step).
+    continue-on-error: true
+    timeout-minutes: 360
+    if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
+    steps:
+      # auto-testing is private: clone it with the dedicated PF token (not
+      # GITHUB_TOKEN) and retry transient GitHub/network failures.
+      - name: Checkout auto-testing scripts (with retry)
+        env:
+          GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+        run: |
+          set -euo pipefail
+          rm -rf auto-testing
+          for attempt in 1 2 3 4 5; do
+            if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then
+              echo "auto-testing cloned (attempt ${attempt})"
+              exit 0
+            fi
+            rm -rf auto-testing
+            echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2
+            sleep $((attempt * 15))
+          done
+          echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2
+          exit 1
+
+      - name: Show environment
+        run: |
+          uname -a
+          jq --version
+          openssl version
+          aws --version
+          df -h /data | tail -1 || true
+
+      - name: Cleanup environment (before)
+        run: |
+          set -euo pipefail
+          read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
+          SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
+          for node in "${NODES[@]}"; do
+            ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${SSH_USER}@${node}" '
+              set -euo pipefail
+              SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
+              ${SUDO} systemctl stop rustfs rustfs-rep2 2>/dev/null || true
+              if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
+                ${SUDO} dpkg -P rustfs
+              fi
+              for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
+              ${SUDO} rm -rf /data/rustfs-rep2 /var/log/rustfs /var/log/rustfs-rep2 /var/lib/rustfs/kms
+            '
+          done
+
+      - name: Run replication suite
+        id: test
+        continue-on-error: true
+        env:
+          LOG_FILE: /tmp/rustfs-replication.log
+        run: |
+          set -euo pipefail
+          chmod +x auto-testing/rustfs-replication-test.sh
+          PACKAGE_URL='${{ inputs.package_url }}'
+          RUSTFS_VERSION='${{ inputs.rustfs_version }}'
+          SUITE='${{ inputs.suite }}'
+          ARGS=(-y --log-file "${LOG_FILE}")
+          if [ "${SUITE}" = "all" ] || [ -z "${SUITE}" ] || [ "${SUITE}" = "null" ]; then
+            ARGS+=(--suite all)
+          else
+            ARGS+=(--suite "${SUITE}")
+          fi
+          if [ -n "${PACKAGE_URL}" ]; then
+            ARGS+=(--package-url "${PACKAGE_URL}")
+          elif [ -n "${RUSTFS_VERSION}" ] && [ "${RUSTFS_VERSION}" != "null" ]; then
+            ARGS+=(--version "${RUSTFS_VERSION}")
+          else
+            ARGS+=(--package-url "${RUSTFS_NIGHTLY_PACKAGE_URL}")
+          fi
+          ./auto-testing/rustfs-replication-test.sh "${ARGS[@]}"
+
+      - name: Generate report
+        if: always()
+        env:
+          LOG_FILE: /tmp/rustfs-replication.log
+          REPORT_FILE: /tmp/rustfs-replication-report.md
+        run: |
+          set -euo pipefail
+          PACKAGE_URL='${{ inputs.package_url }}'
+          RUSTFS_VERSION='${{ inputs.rustfs_version }}'
+          if [ -n "${PACKAGE_URL}" ]; then
+            PACKAGE_SOURCE="${PACKAGE_URL}"
+          elif [ -n "${RUSTFS_VERSION}" ] && [ "${RUSTFS_VERSION}" != "null" ]; then
+            PACKAGE_SOURCE="version ${RUSTFS_VERSION}"
+          else
+            PACKAGE_SOURCE="${RUSTFS_NIGHTLY_PACKAGE_URL}"
+          fi
+          read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
+          SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
+          RUSTFS_VERSION_INFO="N/A"
+          if [ "${#NODES[@]}" -gt 0 ]; then
+            DETECTED_VERSION="$(ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new \
+              "${SSH_USER}@${NODES[0]}" 'rustfs --version' 2>/dev/null | tr -d '\r' | head -n 1 || true)"
+            if [ -n "${DETECTED_VERSION}" ]; then
+              RUSTFS_VERSION_INFO="${DETECTED_VERSION}"
+            fi
+          fi
+          CASE_TABLE="/tmp/rustfs-replication-cases.md"
+          python3 - "${LOG_FILE}" "${CASE_TABLE}" <<'PY'
+          import re
+          import sys
+
+          log_file, out_file = sys.argv[1], sys.argv[2]
+          ansi = re.compile(r'\x1b\[[0-9;]*m')
+          start_re = re.compile(r'^---\s+([A-Z0-9]+-[0-9]+)\s+(.+?)\s+---$')
+          done_re = re.compile(r'^\[(PASS|FAIL|UNSUPPORTED)\]\s+([A-Z0-9]+-[0-9]+)\b')
+
+          rows = []
+          index = {}
+          try:
+              with open(log_file, 'r', encoding='utf-8', errors='replace') as fh:
+                  for raw in fh:
+                      line = ansi.sub('', raw).strip()
+                      m = start_re.match(line)
+                      if m:
+                          case_id, name = m.group(1), m.group(2)
+                          if case_id not in index:
+                              index[case_id] = len(rows)
+                              rows.append([case_id, name, 'RUNNING'])
+                          continue
+                      m = done_re.match(line)
+                      if m:
+                          status, case_id = m.group(1), m.group(2)
+                          if case_id in index:
+                              rows[index[case_id]][2] = status
+                          else:
+                              rows.append([case_id, case_id, status])
+                              index[case_id] = len(rows) - 1
+          except FileNotFoundError:
+              rows = []
+
+          counts = {'PASS': 0, 'FAIL': 0, 'UNSUPPORTED': 0, 'RUNNING': 0}
+          for _, _, status in rows:
+              counts[status] = counts.get(status, 0) + 1
+
+          with open(out_file, 'w', encoding='utf-8') as out:
+              out.write('## Case Summary\n\n')
+              out.write(f"- Total: {len(rows)}\\n")
+              out.write(f"- PASS: {counts.get('PASS', 0)}\\n")
+              out.write(f"- FAIL: {counts.get('FAIL', 0)}\\n")
+              out.write(f"- UNSUPPORTED: {counts.get('UNSUPPORTED', 0)}\\n")
+              out.write('\\n')
+              out.write('| Case | Name | Status |\\n')
+              out.write('| --- | --- | --- |\\n')
+              for case_id, name, status in rows:
+                  out.write(f'| {case_id} | {name} | {status} |\\n')
+          PY
+          {
+            echo "# RustFS replication test report"
+            echo ""
+            echo "- Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+            echo "- Trigger: ${{ github.event_name }}"
+            echo "- Package: ${PACKAGE_SOURCE}"
+            echo "- RustFS Version: ${RUSTFS_VERSION_INFO}"
+            echo "- Test Step Outcome: ${{ steps.test.outcome }}"
+            echo ""
+            cat "${CASE_TABLE}" || true
+            echo ""
+            echo "## Log tail"
+            echo '```text'
+            tail -n 200 "${LOG_FILE}" || true
+            echo '```'
+          } | tee "${REPORT_FILE}"
+          cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
+
+      - name: Upload functional report to dashboard
+        if: always()
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
+          REPORT_FILE: /tmp/rustfs-replication-report.md
+          SUITE: replication
+        run: |
+          set -euo pipefail
+          if [ -z "${GH_TOKEN:-}" ]; then
+            echo "PF_TESTING_GH_TOKEN is not configured; skipping dashboard upload"
+            exit 0
+          fi
+          DATE="$(date -u +%Y-%m-%d)"
+          REPORT_PATH="functional-reports/${SUITE}/${DATE}.md"
+          CONTENT="$(python3 -c 'import base64,sys;print(base64.b64encode(open(sys.argv[1],"rb").read()).decode())' "${REPORT_FILE}")"
+          SHA="$(gh api "repos/rustfs/dashboard/contents/${REPORT_PATH}" -q '.sha' 2>/dev/null || true)"
+          if [ -n "${SHA}" ]; then
+            jq -n --arg msg "report(${SUITE}): ${DATE}" --arg content "${CONTENT}" --arg sha "${SHA}" \
+              '{message:$msg, content:$content, sha:$sha}' \
+              | gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
+          else
+            jq -n --arg msg "report(${SUITE}): ${DATE}" --arg content "${CONTENT}" \
+              '{message:$msg, content:$content}' \
+              | gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
+          fi
+
+      - name: File failure issue in rustfs/backlog
+        if: ${{ always() && (failure() || steps.test.outcome == 'failure' || steps.test.outcome == 'cancelled') }}
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+          SUITE: 'replication'
+          SUITE_LABEL: 'Replication (bucket + site)'
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+          REPORT_FILE: '/tmp/rustfs-replication-report.md'
+          LOG_FILE: '/tmp/rustfs-replication.log'
+        run: |
+          set -euo pipefail
+          if [ -z "${GH_TOKEN:-}" ]; then
+            echo "PF_TESTING_GH_TOKEN is not configured; skipping backlog issue"
+            exit 0
+          fi
+          TITLE="[functional][${SUITE}] ${SUITE_LABEL} suite failed (run ${GITHUB_RUN_ID})"
+          EXISTING="$(gh issue list -R rustfs/backlog --state all \
+            --search "in:title \"run ${GITHUB_RUN_ID}\"" \
+            --json number --jq '.[].number' || true)"
+          if [ -n "${EXISTING}" ]; then
+            echo "backlog issue already exists for run ${GITHUB_RUN_ID}; skipping"
+            exit 0
+          fi
+          redact() {
+            sed -E \
+              -e 's/(RUSTFS_(ACCESS_KEY|SECRET_KEY)[=: ]+)[^[:space:]]+/\1[REDACTED]/Ig' \
+              -e 's/(Authorization:).*/\1 [REDACTED]/Ig' \
+              -e 's/(X-Amz-Signature=)[^&[:space:]]+/\1[REDACTED]/Ig' \
+              -e 's/^.*(password|secret|token)[=: ].*/[REDACTED SENSITIVE LINE]/Ig'
+          }
+          BODY_FILE="$(mktemp)"
+          {
+            echo "The **${SUITE_LABEL}** functional suite failed."
+            echo ""
+            echo "- Suite: \`${SUITE}\`"
+            echo "- Run: ${RUN_URL}"
+            echo "- Trigger: ${GITHUB_EVENT_NAME}"
+            echo "- Date: $(date -u +%Y-%m-%d)"
+            echo ""
+            echo "## Report (errors and symptoms)"
+            echo ""
+            if [ -s "${REPORT_FILE}" ]; then
+              redact < "${REPORT_FILE}"
+            elif [ -s "${LOG_FILE:-}" ]; then
+              echo "(report file missing; log tail below)"
+              echo ""
+              tail -n 200 "${LOG_FILE}" | redact
+            else
+              echo "(no report or log file was produced)"
+            fi
+          } | head -c 55000 > "${BODY_FILE}"
+          gh label create functional-test -R rustfs/backlog --color d73a4a 2>/dev/null || true
+          if ! gh issue create -R rustfs/backlog --title "${TITLE}" \
+              --body-file "${BODY_FILE}" --label functional-test; then
+            gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}"
+          fi
+          echo "filed backlog issue for suite ${SUITE}"
+
+      - name: Upload report and logs
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: rustfs-replication-${{ github.run_id }}
+          path: |
+            /tmp/rustfs-replication.log
+            /tmp/rustfs-replication-report.md
+          if-no-files-found: warn
+
+      - name: Cleanup environment (after)
+        if: always()
+        run: |
+          set -euo pipefail
+          read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
+          SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
+          for node in "${NODES[@]}"; do
+            ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${SSH_USER}@${node}" '
+              set -euo pipefail
+              SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
+              ${SUDO} systemctl stop rustfs rustfs-rep2 2>/dev/null || true
+              if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
+                ${SUDO} dpkg -P rustfs
+              fi
+              for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
+              ${SUDO} rm -rf /data/rustfs-rep2 /var/log/rustfs /var/log/rustfs-rep2
+            '
+          done
+
+      - name: Chain complete
+        # Replication is the last link of the functional chain: nothing to
+        # dispatch after it. This step just records that the chain finished.
+        if: ${{ always() && github.event_name == 'repository_dispatch' }}
+        run: |
+          echo "Functional chain complete: replication (final suite) finished."
+          echo "from_suite=security trigger=${{ github.event_name }} outcome=${{ steps.test.outcome }}"
+
+      - name: Notify on failure
+        if: failure()
+        run: |
+          echo "RustFS replication suite failed"
+          echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}"
+          echo "See the uploaded report and log artifacts for details."

.github/workflows/rustfs-s3-compat-test.yml

@@ -329,7 +329,7 @@ jobs:
           GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
         run: |
           set -uo pipefail
-          if [ -z "${{GH_TOKEN:-}}" ]; then
+          if [ -z "${GH_TOKEN:-}" ]; then
             echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
             exit 1
           fi
@@ -338,31 +338,31 @@ jobs:
             if gh api --method POST repos/rustfs/rustfs/dispatches \
               -f event_type='rustfs-chain-kms' \
               -F 'client_payload[from_suite]=s3'; then
-              echo "dispatched next suite KMS (attempt ${{attempt}})"
+              echo "dispatched next suite KMS (attempt ${attempt})"
               DISPATCHED=1
               break
             fi
-            echo "dispatch attempt ${{attempt}} failed; retrying in ${{attempt}}0s" >&2
-            sleep "${{attempt}}0"
+            echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
+            sleep "${attempt}0"
           done
-          if [ "${{DISPATCHED:-0}}" -ne 1 ]; then
+          if [ "${DISPATCHED:-0}" -ne 1 ]; then
             echo "ERROR: functional chain stalled: could not dispatch KMS after 3 attempts" >&2
-            TITLE="[functional][chain] stalled after s3 (run ${{GITHUB_RUN_ID}})"
+            TITLE="[functional][chain] stalled after s3 (run ${GITHUB_RUN_ID})"
             BODY_FILE="$(mktemp)"
             {
               echo "The functional chain could not hand off from **s3** to **KMS** after 3 attempts."
               echo ""
-              echo "- Failed suite job: ${{GITHUB_SERVER_URL}}/${{GITHUB_REPOSITORY}}/actions/runs/${{GITHUB_RUN_ID}}"
-              echo "- Expected next event: `rustfs-chain-kms`"
+              echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+              echo "- Expected next event: 'rustfs-chain-kms'"
               echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
               echo "- Recovery: re-dispatch manually with"
-              echo "  ```"
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
               echo "  gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-kms'"
-              echo "  ```"
-            } > "${{BODY_FILE}}"
-            gh issue create -R rustfs/backlog --title "${{TITLE}}" \
-              --body-file "${{BODY_FILE}}" --label functional-test \
-              || gh issue create -R rustfs/backlog --title "${{TITLE}}" --body-file "${{BODY_FILE}}" \
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
+            } > "${BODY_FILE}"
+            gh issue create -R rustfs/backlog --title "${TITLE}" \
+              --body-file "${BODY_FILE}" --label functional-test \
+              || gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
               || echo "could not file the stall alert issue either; check the token" >&2
             exit 1
           fi

.github/workflows/rustfs-security-test.yml

@@ -47,7 +47,7 @@ on:
         type: boolean
         default: true
   repository_dispatch:
-    # Chain handoff: dispatched when the pool expansion suite finishes (last link).
+    # Chain handoff: dispatched when the pool expansion suite finishes.
     types: [rustfs-chain-security]
 
 permissions:
@@ -292,6 +292,24 @@ jobs:
             '
           done
 
+      - name: "Continue functional chain (next: Replication)"
+        # Only chain-triggered runs forward to the next suite; standalone
+        # workflow_dispatch runs stop after their own cleanup.
+        if: ${{ always() && github.event_name == 'repository_dispatch' }}
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${GH_TOKEN:-}" ]; then
+            echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
+            exit 1
+          fi
+          echo "Dispatching next functional suite: Replication"
+          gh api --method POST repos/rustfs/rustfs/dispatches \
+            -f event_type='rustfs-chain-replication' \
+            -F 'client_payload[from_suite]=security'
+
       - name: Notify on failure
         if: failure()
         run: |

.github/workflows/rustfs-storage-test.yml

@@ -344,7 +344,7 @@ jobs:
           GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
         run: |
           set -uo pipefail
-          if [ -z "${{GH_TOKEN:-}}" ]; then
+          if [ -z "${GH_TOKEN:-}" ]; then
             echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
             exit 1
           fi
@@ -353,31 +353,31 @@ jobs:
             if gh api --method POST repos/rustfs/rustfs/dispatches \
               -f event_type='rustfs-chain-heal' \
               -F 'client_payload[from_suite]=storage'; then
-              echo "dispatched next suite Heal (attempt ${{attempt}})"
+              echo "dispatched next suite Heal (attempt ${attempt})"
               DISPATCHED=1
               break
             fi
-            echo "dispatch attempt ${{attempt}} failed; retrying in ${{attempt}}0s" >&2
-            sleep "${{attempt}}0"
+            echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
+            sleep "${attempt}0"
           done
-          if [ "${{DISPATCHED:-0}}" -ne 1 ]; then
+          if [ "${DISPATCHED:-0}" -ne 1 ]; then
             echo "ERROR: functional chain stalled: could not dispatch Heal after 3 attempts" >&2
-            TITLE="[functional][chain] stalled after storage (run ${{GITHUB_RUN_ID}})"
+            TITLE="[functional][chain] stalled after storage (run ${GITHUB_RUN_ID})"
             BODY_FILE="$(mktemp)"
             {
               echo "The functional chain could not hand off from **storage** to **Heal** after 3 attempts."
               echo ""
-              echo "- Failed suite job: ${{GITHUB_SERVER_URL}}/${{GITHUB_REPOSITORY}}/actions/runs/${{GITHUB_RUN_ID}}"
-              echo "- Expected next event: `rustfs-chain-heal`"
+              echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+              echo "- Expected next event: 'rustfs-chain-heal'"
               echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
               echo "- Recovery: re-dispatch manually with"
-              echo "  ```"
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
               echo "  gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-heal'"
-              echo "  ```"
-            } > "${{BODY_FILE}}"
-            gh issue create -R rustfs/backlog --title "${{TITLE}}" \
-              --body-file "${{BODY_FILE}}" --label functional-test \
-              || gh issue create -R rustfs/backlog --title "${{TITLE}}" --body-file "${{BODY_FILE}}" \
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
+            } > "${BODY_FILE}"
+            gh issue create -R rustfs/backlog --title "${TITLE}" \
+              --body-file "${BODY_FILE}" --label functional-test \
+              || gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
               || echo "could not file the stall alert issue either; check the token" >&2
             exit 1
           fi

.github/workflows/rustfs-tier-test.yml

@@ -439,7 +439,7 @@ jobs:
           GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
         run: |
           set -uo pipefail
-          if [ -z "${{GH_TOKEN:-}}" ]; then
+          if [ -z "${GH_TOKEN:-}" ]; then
             echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
             exit 1
           fi
@@ -448,31 +448,31 @@ jobs:
             if gh api --method POST repos/rustfs/rustfs/dispatches \
               -f event_type='rustfs-chain-storage' \
               -F 'client_payload[from_suite]=tier'; then
-              echo "dispatched next suite Storage engine (attempt ${{attempt}})"
+              echo "dispatched next suite Storage engine (attempt ${attempt})"
               DISPATCHED=1
               break
             fi
-            echo "dispatch attempt ${{attempt}} failed; retrying in ${{attempt}}0s" >&2
-            sleep "${{attempt}}0"
+            echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
+            sleep "${attempt}0"
           done
-          if [ "${{DISPATCHED:-0}}" -ne 1 ]; then
+          if [ "${DISPATCHED:-0}" -ne 1 ]; then
             echo "ERROR: functional chain stalled: could not dispatch Storage engine after 3 attempts" >&2
-            TITLE="[functional][chain] stalled after tier (run ${{GITHUB_RUN_ID}})"
+            TITLE="[functional][chain] stalled after tier (run ${GITHUB_RUN_ID})"
             BODY_FILE="$(mktemp)"
             {
               echo "The functional chain could not hand off from **tier** to **Storage engine** after 3 attempts."
               echo ""
-              echo "- Failed suite job: ${{GITHUB_SERVER_URL}}/${{GITHUB_REPOSITORY}}/actions/runs/${{GITHUB_RUN_ID}}"
-              echo "- Expected next event: `rustfs-chain-storage`"
+              echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+              echo "- Expected next event: 'rustfs-chain-storage'"
               echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
               echo "- Recovery: re-dispatch manually with"
-              echo "  ```"
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
               echo "  gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-storage'"
-              echo "  ```"
-            } > "${{BODY_FILE}}"
-            gh issue create -R rustfs/backlog --title "${{TITLE}}" \
-              --body-file "${{BODY_FILE}}" --label functional-test \
-              || gh issue create -R rustfs/backlog --title "${{TITLE}}" --body-file "${{BODY_FILE}}" \
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
+            } > "${BODY_FILE}"
+            gh issue create -R rustfs/backlog --title "${TITLE}" \
+              --body-file "${BODY_FILE}" --label functional-test \
+              || gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
               || echo "could not file the stall alert issue either; check the token" >&2
             exit 1
           fi

.github/workflows/rustfs-upgrade-test.yml

@@ -397,7 +397,7 @@ jobs:
           GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
         run: |
           set -uo pipefail
-          if [ -z "${{GH_TOKEN:-}}" ]; then
+          if [ -z "${GH_TOKEN:-}" ]; then
             echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
             exit 1
           fi
@@ -406,31 +406,31 @@ jobs:
             if gh api --method POST repos/rustfs/rustfs/dispatches \
               -f event_type='rustfs-chain-s3' \
               -F 'client_payload[from_suite]=upgrade'; then
-              echo "dispatched next suite S3 compatibility (attempt ${{attempt}})"
+              echo "dispatched next suite S3 compatibility (attempt ${attempt})"
               DISPATCHED=1
               break
             fi
-            echo "dispatch attempt ${{attempt}} failed; retrying in ${{attempt}}0s" >&2
-            sleep "${{attempt}}0"
+            echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
+            sleep "${attempt}0"
           done
-          if [ "${{DISPATCHED:-0}}" -ne 1 ]; then
+          if [ "${DISPATCHED:-0}" -ne 1 ]; then
             echo "ERROR: functional chain stalled: could not dispatch S3 compatibility after 3 attempts" >&2
-            TITLE="[functional][chain] stalled after upgrade (run ${{GITHUB_RUN_ID}})"
+            TITLE="[functional][chain] stalled after upgrade (run ${GITHUB_RUN_ID})"
             BODY_FILE="$(mktemp)"
             {
               echo "The functional chain could not hand off from **upgrade** to **S3 compatibility** after 3 attempts."
               echo ""
-              echo "- Failed suite job: ${{GITHUB_SERVER_URL}}/${{GITHUB_REPOSITORY}}/actions/runs/${{GITHUB_RUN_ID}}"
-              echo "- Expected next event: `rustfs-chain-s3`"
+              echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+              echo "- Expected next event: 'rustfs-chain-s3'"
               echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
               echo "- Recovery: re-dispatch manually with"
-              echo "  ```"
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
               echo "  gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-s3'"
-              echo "  ```"
-            } > "${{BODY_FILE}}"
-            gh issue create -R rustfs/backlog --title "${{TITLE}}" \
-              --body-file "${{BODY_FILE}}" --label functional-test \
-              || gh issue create -R rustfs/backlog --title "${{TITLE}}" --body-file "${{BODY_FILE}}" \
+              FENCE="$(printf "\x60\x60\x60")"; echo "  ${FENCE}"
+            } > "${BODY_FILE}"
+            gh issue create -R rustfs/backlog --title "${TITLE}" \
+              --body-file "${BODY_FILE}" --label functional-test \
+              || gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
               || echo "could not file the stall alert issue either; check the token" >&2
             exit 1
           fi

.github/workflows/star-history.yml

@@ -17,7 +17,7 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 10
     steps:
-      - uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
+      - uses: overtrue/repo-visuals-action@fd79cba437ecfac933d00a69add17eb95d3939c3 # v1.3.1
         with:
           github-token: ${{ github.token }}
           output-branch: star-history

.gitignore

@@ -57,9 +57,6 @@ docs/*
 !docs/operations/**
 !docs/testing/
 !docs/testing/**
-docs/heal-scanner-logging-governance.md
-docs/benchmark/rustfs-target-bench/
-docs/benchmark/*.md
 .codegraph/*
 .docker/test/compat/data/*
 .docker/test/compat/kms/*
@@ -83,6 +80,8 @@ worktrees/*
 
 # Local AI-agent review artifacts (omo evidence dumps)
 .omo/
+# Legacy per-tool skill dir; skills live in .agents/skills (shared by all agents)
+.mimocode/
 
 # insta scratch files; the accepted .snap files ARE the assertions and are committed
 *.snap.new

AGENTS.md

@@ -86,6 +86,7 @@ This file contains repository-wide rules. Use the nearest subdirectory
 - CI gates: `.github/workflows/ci.yml`.
 - PR format: `.github/pull_request_template.md`.
 - Architecture routing: `ARCHITECTURE.md` and `docs/architecture/README.md`.
+- Knowledge-base index and documentation rules: `docs/architecture/README.md`.
 - Agent skills: `.agents/skills/*/SKILL.md`.
 
 Do not commit one-shot plans, trackers, migration ledgers, benchmark snapshots,

ARCHITECTURE.md

@@ -62,7 +62,7 @@ rustfs/                      # Workspace root (virtual manifest)
 │   ├── utils/               # Pure utility functions
 │   ├── ...                  # (see "Crate Reference" below)
 │   └── e2e_test/            # End-to-end integration tests
-└── docs/                    # Design documents and analysis
+└── docs/                    # Agent knowledge base: contracts, runbooks, testing rules (index: docs/architecture/README.md)
 ```
 
 ### Main Crate Layers (`rustfs/src/`)

CLAUDE.md

@@ -27,6 +27,7 @@ make build-docker BUILD_OS=ubuntu22.04
 
 ## Where to look (do not duplicate here)
 
+- Agent knowledge base index and doc-writing rules: [docs/architecture/README.md](docs/architecture/README.md)
 - Crate membership: `Cargo.toml` `[workspace].members`
 - Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md)
 - Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md)

Cargo.lock

@@ -347,9 +347,9 @@ checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56"
 
 [[package]]
 name = "arrow"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "61d285d16bce7d0be61912f7928342b673067b6b7d7ef6cc179258ba7de1fecf"
+checksum = "7c14b3d39f306bc28fd639d59f06e17a0f377d0021e1b7e9054e4d6fedc98774"
 dependencies = [
  "arrow-arith",
  "arrow-array",
@@ -368,9 +368,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-arith"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "757ef1836251e88222542a7da2623bc1c9cb9e20afefa6db2c41e79991cd91d4"
+checksum = "ce2961626677665b2195eb59242af4c7befe7b8737ca2050295389362380104e"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
@@ -382,9 +382,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-array"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bc9a4a4b2b5ecd0e04df03471661cb61f28bed3c7fd50994715129b01b2edb97"
+checksum = "1e5f6adeffdf587d7a31db5d2266189624b526730cd3627f9ff9fedae97ad584"
 dependencies = [
  "ahash",
  "arrow-buffer",
@@ -401,9 +401,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-buffer"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c12b576ef18c1deb80925a248b25ad84f419198d791b8e293fc6aaa60441fe90"
+checksum = "097d193003ce7995d5d087089069ec2a6e0187faf5a6f8c9f38af2645d987182"
 dependencies = [
  "bytes",
  "half",
@@ -413,9 +413,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-cast"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "68338a9096a5dc9bc11927c58c43a8526d96bf6abd2012ef6c0c9f505991cc79"
+checksum = "635c9c635668ad26adf76cce8fb276c4be7cf06e63bd516de7da514f9680ee53"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
@@ -435,9 +435,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-csv"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "25011b52b346407d497ef0030e12b45e4f2d0cc279efc09c4f3d09106db30e36"
+checksum = "4c2ebf8d631e79b02c16cf5ae860561272c26024ec88fce389a56aaddd558e86"
 dependencies = [
  "arrow-array",
  "arrow-cast",
@@ -450,9 +450,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-data"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "723fe4aeed7604e00b9883a465af4ff0a0e6c44c03e41a68c3d1cbc403e0e44d"
+checksum = "9ba2f832eaeca24b8f26143dba750e42ee4ab51cf7d65e701ca9607cfda9f358"
 dependencies = [
  "arrow-buffer",
  "arrow-schema",
@@ -463,9 +463,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-ipc"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "149437b14371f5b9ec60f5ddc751483ae99d7a7072653c0075e5e469156eea7b"
+checksum = "dcc41681ea80f521df14c36725b74d4c60702c47f0793af2be469c04527e2599"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
@@ -479,9 +479,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-json"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f18b9123ccfec418a663f821c9a034af339711678c11ffe00d3ec07da5ff9f7e"
+checksum = "a2f57d7a81969f24ccf80809587b76c09897e6f829d2d65a5976bfb3218851f1"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
@@ -504,9 +504,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-ord"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e6c08dff0686cf23ca4f562803f191ccbeb726dbae6309cd4b4aaf65e0f2c979"
+checksum = "2c900759f3bd8354fd4196bc4403eee846894dc2adf66b4225472006a0bf18c5"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
@@ -517,9 +517,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-row"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bbec439386df71ad570e6758a946111322b9e9dc8db83b5527321f0b4c9119c2"
+checksum = "f4c6425032e28266e3fc4ff680805e57e670d6ea92473043f3e65b7ed6ac79f2"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
@@ -530,9 +530,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-schema"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e6fed2ca0d1eade57e811cbe73b98ad50cc08a1183e13b2d2aa43a7df593f40e"
+checksum = "10fab8d4563491417ba801fab29d205104d20d4bdf37bda6cd1cf425cff598cd"
 dependencies = [
  "serde_core",
  "serde_json",
@@ -540,9 +540,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-select"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "466b19cf75130b891dc1b23a84b343c714c62c64c9c62e365c76aa0ff90a53fb"
+checksum = "fc58569193c2525915f3cc6310edba3792f1200f65d6e9ed330aa33e691493b8"
 dependencies = [
  "ahash",
  "arrow-array",
@@ -554,9 +554,9 @@ dependencies = [
 
 [[package]]
 name = "arrow-string"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c838a25bb3691e919e0f617616ac51a4ff8517a952e29ca133cf0c22b2ce65b1"
+checksum = "2e0813f3c35c1cfea65e14c20a953440f7783c088b7ad2d0db162ccdeefcec14"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
@@ -919,9 +919,9 @@ dependencies = [
 
 [[package]]
 name = "aws-lc-rs"
-version = "1.18.0"
+version = "1.18.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ce2b2dcc879c3bae0d371e77c99f2238400ef24ec001394befa67b6e543add9e"
+checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e"
 dependencies = [
  "aws-lc-sys",
  "untrusted 0.7.1",
@@ -930,9 +930,9 @@ dependencies = [
 
 [[package]]
 name = "aws-lc-sys"
-version = "0.44.0"
+version = "0.45.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f09fae7be8bb3174e05c6afdb34199e6dc0c7c04ba9fa237b1967adfbde27483"
+checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27"
 dependencies = [
  "cc",
  "cmake",
@@ -6106,9 +6106,9 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
 
 [[package]]
 name = "libredox"
-version = "0.1.21"
+version = "0.1.23"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d7955dfc218a8afb29dfeffd540e3a6e96baeb94fe7138228dd7cc6937fbbf96"
+checksum = "8d8f1ea3f21fd3405dcaf6c9b5c1630af9afc422d9073ea39c5f6d6c772e08ed"
 dependencies = [
  "libc",
 ]
@@ -7014,7 +7014,7 @@ version = "5.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d"
 dependencies = [
- "base64 0.21.7",
+ "base64 0.22.1",
  "chrono",
  "getrandom 0.2.17",
  "http 1.5.0",
@@ -7528,9 +7528,9 @@ dependencies = [
 
 [[package]]
 name = "parquet"
-version = "59.2.0"
+version = "59.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7065842956a20c2a536924ce8e4d9955f7422451511b9eb7500d7bfe5077e59c"
+checksum = "ff322f54b1a0f9288e614ed1f2d329b380af5476420db19f46ffb865e1163d73"
 dependencies = [
  "ahash",
  "arrow-array",
@@ -8241,7 +8241,7 @@ version = "0.13.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
 dependencies = [
- "heck 0.4.1",
+ "heck 0.5.0",
  "itertools 0.14.0",
  "log",
  "multimap",
@@ -8261,7 +8261,7 @@ version = "0.14.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
 dependencies = [
- "heck 0.4.1",
+ "heck 0.5.0",
  "itertools 0.14.0",
  "log",
  "multimap",
@@ -10091,7 +10091,6 @@ dependencies = [
  "time",
  "tokio",
  "tracing",
- "url",
  "uuid",
 ]
 
@@ -11084,7 +11083,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
 [[package]]
 name = "s3s"
 version = "0.15.0"
-source = "git+https://github.com/rustfs/s3s.git?rev=28e9ebb23dd2fb7d667084f34121b4aa4807a5c6#28e9ebb23dd2fb7d667084f34121b4aa4807a5c6"
+source = "git+https://github.com/rustfs/s3s.git?rev=bdcb6259339c41369f9f1c60e3a42b5ab8da607b#bdcb6259339c41369f9f1c60e3a42b5ab8da607b"
 dependencies = [
  "arc-swap",
  "arrayvec",
@@ -11142,7 +11141,7 @@ dependencies = [
 [[package]]
 name = "s3s-rfc2047"
 version = "0.16.0-alpha.1"
-source = "git+https://github.com/rustfs/s3s.git?rev=28e9ebb23dd2fb7d667084f34121b4aa4807a5c6#28e9ebb23dd2fb7d667084f34121b4aa4807a5c6"
+source = "git+https://github.com/rustfs/s3s.git?rev=bdcb6259339c41369f9f1c60e3a42b5ab8da607b#bdcb6259339c41369f9f1c60e3a42b5ab8da607b"
 dependencies = [
  "base64-simd",
  "thiserror 2.0.20",
@@ -11151,7 +11150,7 @@ dependencies = [
 [[package]]
 name = "s3s-sigv2"
 version = "0.16.0-alpha.1"
-source = "git+https://github.com/rustfs/s3s.git?rev=28e9ebb23dd2fb7d667084f34121b4aa4807a5c6#28e9ebb23dd2fb7d667084f34121b4aa4807a5c6"
+source = "git+https://github.com/rustfs/s3s.git?rev=bdcb6259339c41369f9f1c60e3a42b5ab8da607b#bdcb6259339c41369f9f1c60e3a42b5ab8da607b"
 dependencies = [
  "base64-simd",
  "hmac 0.13.0",
@@ -11164,7 +11163,7 @@ dependencies = [
 [[package]]
 name = "s3s-sigv4"
 version = "0.16.0-alpha.1"
-source = "git+https://github.com/rustfs/s3s.git?rev=28e9ebb23dd2fb7d667084f34121b4aa4807a5c6#28e9ebb23dd2fb7d667084f34121b4aa4807a5c6"
+source = "git+https://github.com/rustfs/s3s.git?rev=bdcb6259339c41369f9f1c60e3a42b5ab8da607b#bdcb6259339c41369f9f1c60e3a42b5ab8da607b"
 dependencies = [
  "arrayvec",
  "base64-simd",
@@ -11787,9 +11786,9 @@ dependencies = [
 
 [[package]]
 name = "smallvec"
-version = "1.15.2"
+version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90"
+checksum = "b9be42f50aa861c555654aa3a37f52f4b1074bacf4e48fe0ef7fa584e80f1f0f"
 dependencies = [
  "serde",
 ]

Cargo.toml

@@ -307,11 +307,11 @@ rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/rustfs/s3s.git", rev = "28e9ebb23dd2fb7d667084f34121b4aa4807a5c6", version = "0.15.0", features = ["minio"] }
+s3s = { git = "https://github.com/rustfs/s3s.git", rev = "bdcb6259339c41369f9f1c60e3a42b5ab8da607b", version = "0.15.0", features = ["minio"] }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
-smallvec = { version = "1.15.2" }
+smallvec = { version = "1.16.0" }
 compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.3.0" }

README.md

@@ -48,16 +48,33 @@ Unlike other storage systems, RustFS is released under the permissible Apache 2.
 - **Open Source**: Licensed under Apache 2.0, encouraging unrestricted community contributions and commercial usage.
 - **User-Friendly**: Designed with simplicity in mind for easy deployment and management.
 
-| Feature                 | Status       | Feature                  | Status           |
-| :---------------------- | :----------- | :----------------------- | :--------------- |
-| **S3 Core Features**    | ✅ Available | **Bitrot Protection**    | ✅ Available     |
-| **Upload / Download**   | ✅ Available | **Single Node Mode**     | ✅ Available     |
-| **Versioning**          | ✅ Available | **Bucket Replication**   | ✅ Available     |
-| **Logging**             | ✅ Available | **Lifecycle Management** | 🚧 Under Testing |
-| **Event Notifications** | ✅ Available | **Distributed Mode**     | 🚧 Under Testing |
-| **K8s Helm Charts**     | ✅ Available | **RustFS KMS**           | 🚧 Under Testing |
-| **Keystone Auth**       | ✅ Available | **Multi-Tenancy**        | ✅ Available     |
-| **Swift API**           | ✅ Available | **Swift Metadata Ops**   | 🚧 Partial       |
+Status legend: ✅ Available — shipped and covered by CI gates; 🧪 Preview — shipped behind an opt-in flag or with a bounded compatibility claim.
+
+| Feature                          | Status       | Feature                            | Status       |
+| :------------------------------- | :----------- | :--------------------------------- | :----------- |
+| **S3 Core Features**             | ✅ Available | **Distributed Mode**               | ✅ Available |
+| **Upload / Download**            | ✅ Available | **Single Node Mode**               | ✅ Available |
+| **Versioning**                   | ✅ Available | **Bitrot Protection**              | ✅ Available |
+| **Object Lock (WORM)**           | ✅ Available | **Healing & Scanner**              | ✅ Available |
+| **Server-Side Encryption**       | ✅ Available | **Pool Expansion / Decommission**  | ✅ Available |
+| **RustFS KMS**                   | ✅ Available | **Bucket Replication**             | ✅ Available |
+| **Lifecycle Management (ILM)**   | ✅ Available | **Site Replication**               | ✅ Available |
+| **ILM Tiering (Remote S3)**      | ✅ Available | **Bucket Quota**                   | ✅ Available |
+| **S3 Select**                    | ✅ Available | **Event Notifications**            | ✅ Available |
+| **S3 Tables (Iceberg REST)**     | 🧪 Preview   | **Audit Logging**                  | ✅ Available |
+| **IAM / Policies**               | ✅ Available | **Logging & Observability**        | ✅ Available |
+| **OIDC / SSO**                   | ✅ Available | **Web Console**                    | ✅ Available |
+| **Keystone Auth**                | ✅ Available | **K8s Helm Charts**                | ✅ Available |
+| **Swift API**                    | ✅ Available | **FTPS / WebDAV**                  | ✅ Available |
+| **Multi-Tenancy**                | ✅ Available | **SFTP**                           | ✅ Available |
+| **MinIO On-Disk Compatibility**  | 🧪 Preview   |                                    |              |
+
+Notes:
+
+- **RustFS KMS**: Vault (KV2 / Transit) and AWS KMS backends are supported for production. The `Local` and `Static` backends are for development and testing only. See [KMS backend security properties](docs/operations/kms-backend-security.md).
+- **Swift API / SFTP**: opt-in cargo features (`--features swift`, `--features sftp`, or `full`). FTPS and WebDAV are enabled in the default build.
+- **S3 Tables**: ships as an Iceberg REST Catalog with automated PyIceberg and DuckDB coverage; other engines and vendor profiles carry bounded claims listed in the [S3 Tables support matrix](docs/architecture/s3-tables-support-matrix.md).
+- **MinIO On-Disk Compatibility**: gated behind the `rio-v2` feature and not part of the default build. Objects MinIO encrypted are not readable by RustFS. See [MinIO file-format interoperability](docs/architecture/minio-file-format-compat.md).
 
 ## RustFS vs MinIO Performance
 

crates/e2e_test/README.md

@@ -233,8 +233,8 @@ spawn error. Install the pinned CI version before running their profiles.
   [`src/policy/README.md`](src/policy/README.md),
   [`src/protocols/README.md`](src/protocols/README.md),
   [`src/reliant/README.md`](src/reliant/README.md)
-- Authoritative per-module counts:
-  [`docs/testing/e2e-suite-inventory.md`](../../docs/testing/e2e-suite-inventory.md)
+- Per-module counts: `cargo nextest list -p e2e_test --profile <profile>`
+  (one-liner in [`docs/testing/README.md`](../../docs/testing/README.md))
 - Test pyramid & flake policy: [`docs/testing/README.md`](../../docs/testing/README.md)
 
 ## CI smoke subset (`--profile e2e-smoke`)
@@ -271,12 +271,12 @@ Note on `#[serial]`: nextest runs each test in its own process, so
 parallel-safe by construction (random port + isolated temp dir), which the
 current subset is.
 
-### Authoritative test inventory
+### Test inventory
 
-`docs/testing/e2e-suite-inventory.md` records the per-module test counts as
-listed by `cargo nextest list -p e2e_test`. Regenerate it when adding or
-moving e2e tests so acceptance numbers in the test-strategy issues
-(backlog#1147–#1155) stay auditable. When a profile membership change is
+Per-module counts are not committed; list them with
+`cargo nextest list -p e2e_test --profile <profile>` (the result is
+platform-dependent because some modules are linux-only; the `jq` one-liner is
+in `docs/testing/README.md`). When a profile membership change is
 intentional, review its JSON listing before updating the matching
 `.config/e2e-*-selection.txt` test-ID digest. Update only the platform that
 produced the listing:

crates/e2e_test/src/fake_s3_target/README.md

@@ -1,11 +1,15 @@
 # Programmable fake S3 target
 
-This module is the shared failure-injection boundary for replication end-to-end tests. It runs an in-process, path-style S3 endpoint backed by `s3s`; no production crate depends on it.
+This module is the shared failure-injection boundary for replication end-to-end tests and the programmable external source for on-demand-migration (ODM) tests. It runs an in-process, path-style S3 endpoint backed by `s3s`; no production crate depends on it.
 
 `FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
 
-Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
+Supported data operations are HeadBucket, GetBucketVersioning, ListObjectsV2, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets created with `create_bucket` are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
 
-Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions. Each record also journals a `ProxyHeaderSnapshot` — the read-proxy anti-loop marker (`x-{rustfs,minio}-source-proxy-request`), the replication-check exemption header, and the client SSE-C header family (algorithm and key-MD5 values; for the key itself only its presence) — so proxy tests can pin the exact wire contract.
+`create_bucket_with_mode(name, BucketMode::Unversioned)` models a plain migration source: PUT overwrites in place, DELETE removes the key without a delete marker, GetBucketVersioning reports no status, and no `x-amz-version-id` is returned by PUT, GET, HEAD, tagging, or multipart completion. The only `versionId` such a bucket accepts is `null`; any other value is rejected with `InvalidArgument`. The mode is fixed at creation.
 
-The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.
+ListObjectsV2 lists current versions only (a key whose newest version is a delete marker is hidden) in byte order and supports `prefix`, `delimiter`, `max-keys` (clamped to 1000), `start-after`, and `continuation-token`; common prefixes count toward `max-keys`, `IsTruncated` / `NextContinuationToken` / `KeyCount` follow S3, and continuation tokens are opaque. `encoding-type` and `fetch-owner` are accepted but ignored, and ListObjects (v1) is not implemented. GET and HEAD honor `Range` in the `bytes=first-last`, `bytes=first-`, and `bytes=-suffix` forms with a 206 status, exact `Content-Range`, and `Accept-Ranges: bytes`; unsatisfiable ranges answer 416 `InvalidRange` with `Content-Range: bytes */<length>`. PUT and CreateMultipartUpload accept `Content-Type`, `Content-Encoding`, `Content-Disposition`, `Content-Language`, `Cache-Control`, `Expires`, and `x-amz-meta-*` (names stored lowercased), and HEAD/GET replay them verbatim together with `Last-Modified` and the ETag (hex MD5 for single PUTs, `<md5-of-part-md5s>-<parts>` for multipart objects). `put_seed_object` stores an object directly, bypassing the wire, the fault script, and the journal, so a source can be seeded without polluting the assertions a scenario later makes.
+
+Fault actions cover HTTP 401/403/503 responses (`Status`), any 4xx/5xx status paired with the matching S3 error code (`ResponseStatus`), pre-dispatch delay, holding a fully computed successful response before its first byte (`Stall`), connection abort when a logical request-body threshold is reached, GetObject bodies cut off after N bytes while `Content-Length` announces the full size (`TruncateBodyAt`), streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions and `count_requests(operation, key)` counts entries for one exact key. Each record journals the `Range` and `User-Agent` request headers, the ListObjectsV2 `prefix` and `continuation-token` query values, and a `ProxyHeaderSnapshot` — the read-proxy anti-loop marker (`x-{rustfs,minio}-source-proxy-request`), the replication-check exemption header, and the client SSE-C header family (algorithm and key-MD5 values; for the key itself only its presence) — so proxy tests can pin the exact wire contract.
+
+The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type and each standard object header at 1 KiB. By default a PUT or uploaded part is capped at 64 MiB and a completed multipart object and all stored object/part data are capped at 128 MiB; `FakeS3Target::start_with_options(FakeS3TargetOptions { max_object_bytes })` raises the object cap up to 256 MiB, and the total budget then becomes twice the object cap (never below 128 MiB). Body drain, body-permit waits, delay, stall, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -17,15 +17,115 @@
 #[cfg(test)]
 mod tests {
     use crate::chaos::{VersionShardCensus, census_object_version_on_disk, signed_admin_post};
-    use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, admin_request, init_logging};
+    use crate::common::{
+        FAST_DATA_USAGE_SCANNER_ENV, RustFSTestClusterEnvironment, RustFSTestEnvironment, admin_request, init_logging,
+    };
+    use crate::storage_api::RUSTFS_META_BUCKET;
     use aws_sdk_s3::primitives::ByteStream;
     use http::Method;
     use std::collections::HashSet;
     use std::error::Error;
+    use std::net::SocketAddr;
     use std::path::{Path, PathBuf};
+    use std::process::Command;
+    use tokio::net::TcpStream;
     use tokio::time::{Duration, Instant, sleep, timeout};
     use tracing::info;
 
+    const POOL_METADATA_OBJECT: &str = "pool.bin";
+
+    struct TcpPortBlackhole {
+        port: u16,
+        comment: String,
+        use_sudo: bool,
+        active: bool,
+    }
+
+    impl TcpPortBlackhole {
+        fn install(address: &str) -> Result<Self, Box<dyn Error + Send + Sync>> {
+            let address = address.parse::<SocketAddr>()?;
+            if !address.ip().is_loopback() {
+                return Err(format!("refusing to install a test firewall rule for non-loopback address {address}").into());
+            }
+
+            let id = Command::new("id").arg("-u").output()?;
+            if !id.status.success() {
+                return Err(format!("failed to determine the test process uid: {}", String::from_utf8_lossy(&id.stderr)).into());
+            }
+            let use_sudo = String::from_utf8_lossy(&id.stdout).trim() != "0";
+            let mut blackhole = Self {
+                port: address.port(),
+                comment: format!("rustfs-e2e-{}", uuid::Uuid::new_v4()),
+                use_sudo,
+                active: false,
+            };
+            blackhole.run_iptables(true)?;
+            blackhole.active = true;
+            Ok(blackhole)
+        }
+
+        fn restore(&mut self) -> Result<(), Box<dyn Error + Send + Sync>> {
+            if !self.active {
+                return Ok(());
+            }
+            self.run_iptables(false)?;
+            self.active = false;
+            Ok(())
+        }
+
+        fn run_iptables(&self, insert: bool) -> Result<(), Box<dyn Error + Send + Sync>> {
+            let mut command = if self.use_sudo {
+                let mut command = Command::new("sudo");
+                command.args(["-n", "iptables"]);
+                command
+            } else {
+                Command::new("iptables")
+            };
+            command.args(["-w", "5"]);
+            if insert {
+                command.args(["-I", "OUTPUT", "1"]);
+            } else {
+                command.args(["-D", "OUTPUT"]);
+            }
+            let port = self.port.to_string();
+            let output = command
+                .args([
+                    "-p",
+                    "tcp",
+                    "-d",
+                    "127.0.0.1/32",
+                    "--dport",
+                    &port,
+                    "-m",
+                    "comment",
+                    "--comment",
+                    &self.comment,
+                    "-j",
+                    "DROP",
+                ])
+                .output()?;
+            if !output.status.success() {
+                let action = if insert { "install" } else { "remove" };
+                return Err(format!(
+                    "failed to {action} endpoint blackhole rule for port {}: stdout={}, stderr={}",
+                    self.port,
+                    String::from_utf8_lossy(&output.stdout),
+                    String::from_utf8_lossy(&output.stderr)
+                )
+                .into());
+            }
+            Ok(())
+        }
+    }
+
+    impl Drop for TcpPortBlackhole {
+        fn drop(&mut self) {
+            if let Err(error) = self.restore() {
+                eprintln!("failed to remove {} firewall rule during test cleanup: {error}", self.comment);
+            }
+        }
+    }
+
     fn has_file_under(path: &Path) -> bool {
         let Ok(entries) = std::fs::read_dir(path) else {
             return false;
@@ -91,6 +191,62 @@ mod tests {
             .count()
     }
 
+    async fn assert_all_nodes_list_exact_keys(
+        clients: &[aws_sdk_s3::Client],
+        bucket: &str,
+        expected_keys: &HashSet<String>,
+    ) -> Result<(), Box<dyn Error + Send + Sync>> {
+        const PAGE_SIZE: i32 = 10;
+        for (node_index, client) in clients.iter().enumerate() {
+            let mut listed_keys = Vec::new();
+            let mut continuation_token = None;
+            let max_pages = expected_keys.len().div_ceil(PAGE_SIZE as usize) + 1;
+            let mut page_count = 0;
+            loop {
+                page_count += 1;
+                if page_count > max_pages {
+                    return Err(format!("node {node_index} listing exceeded the bounded {max_pages}-page budget").into());
+                }
+                let response = timeout(
+                    Duration::from_secs(15),
+                    client
+                        .list_objects_v2()
+                        .bucket(bucket)
+                        .max_keys(PAGE_SIZE)
+                        .set_continuation_token(continuation_token.clone())
+                        .send(),
+                )
+                .await??;
+                listed_keys.extend(
+                    response
+                        .contents()
+                        .iter()
+                        .filter_map(|object| object.key().map(str::to_owned)),
+                );
+                if !response.is_truncated().unwrap_or(false) {
+                    break;
+                }
+                let next_token = response
+                    .next_continuation_token()
+                    .filter(|token| Some(*token) != continuation_token.as_deref())
+                    .ok_or_else(|| format!("node {node_index} returned a truncated listing without a new continuation token"))?;
+                continuation_token = Some(next_token.to_owned());
+            }
+
+            let listed_key_set = listed_keys.iter().cloned().collect::<HashSet<_>>();
+            assert_eq!(
+                listed_keys.len(),
+                listed_key_set.len(),
+                "node {node_index} returned duplicate keys after recovery: {listed_keys:?}"
+            );
+            assert_eq!(
+                &listed_key_set, expected_keys,
+                "node {node_index} did not expose the complete recovered namespace"
+            );
+        }
+        Ok(())
+    }
+
     fn heal_task_status_diagnostic(body: &str) -> String {
         let Ok(status) = serde_json::from_str::<serde_json::Value>(body) else {
             return body.to_string();
@@ -125,11 +281,12 @@ mod tests {
             && operations["retryingTasks"].as_u64() == Some(0)
     }
 
+    // Queued low-priority repairs cannot execute while the single admin slot is
+    // occupied; ownership is determined by active and retrying tasks only.
     fn only_admin_heal_is_active(status: &serde_json::Value) -> bool {
         let operations = &status["healOperations"];
         status["clusterStatusComplete"] == serde_json::Value::Bool(true)
             && status["state"].as_str() == Some("active")
-            && operations["queueLength"].as_u64() == Some(0)
             && operations["activeTasks"].as_u64() == Some(1)
             && operations["retryingTasks"].as_u64() == Some(0)
             && operations["activeBySource"]["admin"].as_u64() == Some(1)
@@ -547,41 +704,144 @@ mod tests {
         .into())
     }
 
+    async fn wait_for_scanner_cycle_after(
+        cluster: &RustFSTestClusterEnvironment,
+        previous_cycle_end: u64,
+    ) -> Result<u64, Box<dyn Error + Send + Sync>> {
+        let deadline = Instant::now() + Duration::from_secs(60);
+        loop {
+            let mut latest_cycle_end = 0;
+            let mut versions_observed = false;
+            let mut observations = Vec::with_capacity(cluster.nodes.len());
+            for (node_index, node) in cluster.nodes.iter().enumerate() {
+                let (status, body) = timeout(
+                    Duration::from_secs(5),
+                    admin_request(
+                        &node.url,
+                        Method::GET,
+                        "/rustfs/admin/v3/scanner/status",
+                        None,
+                        &cluster.access_key,
+                        &cluster.secret_key,
+                    ),
+                )
+                .await??;
+                assert_eq!(status, 200, "scanner status must be available: {body}");
+                let status: serde_json::Value = serde_json::from_str(&body)?;
+                assert_eq!(status["enabled"].as_bool(), Some(true), "scanner must stay enabled: {status}");
+                let metrics = &status["metrics"];
+                let cycle_end = metrics["last_cycle_end_unix_secs"]
+                    .as_u64()
+                    .ok_or("scanner status is missing its completed-cycle timestamp")?;
+                let versions_scanned = metrics["versions_scanned"]
+                    .as_u64()
+                    .ok_or("scanner status is missing its version-coverage counter")?;
+                latest_cycle_end = latest_cycle_end.max(cycle_end);
+                versions_observed |= versions_scanned > 0;
+                observations.push(format!(
+                    "node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
+                    metrics["current_cycle"],
+                    metrics["current_cycle_active"],
+                    metrics["leader_lock_state"],
+                    metrics["last_cycle_result"],
+                ));
+            }
+            // The coordinator records cycle completion, but remote workers
+            // record scanned versions. Both witnesses need not share a node.
+            if latest_cycle_end > previous_cycle_end && versions_observed {
+                return Ok(latest_cycle_end);
+            }
+            if Instant::now() >= deadline {
+                return Err(format!(
+                    "enabled scanner did not complete an object-scanning cycle after {previous_cycle_end}: {observations:?}"
+                )
+                .into());
+            }
+            sleep(Duration::from_millis(250)).await;
+        }
+    }
+
     // Keep the original unformatted-disk scenario above. This case retains the
     // format identity so only the explicit admin task can rebuild missing data.
     #[tokio::test(flavor = "multi_thread")]
     async fn test_cluster_root_heal_resumes_missing_remote_shards_after_node_restart() -> Result<(), Box<dyn Error + Send + Sync>>
     {
+        run_cluster_root_heal_interruption(InterruptionScenario::IsolatedTargetRestart).await
+    }
+
+    #[tokio::test(flavor = "multi_thread")]
+    async fn test_cluster_root_heal_recovers_remote_shards_after_coordinator_restart() -> Result<(), Box<dyn Error + Send + Sync>>
+    {
+        timeout(
+            Duration::from_secs(420),
+            run_cluster_root_heal_interruption(InterruptionScenario::BackgroundCoordinatorRestart),
+        )
+        .await?
+    }
+
+    #[cfg(target_os = "linux")]
+    #[tokio::test(flavor = "multi_thread")]
+    async fn test_cluster_root_heal_recovers_after_target_endpoint_blackhole() -> Result<(), Box<dyn Error + Send + Sync>> {
+        timeout(
+            Duration::from_secs(420),
+            run_cluster_root_heal_interruption(InterruptionScenario::TargetEndpointBlackhole),
+        )
+        .await?
+    }
+
+    #[derive(Clone, Copy, PartialEq, Eq)]
+    enum InterruptionScenario {
+        IsolatedTargetRestart,
+        BackgroundCoordinatorRestart,
+        TargetEndpointBlackhole,
+    }
+
+    async fn run_cluster_root_heal_interruption(scenario: InterruptionScenario) -> Result<(), Box<dyn Error + Send + Sync>> {
+        let (background_enabled, interruption_node, interruption_kind) = match scenario {
+            InterruptionScenario::IsolatedTargetRestart => (false, 1, "target_restart"),
+            InterruptionScenario::BackgroundCoordinatorRestart => (true, 0, "coordinator_restart"),
+            InterruptionScenario::TargetEndpointBlackhole => (false, 1, "target_endpoint_blackhole"),
+        };
         init_logging();
         info!(
-            event = "heal_restart_started",
+            event = "heal_interruption_started",
             component = "e2e_test",
             subsystem = "heal",
-            "Starting root-heal restart test"
+            background_enabled,
+            interruption_node,
+            interruption_kind,
+            "Starting root-heal interruption test"
         );
 
         let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
         cluster.set_env("RUSTFS_UNSAFE_BYPASS_DISK_CHECK", "true");
         cluster.set_env("RUSTFS_HEAL_ENABLED", "true");
-        cluster.set_env("RUSTFS_HEAL_AUTO_HEAL_ENABLE", "false");
-        cluster.set_env("RUSTFS_HEAL_MRF_ENABLE", "false");
-        cluster.set_env("RUSTFS_SCANNER_ENABLED", "false");
-        cluster.set_env("RUSTFS_HEAL_MAX_CONCURRENT_HEALS", "1");
-        cluster.set_env("RUSTFS_HEAL_MAX_CONCURRENT_PER_SET", "1");
-        cluster.set_env("RUSTFS_HEAL_PAGE_OBJECT_CONCURRENCY", "1");
-        cluster.set_env("RUSTFS_HEAL_PAGE_PARALLEL_ENABLE", "false");
-        // Keep all storage nodes' Heal runtimes enabled so their disk services
-        // complete normal registration after restart. Scanner, auto-heal and
-        // MRF are disabled; the pre-root idle barrier below drains the direct
-        // outage-object repair before the explicit admin task starts.
+        // Heal control uses the first lexicographically sorted grid host.
+        // Keep that coordinator distinct from the remote target at index 1.
+        cluster.nodes.sort_by(|left, right| left.url.cmp(&right.url));
+        cluster.set_env("RUSTFS_HEAL_AUTO_HEAL_ENABLE", background_enabled.to_string());
+        cluster.set_env("RUSTFS_HEAL_MRF_ENABLE", background_enabled.to_string());
+        cluster.set_env("RUSTFS_SCANNER_ENABLED", background_enabled.to_string());
+        if background_enabled {
+            // Only the scanner cadence is accelerated. Keep normal Heal
+            // concurrency and every automatic recovery owner enabled.
+            for &(key, value) in FAST_DATA_USAGE_SCANNER_ENV {
+                cluster.set_env(key, value);
+            }
+        } else {
+            cluster.set_env("RUSTFS_HEAL_MAX_CONCURRENT_HEALS", "1");
+            cluster.set_env("RUSTFS_HEAL_MAX_CONCURRENT_PER_SET", "1");
+            cluster.set_env("RUSTFS_HEAL_PAGE_OBJECT_CONCURRENCY", "1");
+            cluster.set_env("RUSTFS_HEAL_PAGE_PARALLEL_ENABLE", "false");
+        }
+        // Keep every node's Heal runtime enabled for normal disk registration.
         let server_rust_log = std::env::var("RUSTFS_HEAL_CHAOS_SERVER_RUST_LOG")
             .unwrap_or_else(|_| "rustfs::heal::task=info,rustfs=error".to_string());
         cluster.set_env("RUST_LOG", server_rust_log);
-        if let Ok(log_dir) = std::env::var("RUSTFS_HEAL_CHAOS_LOG_DIR") {
-            std::fs::create_dir_all(&log_dir)?;
-            for node_index in 0..cluster.nodes.len() {
-                cluster.set_node_capture_log_path(node_index, format!("{log_dir}/node{node_index}.log"))?;
-            }
+        let log_dir = std::env::var("RUSTFS_HEAL_CHAOS_LOG_DIR").unwrap_or_else(|_| format!("{}/logs", cluster.temp_dir));
+        std::fs::create_dir_all(&log_dir)?;
+        for node_index in 0..cluster.nodes.len() {
+            cluster.set_node_capture_log_path(node_index, format!("{log_dir}/node{node_index}.log"))?;
         }
         cluster.start().await?;
         let clients = cluster.create_all_clients()?;
@@ -634,6 +894,18 @@ mod tests {
             });
         }
 
+        let expected_pool_metadata = if background_enabled {
+            let census = census_object_version_on_disk(&replaced_disk, RUSTFS_META_BUCKET, POOL_METADATA_OBJECT, None)?;
+            assert!(
+                census.is_complete(),
+                "target must hold complete pool metadata before the fault: {census:?}"
+            );
+            wait_for_scanner_cycle_after(&cluster, 0).await?;
+            Some(census)
+        } else {
+            None
+        };
+
         cluster.stop_node(1)?;
         std::fs::remove_dir_all(&replaced_disk)?;
         std::fs::create_dir_all(
@@ -691,25 +963,25 @@ mod tests {
             .find(|index| !outage_peer_erasure_indices.contains(index))
             .ok_or("online outage-object shards leave no erasure index for the replacement target")?;
 
-        // The PUT path may have admitted a direct Internal object repair while
-        // node 1 was offline. Cancel the isolated bucket path before the target
-        // returns; otherwise it could rebuild the outage object and invalidate
-        // the explicit-root ownership assertion below.
-        let cancel_outage_heal_path = format!("/rustfs/admin/v3/heal/{bucket}?forceStop=true");
-        let (cancel_status, cancel_body) = admin_request(
-            &cluster.nodes[0].url,
-            Method::POST,
-            &cancel_outage_heal_path,
-            Some(
-                r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#
-                    .to_string(),
-            ),
-            &cluster.access_key,
-            &cluster.secret_key,
-        )
-        .await?;
-        if !cancel_status.is_success() {
-            return Err(format!("cancel outage heal failed: {cancel_status} {cancel_body}").into());
+        let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
+        if !background_enabled {
+            // The PUT path may have admitted a direct Internal object repair while
+            // node 1 was offline. Cancel the isolated bucket path before the target
+            // returns; otherwise it could rebuild the outage object and invalidate
+            // the explicit-root ownership assertion below.
+            let cancel_outage_heal_path = format!("/rustfs/admin/v3/heal/{bucket}?forceStop=true");
+            let (cancel_status, cancel_body) = admin_request(
+                &cluster.nodes[0].url,
+                Method::POST,
+                &cancel_outage_heal_path,
+                Some(heal_body.to_string()),
+                &cluster.access_key,
+                &cluster.secret_key,
+            )
+            .await?;
+            if !cancel_status.is_success() {
+                return Err(format!("cancel outage heal failed: {cancel_status} {cancel_body}").into());
+            }
         }
 
         cluster.start_node(1).await?;
@@ -724,7 +996,12 @@ mod tests {
             );
             let recovered: serde_json::Value = serde_json::from_str(&status_body)
                 .map_err(|err| format!("background heal status is not JSON ({err}): {status_body}"))?;
-            if cluster_heal_is_idle(&recovered) {
+            let ready = if background_enabled {
+                recovered["clusterStatusComplete"] == serde_json::Value::Bool(true)
+            } else {
+                cluster_heal_is_idle(&recovered)
+            };
+            if ready {
                 break;
             }
             if Instant::now() >= recovery_deadline {
@@ -732,23 +1009,24 @@ mod tests {
             }
             sleep(Duration::from_millis(250)).await;
         }
-        assert_eq!(
-            matching_manifest_count(&replaced_disk, bucket, &expected_manifests)?,
-            0,
-            "non-admin Heal is disabled, so the replacement target must remain empty before the explicit root heal"
-        );
-        assert!(
-            !census_object_version_on_disk(&replaced_disk, bucket, outage_key, None)?.has_xl_meta,
-            "the object written during the outage must be absent before the explicit root heal"
-        );
         let pre_heal_replacement = replacement_recovery_status(&cluster).await?;
-        assert_eq!(
-            pre_heal_replacement["cluster"]["records"].as_array().map(Vec::len),
-            Some(0),
-            "isolated target must not retain an automatic replacement generation: {pre_heal_replacement}"
-        );
+        if !background_enabled {
+            assert_eq!(
+                matching_manifest_count(&replaced_disk, bucket, &expected_manifests)?,
+                0,
+                "non-admin Heal is disabled, so the replacement target must remain empty before the explicit root heal"
+            );
+            assert!(
+                !census_object_version_on_disk(&replaced_disk, bucket, outage_key, None)?.has_xl_meta,
+                "the object written during the outage must be absent before the explicit root heal"
+            );
+            assert_eq!(
+                pre_heal_replacement["cluster"]["records"].as_array().map(Vec::len),
+                Some(0),
+                "isolated target must not retain an automatic replacement generation: {pre_heal_replacement}"
+            );
+        }
 
-        let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
         let heal_url = format!("{}/rustfs/admin/v3/heal/?forceStart=true", cluster.nodes[0].url);
         let heal_start_body = signed_admin_post(&heal_url, Some(heal_body), &cluster.access_key, &cluster.secret_key).await?;
         let heal_start: serde_json::Value = serde_json::from_str(&heal_start_body)
@@ -764,24 +1042,39 @@ mod tests {
             .and_then(|value| value.parse::<u64>().ok())
             .unwrap_or(60);
         let partial_deadline = Instant::now() + Duration::from_secs(partial_timeout_secs);
-        let pre_interrupt_status = loop {
+        loop {
             let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?;
             let active_status: serde_json::Value = serde_json::from_str(&status_body)
                 .map_err(|err| format!("background heal status is not JSON ({err}): {status_body}"))?;
-            if only_admin_heal_is_active(&active_status) {
-                break active_status;
+            let active = if background_enabled {
+                active_status["state"].as_str() == Some("active")
+                    && active_status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1)
+            } else {
+                only_admin_heal_is_active(&active_status)
+            };
+            if active {
+                break;
             }
             if Instant::now() >= partial_deadline {
                 return Err(format!("root heal never became active within {partial_timeout_secs}s: {active_status}").into());
             }
             sleep(Duration::from_millis(50)).await;
-        };
-        let partial_count = loop {
-            let matching = matching_manifest_count(&replaced_disk, bucket, &expected_manifests)?;
-            if matching > 0 && matching < expected_manifests.len() {
-                break matching;
+        }
+        let (partial_count, partial_manifest) = loop {
+            // Hash one committed shard to prove progress without letting a
+            // full-corpus hash pass consume the interruption window.
+            let materialized = metadata_count(&replaced_disk, bucket, &expected_manifests);
+            if materialized > 0
+                && materialized < expected_manifests.len()
+                && let Some(expected) = expected_manifests
+                    .iter()
+                    .find(|expected| object_metadata_exists_on_disk(&replaced_disk, bucket, &expected.key))
+                && census_object_version_on_disk(&replaced_disk, bucket, &expected.key, None)?
+                    .matches_manifest(&expected.shard_census)
+            {
+                break (materialized, expected);
             }
-            if matching == expected_manifests.len() {
+            if materialized == expected_manifests.len() {
                 return Err(format!(
                     "root heal rebuilt all {} baseline objects before the target could be interrupted",
                     expected_manifests.len()
@@ -796,30 +1089,195 @@ mod tests {
             }
             sleep(Duration::from_millis(10)).await;
         };
+
+        let pre_interrupt_status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?;
+        let pre_interrupt_status: serde_json::Value = serde_json::from_str(&pre_interrupt_status_body)
+            .map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {pre_interrupt_status_body}"))?;
+        let pre_interrupt_replacement = replacement_recovery_status(&cluster).await?;
+        let coordinator_log = std::fs::read_to_string(format!("{log_dir}/node0.log"))?;
+        assert!(
+            coordinator_log
+                .lines()
+                .filter_map(|line| serde_json::from_str::<serde_json::Value>(line).ok())
+                .any(|event| {
+                    event["event"] == "heal_task_state"
+                        && event["task_id"] == client_token
+                        && event["heal_type"] == "cluster"
+                        && event["state"] == "started"
+                }),
+            "node 0 must have started the exact admin task before interruption"
+        );
+        let pre_interrupt_operations = &pre_interrupt_status["healOperations"];
+        assert_eq!(
+            pre_interrupt_operations["activeBySource"]["admin"].as_u64(),
+            Some(1),
+            "interruption must occur while the single admin task is active: {pre_interrupt_status}"
+        );
+        if !background_enabled {
+            assert!(
+                only_admin_heal_is_active(&pre_interrupt_status),
+                "isolated interruption must retain only the admin task: {pre_interrupt_status}"
+            );
+            assert_eq!(
+                pre_interrupt_replacement["cluster"]["records"].as_array().map(Vec::len),
+                Some(0),
+                "root-heal interruption point must not retain an automatic replacement generation: {pre_interrupt_replacement}"
+            );
+        }
         info!(
-            event = "heal_restart_checkpoint",
+            event = "heal_interruption_checkpoint",
             component = "e2e_test",
             subsystem = "heal",
-            partial_count,
-            "Verified unique admin owner before target interruption"
+            background_enabled,
+            interruption_node,
+            interruption_kind,
+            partial_metadata_count = partial_count,
+            verified_key = partial_manifest.key,
+            "Observed partial rebuild before interruption"
         );
 
-        cluster.stop_node(1)?;
-        let stopped_count = matching_manifest_count(&replaced_disk, bucket, &expected_manifests)?;
-        assert!(
-            stopped_count > 0 && stopped_count < expected_manifests.len(),
-            "the target must stop after a partial rebuild, observed before stop={partial_count}, after stop={stopped_count}, total={}",
-            expected_manifests.len()
-        );
-        let unclean_shutdown_marker = replaced_disk.join(".rustfs.sys").join("unclean-shutdown");
-        match std::fs::remove_file(&unclean_shutdown_marker) {
-            Ok(()) => {}
-            Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
-            Err(error) => {
-                return Err(format!("failed to isolate unclean recovery marker {unclean_shutdown_marker:?}: {error}").into());
+        let target_pid = cluster.nodes[1].process.as_ref().ok_or("target process is not running")?.id();
+        if scenario == InterruptionScenario::TargetEndpointBlackhole {
+            let node_pids = cluster
+                .nodes
+                .iter()
+                .map(|node| {
+                    node.process
+                        .as_ref()
+                        .ok_or("cluster process is not running")
+                        .map(std::process::Child::id)
+                })
+                .collect::<Result<Vec<_>, _>>()?;
+
+            timeout(Duration::from_secs(2), TcpStream::connect(&cluster.nodes[1].address)).await??;
+            let mut blackhole = TcpPortBlackhole::install(&cluster.nodes[1].address)?;
+            let blocked_connect = timeout(Duration::from_millis(500), TcpStream::connect(&cluster.nodes[1].address)).await;
+            assert!(
+                blocked_connect.is_err(),
+                "target endpoint connection must time out while the OUTPUT DROP rule is active: {blocked_connect:?}"
+            );
+
+            let stable_window_secs = std::env::var("RUSTFS_HEAL_CHAOS_BLACKHOLE_STABLE_SECS")
+                .ok()
+                .and_then(|value| value.parse::<u64>().ok())
+                .unwrap_or(2)
+                .clamp(1, 5);
+            let blackhole_timeout_secs = std::env::var("RUSTFS_HEAL_CHAOS_BLACKHOLE_TIMEOUT_SECS")
+                .ok()
+                .and_then(|value| value.parse::<u64>().ok())
+                .unwrap_or(20)
+                .clamp(stable_window_secs + 1, 60);
+            let blackhole_deadline = Instant::now() + Duration::from_secs(blackhole_timeout_secs);
+            let mut stable_count = metadata_count(&replaced_disk, bucket, &expected_manifests);
+            let mut stable_since = Instant::now();
+            loop {
+                for (node_index, (node, expected_pid)) in cluster.nodes.iter_mut().zip(&node_pids).enumerate() {
+                    let process = node
+                        .process
+                        .as_mut()
+                        .ok_or_else(|| format!("node {node_index} process disappeared"))?;
+                    assert_eq!(process.id(), *expected_pid, "node {node_index} PID changed during endpoint blackhole");
+                    assert!(process.try_wait()?.is_none(), "node {node_index} exited during endpoint blackhole");
+                }
+
+                let current_count = metadata_count(&replaced_disk, bucket, &expected_manifests);
+                if current_count == expected_manifests.len() {
+                    return Err("root heal completed before the endpoint blackhole became observable".into());
+                }
+                if current_count != stable_count {
+                    stable_count = current_count;
+                    stable_since = Instant::now();
+                }
+                if stable_since.elapsed() >= Duration::from_secs(stable_window_secs) {
+                    break;
+                }
+                if Instant::now() >= blackhole_deadline {
+                    return Err(format!(
+                        "target rebuild never remained stable for {stable_window_secs}s during the endpoint blackhole: last_count={stable_count}, total={}",
+                        expected_manifests.len()
+                    )
+                    .into());
+                }
+                sleep(Duration::from_millis(100)).await;
+            }
+
+            let blocked_task_body = timeout(
+                Duration::from_secs(5),
+                signed_admin_post(&task_status_url, None, &cluster.access_key, &cluster.secret_key),
+            )
+            .await??;
+            let blocked_task: serde_json::Value = serde_json::from_str(&blocked_task_body)
+                .map_err(|err| format!("blackholed task status is not JSON ({err}): {blocked_task_body}"))?;
+            assert_eq!(
+                blocked_task["summary"].as_str(),
+                Some("running"),
+                "the original admin task must remain resumable during the endpoint blackhole: {blocked_task}"
+            );
+            assert!(
+                census_object_version_on_disk(&replaced_disk, bucket, &partial_manifest.key, None)?
+                    .matches_manifest(&partial_manifest.shard_census),
+                "the witnessed complete shard must survive the endpoint blackhole"
+            );
+
+            blackhole.restore()?;
+            timeout(Duration::from_secs(2), TcpStream::connect(&cluster.nodes[1].address)).await??;
+            info!(
+                event = "heal_endpoint_blackhole_restored",
+                component = "e2e_test",
+                subsystem = "heal",
+                interruption_kind,
+                stable_metadata_count = stable_count,
+                stable_window_secs,
+                "Restored target endpoint forwarding"
+            );
+        } else {
+            cluster.stop_node(interruption_node)?;
+            let stopped_count = metadata_count(&replaced_disk, bucket, &expected_manifests);
+            assert!(
+                stopped_count > 0 && stopped_count < expected_manifests.len(),
+                "node {interruption_node} must stop during a partial rebuild, observed before stop={partial_count}, after stop={stopped_count}, total={}",
+                expected_manifests.len()
+            );
+            assert!(
+                census_object_version_on_disk(&replaced_disk, bucket, &partial_manifest.key, None)?
+                    .matches_manifest(&partial_manifest.shard_census),
+                "the witnessed complete shard must survive interruption"
+            );
+            let unclean_shutdown_marker = Path::new(&cluster.nodes[interruption_node].data_dir)
+                .join(".rustfs.sys")
+                .join("unclean-shutdown");
+            if background_enabled {
+                assert!(
+                    unclean_shutdown_marker.is_file(),
+                    "background restart must retain the real unclean-shutdown marker"
+                );
+            } else {
+                match std::fs::remove_file(&unclean_shutdown_marker) {
+                    Ok(()) => {}
+                    Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
+                    Err(error) => {
+                        return Err(
+                            format!("failed to isolate unclean recovery marker {unclean_shutdown_marker:?}: {error}").into()
+                        );
+                    }
+                }
+            }
+            cluster.start_node(interruption_node).await?;
+            if interruption_node == 0 {
+                let target = cluster.nodes[1]
+                    .process
+                    .as_mut()
+                    .ok_or("target process disappeared during coordinator restart")?;
+                assert_eq!(target.id(), target_pid, "coordinator restart must not replace the target process");
+                assert!(target.try_wait()?.is_none(), "the target must remain alive during coordinator restart");
             }
         }
-        cluster.start_node(1).await?;
+
+        let scanner_cycle_floor = if background_enabled {
+            Some(std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs())
+        } else {
+            None
+        };
 
         let heal_timeout_secs = std::env::var("RUSTFS_HEAL_REPLACED_DISK_TIMEOUT_SECS")
             .ok()
@@ -832,13 +1290,22 @@ mod tests {
             {
                 let matching = matching_manifest_count(&replaced_disk, bucket, &expected_manifests)?;
                 let outage_census = census_object_version_on_disk(&replaced_disk, bucket, outage_key, None)?;
-                if matching == expected_manifests.len() && outage_census.is_complete() {
+                let pool_metadata_matches = match &expected_pool_metadata {
+                    Some(expected) => {
+                        census_object_version_on_disk(&replaced_disk, RUSTFS_META_BUCKET, POOL_METADATA_OBJECT, None)?
+                            .matches_manifest(expected)
+                    }
+                    None => true,
+                };
+                if matching == expected_manifests.len() && outage_census.is_complete() && pool_metadata_matches {
                     break;
                 }
             }
             if Instant::now() >= heal_deadline {
                 let matching = matching_manifest_count(&replaced_disk, bucket, &expected_manifests)?;
                 let outage_census = census_object_version_on_disk(&replaced_disk, bucket, outage_key, None)?;
+                let pool_metadata =
+                    census_object_version_on_disk(&replaced_disk, RUSTFS_META_BUCKET, POOL_METADATA_OBJECT, None)?;
                 let final_status = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key)
                     .await
                     .unwrap_or_else(|err| format!("status request failed: {err}"));
@@ -858,7 +1325,7 @@ mod tests {
                     Err(_) => "replacement status request exceeded 5s diagnostic budget".to_string(),
                 };
                 return Err(format!(
-                    "root heal did not resume after target restart within {heal_timeout_secs}s: baseline={matching}/{}, outage={outage_census:?}, status={final_status}, task_status={task_status}, pre_interrupt_status={pre_interrupt_status}, pre_heal_replacement={pre_heal_replacement}, replacement_status={replacement_status}",
+                    "root heal did not recover after {interruption_kind} within {heal_timeout_secs}s: baseline={matching}/{}, outage={outage_census:?}, pool_metadata={pool_metadata:?}, status={final_status}, task_status={task_status}, pre_interrupt_status={pre_interrupt_status}, pre_heal_replacement={pre_heal_replacement}, pre_interrupt_replacement={pre_interrupt_replacement}, replacement_status={replacement_status}",
                     expected_manifests.len()
                 )
                 .into());
@@ -885,6 +1352,17 @@ mod tests {
             "the outage object must be rebuilt into its own missing erasure slot"
         );
 
+        if let Some(cycle_end) = scanner_cycle_floor {
+            wait_for_scanner_cycle_after(&cluster, cycle_end).await?;
+        }
+
+        let mut expected_keys = expected_manifests
+            .iter()
+            .map(|manifest| manifest.key.clone())
+            .collect::<HashSet<_>>();
+        assert!(expected_keys.insert(outage_key.to_string()));
+        assert_all_nodes_list_exact_keys(&clients, bucket, &expected_keys).await?;
+
         let target_client = cluster.create_s3_client(1)?;
         for expected in &expected_manifests {
             let response = target_client.get_object().bucket(bucket).key(&expected.key).send().await?;
@@ -914,6 +1392,30 @@ mod tests {
         let task_status_body = signed_admin_post(&task_status_url, None, &cluster.access_key, &cluster.secret_key).await?;
         let task_status: serde_json::Value = serde_json::from_str(&task_status_body)
             .map_err(|err| format!("heal task status is not JSON ({err}): {task_status_body}"))?;
+        if interruption_node == 0 {
+            // Admin tasks are process-local. Physical and queue convergence
+            // above establish recovery; a lost task must not report success.
+            assert_eq!(
+                task_status["summary"].as_str(),
+                Some("notFound"),
+                "interrupted task status: {task_status}"
+            );
+            assert_eq!(
+                task_status["detail"].as_str(),
+                Some("heal task not found or expired"),
+                "interrupted admin task must be explicitly unavailable: {task_status}"
+            );
+            info!(
+                event = "heal_interruption_recovered",
+                component = "e2e_test",
+                subsystem = "heal",
+                interruption_node,
+                interruption_kind,
+                task_state = "not_found",
+                "Physical recovery completed after coordinator restart"
+            );
+            return Ok(());
+        }
         if task_status["summary"].as_str() != Some("finished") {
             return Err(format!("heal data rebuilt but task did not finish successfully: {task_status}").into());
         }

crates/e2e_test/src/lib.rs

@@ -23,10 +23,17 @@ pub mod common;
 #[cfg(test)]
 pub mod chaos;
 
-// Programmable S3 target for replication failure-path tests (backlog#1147 repl-8).
+// Programmable S3 target for replication failure-path tests (backlog#1147 repl-8)
+// and on-demand-migration source scenarios (backlog#2151).
 #[cfg(test)]
 pub mod fake_s3_target;
 
+// On-demand migration (backlog#2147): shared two-server environment, admin
+// wrappers, and the harness self-test (backlog#2151). Behavior scenarios are
+// added by later ODM tasks.
+#[cfg(test)]
+pub mod on_demand_migration;
+
 // Socket-level network fault-injection proxy for black-box cluster tests
 // (backlog#1325 network fault-injection block): latency / blackhole / one-way
 // partition on the wire between nodes. Serves #1312/#1319 (lock-plane one-way

crates/e2e_test/src/negative_sigv4_test.rs

@@ -225,8 +225,8 @@ fn encode_unsigned_aws_chunked_with_sha256_trailer(decoded: &[u8]) -> Vec<u8> {
     let checksum = sha256_base64(decoded);
     let mut encoded = format!("{:x}\r\n", decoded.len()).into_bytes();
     encoded.extend_from_slice(decoded);
-    encoded.extend_from_slice(b"\r\n0\r\n\r\n");
-    encoded.extend_from_slice(format!("x-amz-checksum-sha256:{checksum}").as_bytes());
+    encoded.extend_from_slice(b"\r\n0\r\n");
+    encoded.extend_from_slice(format!("x-amz-checksum-sha256:{checksum}\r\n\r\n").as_bytes());
     encoded
 }
 
@@ -549,6 +549,68 @@ async fn tampered_upload_part_payload_is_rejected() -> Result<(), Box<dyn std::e
     Ok(())
 }
 
+/// s3s v0.16 validates the aws-chunked decoded length while RustFS consumes the
+/// body stream. Mismatches are client body errors and must not leak as 500s.
+#[tokio::test]
+async fn aws_chunked_decoded_length_mismatch_returns_incomplete_body() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    setup(&mut env).await?;
+
+    for (key, declared_len) in [
+        ("decoded-length-overrun.bin", 3_usize),
+        ("decoded-length-shortfall.bin", 9_usize),
+    ] {
+        let decoded = b"decoded";
+        assert_ne!(declared_len, decoded.len(), "test case must exercise a mismatch");
+        let encoded_body = encode_unsigned_aws_chunked_with_sha256_trailer(decoded);
+        let decoded_content_length = declared_len.to_string();
+        let path = format!("/{BUCKET}/{key}");
+        let signer = SigV4::new(&env);
+        let extra_signed_headers = [
+            ("content-encoding", "aws-chunked"),
+            ("x-amz-decoded-content-length", decoded_content_length.as_str()),
+            ("x-amz-trailer", "x-amz-checksum-sha256"),
+        ];
+        let headers = signer.sign_with_extra_headers("PUT", &path, "", UNSIGNED_PAYLOAD_TRAILER, &extra_signed_headers);
+
+        let response = local_http_client()
+            .put(format!("{}{}", env.url, path))
+            .header("authorization", &headers.authorization)
+            .header("content-encoding", "aws-chunked")
+            .header("x-amz-content-sha256", &headers.content_sha256)
+            .header("x-amz-date", &headers.amz_date)
+            .header("x-amz-decoded-content-length", &decoded_content_length)
+            .header("x-amz-trailer", "x-amz-checksum-sha256")
+            .body(encoded_body)
+            .timeout(std::time::Duration::from_secs(10))
+            .send()
+            .await?;
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        assert_eq!(
+            status,
+            reqwest::StatusCode::BAD_REQUEST,
+            "decoded length mismatch must be a client error, body:\n{body}"
+        );
+        assert_error_code(&body, "IncompleteBody");
+
+        let absent = env
+            .create_s3_client()
+            .get_object()
+            .bucket(BUCKET)
+            .key(key)
+            .send()
+            .await
+            .expect_err("decoded length mismatch must not publish an object");
+        assert_eq!(absent.raw_response().map(|response| response.status().as_u16()), Some(404));
+        assert_eq!(absent.as_service_error().and_then(ProvideErrorMetadata::code), Some("NoSuchKey"));
+    }
+
+    env.stop_server();
+    Ok(())
+}
+
 /// (e) A request whose `x-amz-date` is skewed beyond the server's tolerance
 /// (s3s default 900s / 15 min) must be rejected with RequestTimeTooSkewed /
 /// 403. The signature is otherwise valid: the credential-scope date and

crates/e2e_test/src/on_demand_migration/common.rs

@@ -0,0 +1,452 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Shared environment for on-demand migration (ODM) end-to-end tests.
+//!
+//! [`OdmTestEnv`] pairs one RustFS server under test with one in-process
+//! programmable S3 source ([`FakeS3Target`]). Admin calls target the route
+//! convention fixed by the tracking plan
+//! (`/rustfs/admin/v3/on-demand-migration/{bucket}`, JSON bodies); the
+//! server side lands with ODM-07, so until then the wrappers compile but are
+//! not exercised by the harness self-test.
+
+use crate::common::{RustFSTestEnvironment, signed_request};
+use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FakeS3TargetOptions, SeedMetadata};
+use aws_config::retry::RetryConfig;
+use aws_sdk_s3::Client;
+use aws_sdk_s3::config::{Credentials, Region};
+use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
+use bytes::Bytes;
+use serde::Serialize;
+use std::fmt;
+
+pub type BoxError = Box<dyn std::error::Error + Send + Sync>;
+
+/// Module switch the server reads at startup (`false` before GA). The harness
+/// turns it on so scenario tests exercise the feature without repeating it.
+pub const ODM_MODULE_SWITCH_ENV: &str = "RUSTFS_ON_DEMAND_MIGRATION_ENABLED";
+/// Admin route prefix; the bucket name is appended as one path segment.
+pub const ODM_ADMIN_ROUTE: &str = "/rustfs/admin/v3/on-demand-migration";
+/// Region the fake source is addressed with (it accepts any SigV4 region).
+pub const FAKE_SOURCE_REGION: &str = "us-east-1";
+
+/// Wire form of the bucket-level ODM configuration (ODM-01 model). Every
+/// field is public so a scenario can tweak one knob and serialize the rest
+/// with the documented defaults.
+#[derive(Debug, Clone, Serialize)]
+pub struct OdmSourceSpec {
+    pub version: u32,
+    pub enabled: bool,
+    pub source: OdmSource,
+    pub filter: OdmFilter,
+    pub policy: OdmPolicy,
+}
+
+#[derive(Debug, Clone, Serialize)]
+pub struct OdmSource {
+    pub provider: String,
+    pub endpoint: String,
+    pub region: String,
+    pub bucket: String,
+    pub path_style: String,
+    pub credentials: Option<OdmCredentials>,
+    pub tls: OdmTls,
+}
+
+#[derive(Clone, Serialize)]
+pub struct OdmCredentials {
+    pub access_key: String,
+    pub secret_key: String,
+    pub session_token: Option<String>,
+}
+
+impl fmt::Debug for OdmCredentials {
+    /// Test logs are captured into CI artifacts; keep the secret out of them.
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        f.debug_struct("OdmCredentials")
+            .field("access_key", &self.access_key)
+            .field("secret_key", &"REDACTED")
+            .field("session_token", &self.session_token.as_ref().map(|_| "REDACTED"))
+            .finish()
+    }
+}
+
+#[derive(Debug, Clone, Default, Serialize)]
+pub struct OdmTls {
+    pub skip_verify: bool,
+    pub ca_cert_pem: Option<String>,
+}
+
+#[derive(Debug, Clone, Default, Serialize)]
+pub struct OdmFilter {
+    pub prefix: Option<String>,
+    pub source_prefix: Option<String>,
+}
+
+#[derive(Debug, Clone, Serialize)]
+pub struct OdmPolicy {
+    pub head: String,
+    pub range_get: String,
+    pub source_error: String,
+    pub respect_local_delete_marker: bool,
+    pub preserve_etag: bool,
+    pub copy_tags: bool,
+    pub emit_events: bool,
+    pub negative_cache_ttl_secs: u64,
+    pub inline_max_bytes: u64,
+    pub multipart_part_size_bytes: u64,
+    pub max_concurrent_pulls: u32,
+    pub pull_queue_capacity: u32,
+    pub source_timeout: OdmSourceTimeout,
+    pub bandwidth_limit_bytes_per_sec: Option<u64>,
+}
+
+#[derive(Debug, Clone, Serialize)]
+pub struct OdmSourceTimeout {
+    pub connect_ms: u64,
+    pub first_byte_ms: u64,
+    pub idle_ms: u64,
+}
+
+impl Default for OdmPolicy {
+    /// The ODM-01 defaults verbatim.
+    fn default() -> Self {
+        Self {
+            head: "proxy".to_string(),
+            range_get: "serve_and_backfill".to_string(),
+            source_error: "propagate".to_string(),
+            respect_local_delete_marker: true,
+            preserve_etag: true,
+            copy_tags: false,
+            emit_events: true,
+            negative_cache_ttl_secs: 30,
+            inline_max_bytes: 16 * 1024 * 1024,
+            multipart_part_size_bytes: 64 * 1024 * 1024,
+            max_concurrent_pulls: 8,
+            pull_queue_capacity: 1024,
+            source_timeout: OdmSourceTimeout {
+                connect_ms: 5_000,
+                first_byte_ms: 15_000,
+                idle_ms: 30_000,
+            },
+            bandwidth_limit_bytes_per_sec: None,
+        }
+    }
+}
+
+impl OdmSourceSpec {
+    /// Enabled configuration pointing at a bucket on the fake source with the
+    /// fixture credentials, path-style addressing, and default policy.
+    pub fn for_fake_source(source: &FakeS3Target, source_bucket: impl Into<String>) -> Self {
+        Self::new(
+            "s3",
+            source.endpoint(),
+            FAKE_SOURCE_REGION,
+            source_bucket,
+            FAKE_ACCESS_KEY,
+            FAKE_SECRET_KEY,
+        )
+    }
+
+    /// Enabled configuration pointing at a bucket on a second RustFS server
+    /// (see [`start_source_rustfs`]).
+    pub fn for_rustfs_source(source: &RustFSTestEnvironment, source_bucket: impl Into<String>) -> Self {
+        Self::new(
+            "rustfs",
+            &source.url,
+            FAKE_SOURCE_REGION,
+            source_bucket,
+            &source.access_key,
+            &source.secret_key,
+        )
+    }
+
+    fn new(
+        provider: &str,
+        endpoint: &str,
+        region: &str,
+        source_bucket: impl Into<String>,
+        access_key: &str,
+        secret_key: &str,
+    ) -> Self {
+        Self {
+            version: 1,
+            enabled: true,
+            source: OdmSource {
+                provider: provider.to_string(),
+                endpoint: endpoint.to_string(),
+                region: region.to_string(),
+                bucket: source_bucket.into(),
+                path_style: "path".to_string(),
+                credentials: Some(OdmCredentials {
+                    access_key: access_key.to_string(),
+                    secret_key: secret_key.to_string(),
+                    session_token: None,
+                }),
+                tls: OdmTls::default(),
+            },
+            filter: OdmFilter::default(),
+            policy: OdmPolicy::default(),
+        }
+    }
+
+    pub fn to_json(&self) -> serde_json::Value {
+        serde_json::to_value(self).expect("ODM source spec serializes")
+    }
+}
+
+/// Backfill job control (ODM-12 route shape).
+#[derive(Debug, Clone)]
+pub enum BackfillOp {
+    Start(BackfillRequest),
+    Cancel,
+    Status,
+}
+
+#[derive(Debug, Clone, Default, Serialize)]
+pub struct BackfillRequest {
+    pub prefix: Option<String>,
+    pub skip_existing: Option<String>,
+    pub dry_run: bool,
+}
+
+/// Status plus raw body of an admin call, so a scenario can assert on the
+/// HTTP status first and only then parse the JSON.
+#[derive(Debug, Clone)]
+pub struct AdminResponse {
+    pub status: u16,
+    pub body: String,
+}
+
+impl AdminResponse {
+    pub fn json(&self) -> Result<serde_json::Value, BoxError> {
+        Ok(serde_json::from_str(&self.body)?)
+    }
+}
+
+/// One object to seed into the source.
+#[derive(Clone)]
+pub struct SeedObject {
+    pub key: String,
+    pub body: Bytes,
+    pub metadata: SeedMetadata,
+}
+
+impl SeedObject {
+    pub fn new(key: impl Into<String>, body: impl Into<Bytes>) -> Self {
+        Self {
+            key: key.into(),
+            body: body.into(),
+            metadata: SeedMetadata::new(),
+        }
+    }
+
+    pub fn with_metadata(mut self, metadata: SeedMetadata) -> Self {
+        self.metadata = metadata;
+        self
+    }
+}
+
+/// RustFS under test plus its fake S3 source.
+pub struct OdmTestEnv {
+    pub rustfs: RustFSTestEnvironment,
+    pub source: FakeS3Target,
+    /// S3 client for the RustFS under test.
+    pub client: Client,
+}
+
+impl OdmTestEnv {
+    /// Start a fake source with default limits and a RustFS server with the
+    /// ODM module switch enabled.
+    pub async fn start() -> Result<Self, BoxError> {
+        Self::start_with_options(FakeS3TargetOptions::default()).await
+    }
+
+    pub async fn start_with_options(options: FakeS3TargetOptions) -> Result<Self, BoxError> {
+        let source = FakeS3Target::start_with_options(options).await?;
+        let mut rustfs = RustFSTestEnvironment::new().await?;
+        rustfs
+            .start_rustfs_server_with_env(vec![], &[(ODM_MODULE_SWITCH_ENV, "true")])
+            .await?;
+        let client = rustfs.create_s3_client();
+        Ok(Self { rustfs, source, client })
+    }
+
+    /// S3 client addressing the fake source directly, for assertions on the
+    /// source's own state. Retries are off so a scripted fault is consumed by
+    /// exactly the request the test issued.
+    pub fn source_client(&self) -> Client {
+        fake_source_client(&self.source)
+    }
+
+    /// Enabled ODM configuration for `source_bucket` on the fake source.
+    pub fn fake_source_spec(&self, source_bucket: impl Into<String>) -> OdmSourceSpec {
+        OdmSourceSpec::for_fake_source(&self.source, source_bucket)
+    }
+
+    /// `PUT /rustfs/admin/v3/on-demand-migration/{bucket}` with the JSON spec.
+    pub async fn configure_source(&self, bucket: &str, spec: &OdmSourceSpec) -> Result<AdminResponse, BoxError> {
+        self.admin(http::Method::PUT, &format!("/{bucket}"), Some(spec.to_json()))
+            .await
+    }
+
+    /// Same as [`Self::configure_source`] with `dry-run=true`: validate and
+    /// probe without persisting.
+    pub async fn validate_source(&self, bucket: &str, spec: &OdmSourceSpec) -> Result<AdminResponse, BoxError> {
+        self.admin(http::Method::PUT, &format!("/{bucket}?dry-run=true"), Some(spec.to_json()))
+            .await
+    }
+
+    /// `GET .../{bucket}`: redacted configuration, 404 when unconfigured.
+    pub async fn get_config(&self, bucket: &str) -> Result<AdminResponse, BoxError> {
+        self.admin(http::Method::GET, &format!("/{bucket}"), None).await
+    }
+
+    /// `DELETE .../{bucket}`: remove the configuration (idempotent).
+    pub async fn disable(&self, bucket: &str) -> Result<AdminResponse, BoxError> {
+        self.admin(http::Method::DELETE, &format!("/{bucket}"), None).await
+    }
+
+    /// `GET .../{bucket}/status`: runtime snapshot.
+    pub async fn status(&self, bucket: &str) -> Result<AdminResponse, BoxError> {
+        self.admin(http::Method::GET, &format!("/{bucket}/status"), None).await
+    }
+
+    /// Backfill control: `POST .../{bucket}/backfill?op=start|cancel` or
+    /// `GET .../{bucket}/backfill` for the checkpoint.
+    pub async fn backfill(&self, bucket: &str, op: BackfillOp) -> Result<AdminResponse, BoxError> {
+        match op {
+            BackfillOp::Start(request) => {
+                self.admin(
+                    http::Method::POST,
+                    &format!("/{bucket}/backfill?op=start"),
+                    Some(serde_json::to_value(request)?),
+                )
+                .await
+            }
+            BackfillOp::Cancel => {
+                self.admin(http::Method::POST, &format!("/{bucket}/backfill?op=cancel"), None)
+                    .await
+            }
+            BackfillOp::Status => self.admin(http::Method::GET, &format!("/{bucket}/backfill"), None).await,
+        }
+    }
+
+    async fn admin(
+        &self,
+        method: http::Method,
+        path_and_query: &str,
+        body: Option<serde_json::Value>,
+    ) -> Result<AdminResponse, BoxError> {
+        let url = format!("{}{ODM_ADMIN_ROUTE}{path_and_query}", self.rustfs.url);
+        let body = body.map(|value| serde_json::to_vec(&value)).transpose()?;
+        let content_type = body.is_some().then_some("application/json");
+        let response = signed_request(method, &url, &self.rustfs.access_key, &self.rustfs.secret_key, body, content_type).await?;
+        Ok(AdminResponse {
+            status: response.status().as_u16(),
+            body: response.text().await?,
+        })
+    }
+
+    /// Store objects directly in the fake source (no wire traffic, no journal
+    /// entries). Returns the ETags in input order.
+    pub fn seed_source(&self, source_bucket: &str, objects: &[SeedObject]) -> Vec<String> {
+        objects
+            .iter()
+            .map(|object| {
+                self.source
+                    .put_seed_object(source_bucket, object.key.clone(), object.body.clone(), &object.metadata)
+            })
+            .collect()
+    }
+
+    /// Whether `key` is listed by the RustFS under test. Listing is served from
+    /// local state only, so this does not trigger a migration the way GET or
+    /// HEAD would.
+    pub async fn local_key_listed(&self, bucket: &str, key: &str) -> Result<bool, BoxError> {
+        let listed = self
+            .client
+            .list_objects_v2()
+            .bucket(bucket)
+            .prefix(key)
+            .max_keys(1)
+            .send()
+            .await?;
+        Ok(listed.contents().iter().any(|object| object.key() == Some(key)))
+    }
+
+    /// Panics unless `key` is stored locally with exactly `expected` bytes.
+    /// Presence is checked through listing first so a missing object fails
+    /// here instead of being pulled from the source by the GET.
+    pub async fn assert_local_present(&self, bucket: &str, key: &str, expected: &[u8]) {
+        assert!(
+            self.local_key_listed(bucket, key)
+                .await
+                .unwrap_or_else(|error| panic!("listing {bucket}/{key} failed: {error}")),
+            "{bucket}/{key} must be present locally"
+        );
+        let body = self
+            .client
+            .get_object()
+            .bucket(bucket)
+            .key(key)
+            .send()
+            .await
+            .unwrap_or_else(|error| panic!("GET {bucket}/{key} failed: {error}"))
+            .body
+            .collect()
+            .await
+            .unwrap_or_else(|error| panic!("reading {bucket}/{key} failed: {error}"))
+            .into_bytes();
+        assert_eq!(body.as_ref(), expected, "{bucket}/{key} local content mismatch");
+    }
+
+    /// Panics if `key` is listed locally.
+    pub async fn assert_local_absent(&self, bucket: &str, key: &str) {
+        assert!(
+            !self
+                .local_key_listed(bucket, key)
+                .await
+                .unwrap_or_else(|error| panic!("listing {bucket}/{key} failed: {error}")),
+            "{bucket}/{key} must be absent locally"
+        );
+    }
+}
+
+/// S3 client for the fake source with retries disabled (see
+/// [`OdmTestEnv::source_client`]).
+pub fn fake_source_client(source: &FakeS3Target) -> Client {
+    let credentials = Credentials::new(FAKE_ACCESS_KEY, FAKE_SECRET_KEY, None, None, "odm-fake-source");
+    Client::from_conf(
+        aws_sdk_s3::Config::builder()
+            .credentials_provider(credentials)
+            .region(Region::new(FAKE_SOURCE_REGION))
+            .endpoint_url(source.endpoint())
+            .force_path_style(true)
+            .behavior_version_latest()
+            .retry_config(RetryConfig::standard().with_max_attempts(1))
+            .http_client(SmithyHttpClientBuilder::new().build_http())
+            .build(),
+    )
+}
+
+/// Start a second, fully independent RustFS process (own port, data
+/// directory, and default credentials) to act as a real S3 source. It is
+/// spawned the same way `reliant::tiering` starts its cold tier; the process
+/// is stopped and its directory removed when the returned environment drops.
+pub async fn start_source_rustfs() -> Result<RustFSTestEnvironment, BoxError> {
+    let mut source = RustFSTestEnvironment::new().await?;
+    source.start_rustfs_server_without_cleanup(vec![]).await?;
+    Ok(source)
+}

crates/e2e_test/src/on_demand_migration/harness_self_test.rs

@@ -0,0 +1,606 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Self-test of the ODM harness (rustfs/backlog#2151): the fake source's
+//! migration-facing surface (ListObjectsV2 paging, `Range`, unversioned
+//! buckets, metadata replay, fault actions) and the two-server environment.
+//! No ODM behavior is exercised here.
+
+use super::common::{OdmTestEnv, SeedObject, fake_source_client, start_source_rustfs};
+use crate::fake_s3_target::{BucketMode, FakeS3Target, FakeS3TargetOptions, FaultAction, Operation, SeedMetadata};
+use aws_sdk_s3::Client;
+use aws_sdk_s3::error::ProvideErrorMetadata;
+use aws_sdk_s3::primitives::{ByteStream, DateTime};
+use bytes::Bytes;
+use std::collections::BTreeSet;
+use std::time::{Duration, Instant};
+
+type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+
+const SOURCE_BUCKET: &str = "odm-source";
+
+/// Position-dependent payload so a misaligned range read is caught.
+fn payload(len: usize) -> Bytes {
+    (0..len).map(|index| (index % 251) as u8).collect::<Vec<u8>>().into()
+}
+
+async fn fake_source() -> Result<(FakeS3Target, Client), Box<dyn std::error::Error + Send + Sync>> {
+    let source = FakeS3Target::start().await?;
+    source.create_bucket(SOURCE_BUCKET);
+    let client = fake_source_client(&source);
+    Ok((source, client))
+}
+
+/// Full ListObjectsV2 traversal. Returns `(keys, common prefixes, pages)` and
+/// checks the page shape on the way: every page except the last is full and
+/// truncated, the last carries no continuation token.
+async fn list_all(
+    client: &Client,
+    prefix: Option<&str>,
+    delimiter: Option<&str>,
+    start_after: Option<&str>,
+    max_keys: i32,
+) -> Result<(Vec<String>, Vec<String>, usize), Box<dyn std::error::Error + Send + Sync>> {
+    let mut keys = Vec::new();
+    let mut prefixes = Vec::new();
+    let mut pages = 0usize;
+    let mut token: Option<String> = None;
+    loop {
+        let page = client
+            .list_objects_v2()
+            .bucket(SOURCE_BUCKET)
+            .set_prefix(prefix.map(str::to_string))
+            .set_delimiter(delimiter.map(str::to_string))
+            .set_start_after(start_after.map(str::to_string))
+            .max_keys(max_keys)
+            .set_continuation_token(token.clone())
+            .send()
+            .await?;
+        pages += 1;
+        let page_keys: Vec<String> = page
+            .contents()
+            .iter()
+            .filter_map(|object| object.key().map(str::to_string))
+            .collect();
+        let page_prefixes: Vec<String> = page
+            .common_prefixes()
+            .iter()
+            .filter_map(|common| common.prefix().map(str::to_string))
+            .collect();
+        let entries = page_keys.len() + page_prefixes.len();
+        assert_eq!(page.key_count(), Some(entries as i32), "KeyCount must count keys and prefixes");
+        assert_eq!(page.continuation_token(), token.as_deref(), "the request token must be echoed");
+        keys.extend(page_keys);
+        prefixes.extend(page_prefixes);
+        if page.is_truncated() == Some(true) {
+            assert_eq!(entries as i32, max_keys, "every truncated page must be full");
+            token = Some(
+                page.next_continuation_token()
+                    .expect("truncated page must carry a continuation token")
+                    .to_string(),
+            );
+        } else {
+            assert!(page.next_continuation_token().is_none(), "final page must not carry a token");
+            return Ok((keys, prefixes, pages));
+        }
+    }
+}
+
+#[tokio::test]
+async fn fake_source_list_objects_v2_paginates_with_delimiter() -> TestResult {
+    let (source, client) = fake_source().await?;
+    let mut expected_keys = BTreeSet::new();
+    for directory in 0..30 {
+        for file in 0..30 {
+            expected_keys.insert(format!("d{directory:02}/k{file:03}"));
+        }
+    }
+    for index in 0..100 {
+        expected_keys.insert(format!("top-{index:03}"));
+    }
+    assert_eq!(expected_keys.len(), 1000);
+    for key in &expected_keys {
+        source.put_seed_object(SOURCE_BUCKET, key.clone(), Bytes::from(key.clone()), &SeedMetadata::new());
+    }
+    // A key whose current version is a delete marker must stay hidden.
+    client
+        .put_object()
+        .bucket(SOURCE_BUCKET)
+        .key("hidden/marker")
+        .body(ByteStream::from_static(b"gone"))
+        .send()
+        .await?;
+    client
+        .delete_object()
+        .bucket(SOURCE_BUCKET)
+        .key("hidden/marker")
+        .send()
+        .await?;
+    let expected_sorted: Vec<String> = expected_keys.iter().cloned().collect();
+    let expected_prefixes: Vec<String> = (0..30).map(|directory| format!("d{directory:02}/")).collect();
+    let expected_top: Vec<String> = (0..100).map(|index| format!("top-{index:03}")).collect();
+
+    // Flat traversal in byte order, 1000 keys in pages of 7.
+    let (keys, prefixes, pages) = list_all(&client, None, None, None, 7).await?;
+    assert_eq!(keys, expected_sorted);
+    assert!(prefixes.is_empty());
+    assert_eq!(pages, 143);
+
+    // Delimiter folding: 30 common prefixes then 100 top-level keys, pages of 7.
+    let (keys, prefixes, pages) = list_all(&client, None, Some("/"), None, 7).await?;
+    assert_eq!(prefixes, expected_prefixes);
+    assert_eq!(keys, expected_top);
+    assert_eq!(pages, 19);
+
+    // Empty prefix equals no prefix.
+    let (keys, _, _) = list_all(&client, Some(""), None, None, 1000).await?;
+    assert_eq!(keys, expected_sorted);
+
+    // No match: empty, not truncated, no token.
+    let (keys, prefixes, pages) = list_all(&client, Some("zzz/"), Some("/"), None, 7).await?;
+    assert!(keys.is_empty() && prefixes.is_empty());
+    assert_eq!(pages, 1);
+    let (keys, _, _) = list_all(&client, Some("hidden/"), None, None, 7).await?;
+    assert!(keys.is_empty(), "a current delete marker must hide its key");
+
+    // Exact page boundary: 30 keys under one directory, max-keys=30 -> one
+    // untruncated page.
+    let (keys, prefixes, pages) = list_all(&client, Some("d05/"), Some("/"), None, 30).await?;
+    assert_eq!(keys.len(), 30);
+    assert!(prefixes.is_empty());
+    assert_eq!(pages, 1);
+
+    // start-after skips keys at or before the marker.
+    let (keys, _, _) = list_all(&client, None, None, Some("top-097"), 1000).await?;
+    assert_eq!(keys, ["top-098", "top-099"]);
+
+    // max-keys is clamped to 1000; exactly 1000 keys fit in one page.
+    let (keys, _, pages) = list_all(&client, None, None, None, 5000).await?;
+    assert_eq!(keys.len(), 1000);
+    assert_eq!(pages, 1);
+
+    let listings: Vec<_> = source
+        .requests()
+        .into_iter()
+        .filter(|record| record.operation == Operation::ListObjectsV2)
+        .collect();
+    assert!(listings.len() >= 143 + 19);
+    assert!(listings.iter().any(|record| record.prefix.as_deref() == Some("d05/")));
+    assert!(
+        listings.iter().any(|record| record.continuation_token.is_some()),
+        "resumed pages must journal their continuation token"
+    );
+    assert!(listings.iter().all(|record| record.user_agent.is_some()));
+    source.shutdown().await;
+    Ok(())
+}
+
+#[tokio::test]
+async fn fake_source_range_get_variants_and_416() -> TestResult {
+    let (source, client) = fake_source().await?;
+    let body = payload(1000);
+    source.put_seed_object(SOURCE_BUCKET, "ranged", body.clone(), &SeedMetadata::new());
+
+    for (range, expected_range, expected_slice) in [
+        ("bytes=10-19", "bytes 10-19/1000", &body[10..20]),
+        ("bytes=990-", "bytes 990-999/1000", &body[990..]),
+        ("bytes=-5", "bytes 995-999/1000", &body[995..]),
+        ("bytes=0-5000", "bytes 0-999/1000", &body[..]),
+    ] {
+        let output = client
+            .get_object()
+            .bucket(SOURCE_BUCKET)
+            .key("ranged")
+            .range(range)
+            .send()
+            .await?;
+        assert_eq!(output.content_range(), Some(expected_range), "{range}");
+        assert_eq!(output.accept_ranges(), Some("bytes"), "{range}");
+        assert_eq!(output.content_length(), Some(expected_slice.len() as i64), "{range}");
+        let collected = output.body.collect().await?.into_bytes();
+        assert_eq!(collected.as_ref(), expected_slice, "{range}");
+    }
+    let head = client
+        .head_object()
+        .bucket(SOURCE_BUCKET)
+        .key("ranged")
+        .range("bytes=10-19")
+        .send()
+        .await?;
+    assert_eq!(head.content_range(), Some("bytes 10-19/1000"));
+    assert_eq!(head.content_length(), Some(10));
+
+    for range in ["bytes=1000-", "bytes=-0"] {
+        let error = client
+            .get_object()
+            .bucket(SOURCE_BUCKET)
+            .key("ranged")
+            .range(range)
+            .send()
+            .await
+            .expect_err("unsatisfiable range must fail");
+        let response = error.raw_response().expect("416 must retain the raw response");
+        assert_eq!(response.status().as_u16(), 416, "{range}");
+        assert_eq!(response.headers().get("content-range"), Some("bytes */1000"), "{range}");
+        assert_eq!(error.code(), Some("InvalidRange"), "{range}");
+    }
+
+    let ranged = source
+        .requests()
+        .into_iter()
+        .find(|record| record.operation == Operation::GetObject && record.range.as_deref() == Some("bytes=10-19"))
+        .expect("the Range header must be journaled verbatim");
+    assert_eq!(ranged.key.as_deref(), Some("ranged"));
+    assert!(source.count_requests(Operation::GetObject, "ranged") >= 6);
+    source.shutdown().await;
+    Ok(())
+}
+
+#[tokio::test]
+async fn fake_source_unversioned_bucket_overwrites_and_deletes() -> TestResult {
+    let (source, client) = fake_source().await?;
+    source.create_bucket_with_mode("plain-source", BucketMode::Unversioned);
+    let versioning = client.get_bucket_versioning().bucket("plain-source").send().await?;
+    assert!(versioning.status().is_none(), "unversioned bucket must report no versioning status");
+
+    let first = client
+        .put_object()
+        .bucket("plain-source")
+        .key("doc")
+        .body(ByteStream::from_static(b"first"))
+        .send()
+        .await?;
+    assert!(first.version_id().is_none());
+    let second = client
+        .put_object()
+        .bucket("plain-source")
+        .key("doc")
+        .body(ByteStream::from_static(b"second"))
+        .send()
+        .await?;
+    assert!(second.version_id().is_none());
+    let get = client.get_object().bucket("plain-source").key("doc").send().await?;
+    assert!(get.version_id().is_none(), "GET must not return x-amz-version-id");
+    assert_eq!(get.body.collect().await?.into_bytes().as_ref(), b"second");
+    let head = client.head_object().bucket("plain-source").key("doc").send().await?;
+    assert!(head.version_id().is_none(), "HEAD must not return x-amz-version-id");
+    assert_eq!(source.stored_versions("plain-source", "doc").len(), 1, "overwrite must replace in place");
+
+    let deleted = client.delete_object().bucket("plain-source").key("doc").send().await?;
+    assert!(deleted.delete_marker().is_none() && deleted.version_id().is_none());
+    let missing = client
+        .get_object()
+        .bucket("plain-source")
+        .key("doc")
+        .send()
+        .await
+        .expect_err("deleted object must be gone");
+    assert_eq!(missing.raw_response().map(|response| response.status().as_u16()), Some(404));
+    assert_eq!(missing.code(), Some("NoSuchKey"));
+    let missing_head = client
+        .head_object()
+        .bucket("plain-source")
+        .key("doc")
+        .send()
+        .await
+        .expect_err("deleted object must fail HEAD");
+    assert_eq!(missing_head.raw_response().map(|response| response.status().as_u16()), Some(404));
+    assert!(source.stored_versions("plain-source", "doc").is_empty(), "DELETE must not leave a marker");
+
+    // The versioned bucket on the same target keeps its version ids.
+    let versioned = client
+        .put_object()
+        .bucket(SOURCE_BUCKET)
+        .key("doc")
+        .body(ByteStream::from_static(b"versioned"))
+        .send()
+        .await?;
+    assert!(versioned.version_id().is_some());
+    source.shutdown().await;
+    Ok(())
+}
+
+#[tokio::test]
+async fn fake_source_replays_standard_and_user_metadata() -> TestResult {
+    let (source, client) = fake_source().await?;
+    let body = payload(4096);
+    let expected_etag = format!("\"{}\"", {
+        use md5::Digest as _;
+        hex_simd::encode_to_string(md5::Md5::digest(&body), hex_simd::AsciiCase::Lower)
+    });
+    // 2026-01-01T00:00:00Z rendered as an HTTP date by the SDK.
+    let expires = DateTime::from_secs(1_767_225_600);
+    client
+        .put_object()
+        .bucket(SOURCE_BUCKET)
+        .key("meta")
+        .body(ByteStream::from(body.clone()))
+        .content_type("application/x-odm")
+        .content_encoding("gzip")
+        .content_disposition("attachment; filename=\"meta.bin\"")
+        .content_language("en-US")
+        .cache_control("max-age=60")
+        .expires(expires)
+        .metadata("Foo-Bar", "mixed case name")
+        .metadata("UPPER", "upper name")
+        .metadata("already-lower", "lower name")
+        .send()
+        .await?;
+
+    let head = client.head_object().bucket(SOURCE_BUCKET).key("meta").send().await?;
+    let get = client.get_object().bucket(SOURCE_BUCKET).key("meta").send().await?;
+    for (label, content_type, content_encoding, content_disposition, content_language, cache_control, expires_string, e_tag) in [
+        (
+            "HEAD",
+            head.content_type(),
+            head.content_encoding(),
+            head.content_disposition(),
+            head.content_language(),
+            head.cache_control(),
+            head.expires_string(),
+            head.e_tag(),
+        ),
+        (
+            "GET",
+            get.content_type(),
+            get.content_encoding(),
+            get.content_disposition(),
+            get.content_language(),
+            get.cache_control(),
+            get.expires_string(),
+            get.e_tag(),
+        ),
+    ] {
+        assert_eq!(content_type, Some("application/x-odm"), "{label}");
+        assert_eq!(content_encoding, Some("gzip"), "{label}");
+        assert_eq!(content_disposition, Some("attachment; filename=\"meta.bin\""), "{label}");
+        assert_eq!(content_language, Some("en-US"), "{label}");
+        assert_eq!(cache_control, Some("max-age=60"), "{label}");
+        assert_eq!(expires_string, Some("Thu, 01 Jan 2026 00:00:00 GMT"), "{label}");
+        assert_eq!(e_tag, Some(expected_etag.as_str()), "{label}");
+    }
+    for metadata in [head.metadata(), get.metadata()] {
+        let metadata = metadata.expect("user metadata must be replayed");
+        assert_eq!(metadata.get("foo-bar").map(String::as_str), Some("mixed case name"));
+        assert_eq!(metadata.get("upper").map(String::as_str), Some("upper name"));
+        assert_eq!(metadata.get("already-lower").map(String::as_str), Some("lower name"));
+        assert!(!metadata.contains_key("Foo-Bar") && !metadata.contains_key("UPPER"));
+    }
+    assert!(head.last_modified().is_some());
+    assert_eq!(head.last_modified(), get.last_modified());
+    assert_eq!(head.content_length(), Some(4096));
+    assert_eq!(get.body.collect().await?.into_bytes(), body);
+
+    // Seeded objects replay the same way.
+    let seeded_etag = source.put_seed_object(
+        SOURCE_BUCKET,
+        "seeded",
+        Bytes::from_static(b"seeded"),
+        &SeedMetadata::new()
+            .content_type("text/plain")
+            .content_encoding("identity")
+            .cache_control("no-store")
+            .user_metadata("Origin", "seed"),
+    );
+    let seeded = client.head_object().bucket(SOURCE_BUCKET).key("seeded").send().await?;
+    assert_eq!(seeded.e_tag(), Some(format!("\"{seeded_etag}\"").as_str()));
+    assert_eq!(seeded.content_type(), Some("text/plain"));
+    assert_eq!(seeded.content_encoding(), Some("identity"));
+    assert_eq!(seeded.cache_control(), Some("no-store"));
+    assert_eq!(
+        seeded
+            .metadata()
+            .and_then(|metadata| metadata.get("origin"))
+            .map(String::as_str),
+        Some("seed")
+    );
+    source.shutdown().await;
+    Ok(())
+}
+
+#[tokio::test]
+async fn fake_source_fault_actions_truncate_stall_and_status() -> TestResult {
+    let (source, client) = fake_source().await?;
+    let body = payload(4096);
+    source.put_seed_object(SOURCE_BUCKET, "faulty", body.clone(), &SeedMetadata::new());
+
+    // TruncateBodyAt: headers promise 4096 bytes, the body ends after 100.
+    source.inject_for_key(Operation::GetObject, "faulty", FaultAction::TruncateBodyAt(100), 1);
+    let truncated = client.get_object().bucket(SOURCE_BUCKET).key("faulty").send().await?;
+    assert_eq!(truncated.content_length(), Some(4096));
+    let short_read = truncated
+        .body
+        .collect()
+        .await
+        .expect_err("a truncated body must fail to collect");
+    let short_read = short_read.to_string();
+    assert!(!short_read.is_empty());
+
+    // ResponseStatus: arbitrary status with the matching S3 error code.
+    for (code, expected_code) in [
+        (429u16, "SlowDown"),
+        (404, "NoSuchKey"),
+        (500, "InternalError"),
+        (503, "ServiceUnavailable"),
+    ] {
+        source.inject(Operation::GetObject, FaultAction::ResponseStatus(code), 1);
+        let error = client
+            .get_object()
+            .bucket(SOURCE_BUCKET)
+            .key("faulty")
+            .send()
+            .await
+            .expect_err("scripted status must fail");
+        assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(code));
+        assert_eq!(error.code(), Some(expected_code));
+    }
+
+    // Stall: the fully computed response is held before its first byte.
+    source.inject(Operation::HeadObject, FaultAction::Stall(Duration::from_millis(400)), 1);
+    let started = Instant::now();
+    let stalled = client.head_object().bucket(SOURCE_BUCKET).key("faulty").send().await?;
+    assert!(started.elapsed() >= Duration::from_millis(350), "stall must delay the first byte");
+    assert_eq!(stalled.content_length(), Some(4096));
+    let post_stall_started = Instant::now();
+    client.head_object().bucket(SOURCE_BUCKET).key("faulty").send().await?;
+    assert!(post_stall_started.elapsed() < Duration::from_millis(350), "stall is consumed once");
+
+    // The object is intact once the script is drained.
+    let intact = client.get_object().bucket(SOURCE_BUCKET).key("faulty").send().await?;
+    assert_eq!(intact.body.collect().await?.into_bytes(), body);
+
+    assert_eq!(source.count_requests(Operation::GetObject, "faulty"), 6);
+    assert_eq!(source.count_requests(Operation::HeadObject, "faulty"), 2);
+    assert_eq!(source.count_requests(Operation::GetObject, "other"), 0);
+    let records = source.requests();
+    assert!(
+        records.iter().all(|record| record
+            .user_agent
+            .as_deref()
+            .is_some_and(|agent| agent.contains("aws-sdk-rust"))),
+        "the SDK user agent must be journaled"
+    );
+    assert!(
+        records
+            .iter()
+            .any(|record| record.fault == Some(FaultAction::TruncateBodyAt(100)))
+    );
+    assert!(
+        records
+            .iter()
+            .any(|record| record.fault == Some(FaultAction::Stall(Duration::from_millis(400))))
+    );
+    source.shutdown().await;
+    Ok(())
+}
+
+#[tokio::test]
+async fn fake_source_raised_object_cap_accepts_large_put() -> TestResult {
+    let source = FakeS3Target::start_with_options(FakeS3TargetOptions {
+        max_object_bytes: 96 * 1024 * 1024,
+    })
+    .await?;
+    source.create_bucket(SOURCE_BUCKET);
+    let client = fake_source_client(&source);
+    let len = 64 * 1024 * 1024 + 1;
+    client
+        .put_object()
+        .bucket(SOURCE_BUCKET)
+        .key("large")
+        .body(ByteStream::from(vec![7u8; len]))
+        .send()
+        .await?;
+    let head = client.head_object().bucket(SOURCE_BUCKET).key("large").send().await?;
+    assert_eq!(head.content_length(), Some(len as i64));
+    let tail = client
+        .get_object()
+        .bucket(SOURCE_BUCKET)
+        .key("large")
+        .range("bytes=-1")
+        .send()
+        .await?;
+    assert_eq!(tail.content_range(), Some(format!("bytes {}-{}/{len}", len - 1, len - 1).as_str()));
+    source.shutdown().await;
+    Ok(())
+}
+
+#[tokio::test]
+async fn odm_env_starts_rustfs_and_fake_source() -> TestResult {
+    let env = OdmTestEnv::start().await?;
+    env.source.create_bucket(SOURCE_BUCKET);
+    let local_bucket = "odm-local";
+    env.rustfs.create_test_bucket(local_bucket).await?;
+
+    let etags = env.seed_source(
+        SOURCE_BUCKET,
+        &[
+            SeedObject::new("seed/a", Bytes::from_static(b"alpha")),
+            SeedObject::new("seed/b", Bytes::from_static(b"beta"))
+                .with_metadata(SeedMetadata::new().content_type("text/plain").user_metadata("Kind", "seed")),
+        ],
+    );
+    assert_eq!(etags.len(), 2);
+    assert!(env.source.requests().is_empty(), "seeding must not touch the journal");
+    let source_client = env.source_client();
+    let seeded = source_client.head_object().bucket(SOURCE_BUCKET).key("seed/b").send().await?;
+    assert_eq!(seeded.content_type(), Some("text/plain"));
+    assert_eq!(seeded.e_tag(), Some(format!("\"{}\"", etags[1]).as_str()));
+    assert_eq!(env.source.count_requests(Operation::HeadObject, "seed/b"), 1);
+
+    env.assert_local_absent(local_bucket, "seed/a").await;
+    env.client
+        .put_object()
+        .bucket(local_bucket)
+        .key("seed/a")
+        .body(ByteStream::from_static(b"alpha"))
+        .send()
+        .await?;
+    env.assert_local_present(local_bucket, "seed/a", b"alpha").await;
+    env.assert_local_absent(local_bucket, "seed/b").await;
+
+    let spec = env.fake_source_spec(SOURCE_BUCKET).to_json();
+    assert_eq!(spec["version"], 1);
+    assert_eq!(spec["enabled"], true);
+    assert_eq!(spec["source"]["provider"], "s3");
+    assert_eq!(spec["source"]["endpoint"], env.source.endpoint());
+    assert_eq!(spec["source"]["bucket"], SOURCE_BUCKET);
+    assert_eq!(spec["source"]["credentials"]["secret_key"], "fake-secret");
+    assert_eq!(spec["policy"]["source_timeout"]["first_byte_ms"], 15_000);
+    assert!(spec["policy"]["bandwidth_limit_bytes_per_sec"].is_null());
+    let debug = format!("{:?}", env.fake_source_spec(SOURCE_BUCKET));
+    assert!(!debug.contains("fake-secret"), "Debug output must redact the secret");
+    Ok(())
+}
+
+#[tokio::test]
+async fn start_source_rustfs_round_trips_put_get() -> TestResult {
+    let env = OdmTestEnv::start().await?;
+    let source = start_source_rustfs().await?;
+    assert_ne!(source.url, env.rustfs.url, "the source must be a separate instance");
+
+    source.create_test_bucket(SOURCE_BUCKET).await?;
+    let source_client = source.create_s3_client();
+    let body = payload(70_000);
+    let put = source_client
+        .put_object()
+        .bucket(SOURCE_BUCKET)
+        .key("real/object")
+        .body(ByteStream::from(body.clone()))
+        .content_type("application/octet-stream")
+        .send()
+        .await?;
+    assert!(put.e_tag().is_some());
+    let get = source_client
+        .get_object()
+        .bucket(SOURCE_BUCKET)
+        .key("real/object")
+        .send()
+        .await?;
+    assert_eq!(get.content_type(), Some("application/octet-stream"));
+    assert_eq!(get.body.collect().await?.into_bytes(), body);
+
+    let visible_to_primary = env
+        .client
+        .list_buckets()
+        .send()
+        .await?
+        .buckets()
+        .iter()
+        .any(|bucket| bucket.name() == Some(SOURCE_BUCKET));
+    assert!(!visible_to_primary, "the two servers must not share state");
+    let spec = super::common::OdmSourceSpec::for_rustfs_source(&source, SOURCE_BUCKET).to_json();
+    assert_eq!(spec["source"]["provider"], "rustfs");
+    assert_eq!(spec["source"]["endpoint"], source.url);
+    Ok(())
+}

crates/e2e_test/src/on_demand_migration/mod.rs

@@ -0,0 +1,24 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! On-demand migration (ODM) end-to-end suite (rustfs/backlog#2147).
+//!
+//! `common` is the shared environment: one RustFS under test, one programmable
+//! fake S3 source, admin-API wrappers, seeding and local-state assertions.
+//! `harness_self_test` proves the harness itself; ODM behavior scenarios are
+//! separate modules wired by later tasks.
+
+pub mod common;
+
+mod harness_self_test;

crates/e2e_test/src/storage_api.rs

@@ -15,7 +15,7 @@
 #[cfg(test)]
 pub(crate) use rustfs_ecstore::api::bucket::bucket_target_sys::BucketTargetSys;
 #[cfg(test)]
-pub(crate) use rustfs_ecstore::api::disk::{VolumeInfo, WalkDirOptions};
+pub(crate) use rustfs_ecstore::api::disk::{RUSTFS_META_BUCKET, VolumeInfo, WalkDirOptions};
 pub(crate) use rustfs_ecstore::api::rpc::{AuthenticatedChannel, TonicInterceptor, node_service_time_out_client_no_auth};
 #[cfg(test)]
 pub(crate) use rustfs_ecstore::api::rpc::{

crates/ecstore/src/api/mod.rs

@@ -440,6 +440,11 @@ pub mod object {
         PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
         SnapshotConsistencyError,
     };
+
+    #[cfg(feature = "test-util")]
+    pub mod test_util {
+        pub use crate::store::DeleteAfterObjectLockSnapshotBarrier;
+    }
 }
 
 pub mod rebalance {

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -921,7 +921,7 @@ impl ExpiryState {
         Ok(())
     }
 
-    pub fn enqueue_free_version(&mut self, oi: ObjectInfo) -> bool {
+    pub fn enqueue_free_version(&self, oi: ObjectInfo) -> bool {
         let task = FreeVersionTask(oi);
         let wrkr = self.get_worker_ch(task.op_hash());
         if wrkr.is_none() {
@@ -1215,6 +1215,22 @@ impl ExpiryState {
     }
 }
 
+pub(crate) async fn enqueue_committed_free_versions(api: &ECStore, free_versions: Vec<ObjectInfo>) -> usize {
+    if free_versions.is_empty() {
+        return 0;
+    }
+
+    let expiry_state = api.ctx.expiry_state();
+    let state = expiry_state.read().await;
+    let mut queued = 0;
+    for free_version in free_versions {
+        if state.enqueue_free_version(free_version) {
+            queued += 1;
+        }
+    }
+    queued
+}
+
 async fn enqueue_recovered_free_version_with_state(state: &Arc<RwLock<ExpiryState>>, oi: ObjectInfo) -> bool {
     let task = FreeVersionTask(oi);
     let hash = task.op_hash();
@@ -6627,7 +6643,7 @@ mod tests {
     async fn enqueue_free_version_reports_false_without_worker_channel() {
         let state = ExpiryState::new();
         let recovery_notify = Arc::clone(&state.read().await.recovery_notify);
-        let mut state = state.write().await;
+        let state = state.write().await;
         let oi = ObjectInfo {
             bucket: "bucket".to_string(),
             name: "object".to_string(),
@@ -6819,7 +6835,7 @@ mod tests {
             },
             ..Default::default()
         };
-        let mut state = state.write().await;
+        let state = state.write().await;
 
         assert!(state.enqueue_free_version(oi.clone()));
         assert!(recovery_notify.notified().now_or_never().is_none());

crates/ecstore/src/bucket/metadata_sys.rs

@@ -21,7 +21,7 @@ use crate::bucket::bucket_target_sys::BucketTargetSys;
 use crate::bucket::metadata::{load_bucket_metadata_parse, load_bucket_metadata_parse_with_presence};
 use crate::bucket::utils::is_meta_bucketname;
 use crate::disk::RUSTFS_META_BUCKET;
-use crate::error::{Error, Result, is_err_bucket_not_found};
+use crate::error::{Error, Result, is_err_bucket_not_found, is_err_strict_volume_not_found};
 use crate::runtime::sources as runtime_sources;
 use crate::storage_api_contracts::heal::HealOperations as _;
 use crate::storage_api_contracts::namespace::NamespaceLocking as _;
@@ -659,13 +659,12 @@ async fn acquire_config_write_guard_for_incarnation(
             async {
                 match metadata_sys
                     .api
-                    .peer_sys
-                    .get_bucket_info(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
+                    .get_bucket_info_from_sets(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
                     .await
                 {
                     Ok(_) => Ok(()),
-                    Err(crate::disk::error::Error::VolumeNotFound) => Err(Error::BucketNotFound(bucket.to_string())),
-                    Err(err) => Err(err.into()),
+                    Err(err) if is_err_strict_volume_not_found(&err) => Err(Error::BucketNotFound(bucket.to_string())),
+                    Err(err) => Err(err),
                 }
             },
         ),
@@ -1274,7 +1273,6 @@ pub struct BucketMetadataSys {
     /// name floods while avoiding repeated namespace and erasure reads.
     missing_buckets: moka::future::Cache<String, ()>,
     api: Arc<ECStore>,
-    initialized: Arc<RwLock<bool>>,
 }
 
 impl BucketMetadataSys {
@@ -1303,7 +1301,6 @@ impl BucketMetadataSys {
                 .time_to_live(MISSING_BUCKET_TTL)
                 .build(),
             api,
-            initialized: Arc::new(RwLock::new(false)),
         }
     }
 
@@ -1364,13 +1361,12 @@ impl BucketMetadataSys {
         await_bucket_namespace_operation(Some(namespace_guard), bucket, operation, async {
             match self
                 .api
-                .peer_sys
-                .get_bucket_info(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
+                .get_bucket_info_from_sets(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
                 .await
             {
                 Ok(_) => Ok(true),
-                Err(crate::disk::error::Error::VolumeNotFound) => Ok(false),
-                Err(err) => Err(err.into()),
+                Err(Error::VolumeNotFound) => Ok(false),
+                Err(err) => Err(err),
             }
         })
         .await
@@ -1380,9 +1376,15 @@ impl BucketMetadataSys {
         let _ = self.init_internal(buckets).await;
     }
     async fn init_internal(&self, buckets: Vec<String>) -> Result<()> {
-        let count = runtime_sources::endpoint_erasure_set_count()
-            .map(|count| count * 10)
-            .ok_or_else(|| Error::other("endpoint pools not initialized"))?;
+        let count = self
+            .api
+            .pools
+            .iter()
+            .map(|pool| pool.disk_set.len())
+            .sum::<usize>()
+            .checked_mul(10)
+            .filter(|count| *count != 0)
+            .ok_or_else(|| Error::other("bucket metadata store has no erasure sets"))?;
 
         let mut failed_buckets: HashSet<String> = HashSet::new();
         let mut buckets = buckets.as_slice();
@@ -1400,9 +1402,6 @@ impl BucketMetadataSys {
             buckets = &buckets[count..]
         }
 
-        let mut initialized = self.initialized.write().await;
-        *initialized = true;
-
         Ok(())
     }
 
@@ -1479,14 +1478,6 @@ impl BucketMetadataSys {
         expected: Option<&Arc<BucketMetadata>>,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
-        await_bucket_namespace_operation(
-            Some(namespace_guard),
-            bucket,
-            "bucket metadata heal",
-            self.api.heal_bucket(bucket, &HealOpts::default()),
-        )
-        .await?;
-
         if !self
             .bucket_exists(bucket, namespace_guard, "bucket metadata existence check")
             .await?
@@ -1506,6 +1497,20 @@ impl BucketMetadataSys {
             return Ok(());
         }
 
+        await_bucket_namespace_operation(
+            Some(namespace_guard),
+            bucket,
+            "bucket metadata heal",
+            self.api.heal_bucket(
+                bucket,
+                &HealOpts {
+                    recreate: true,
+                    ..Default::default()
+                },
+            ),
+        )
+        .await?;
+
         let (bm, persisted) = await_bucket_namespace_operation(
             Some(namespace_guard),
             bucket,
@@ -1887,23 +1892,13 @@ impl BucketMetadataSys {
                     "lazy metadata IO must start while the bucket namespace read lock is held"
                 );
             }
-            let (bm, persisted) = match await_bucket_namespace_operation(
+            let (bm, persisted) = await_bucket_namespace_operation(
                 Some(&guard),
                 bucket,
                 "lazy bucket metadata load",
                 Box::pin(load_bucket_metadata_parse_with_presence(self.api.clone(), bucket, true)),
             )
-            .await
-            {
-                Ok(res) => res,
-                Err(err) => {
-                    return if *self.initialized.read().await {
-                        Err(Error::other("errBucketMetadataNotInitialized"))
-                    } else {
-                        Err(err)
-                    };
-                }
-            };
+            .await?;
 
             let bm = Arc::new(bm);
 
@@ -1914,11 +1909,9 @@ impl BucketMetadataSys {
                     "lazy bucket metadata existence check",
                     Box::pin(async {
                         self.api
-                            .peer_sys
-                            .get_bucket_info(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
+                            .get_bucket_info_from_sets(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
                             .await
                             .map(|_| ())
-                            .map_err(Into::into)
                     }),
                 )
                 .await?;
@@ -2197,10 +2190,8 @@ impl BucketMetadataSys {
             "legacy bucket metadata existence check",
             async {
                 self.api
-                    .peer_sys
-                    .get_bucket_info(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
+                    .get_bucket_info_from_sets(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
                     .await
-                    .map_err(crate::error::StorageError::from)
             },
         )
         .await
@@ -2302,10 +2293,8 @@ impl BucketMetadataSys {
             "bucket metadata snapshot existence check",
             async {
                 self.api
-                    .peer_sys
-                    .get_bucket_info(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
+                    .get_bucket_info_from_sets(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
                     .await
-                    .map_err(crate::error::StorageError::from)
             },
         )
         .await

crates/ecstore/src/bucket/replication/replication_target_boundary.rs

@@ -260,8 +260,15 @@ pub(crate) fn replication_put_object_options(sc: &str, object_info: &ObjectInfo)
             // the response path relies on). Keep the object's own multipart
             // flag so encrypted objects stay on the multipart route.
         } else {
-            let (checksum_meta, is_mp) = object_info.decrypt_checksums(0, &HeaderMap::new())?;
-            is_multipart = is_mp;
+            let (checksum_meta, checksum_record_is_multipart) = object_info.decrypt_checksums(0, &HeaderMap::new())?;
+            // The checksum record describes how the *checksum* is composed,
+            // not how the object is stored. A full-object checksum carries no
+            // MULTIPART flag even on a multipart upload, so trusting it here
+            // routed a 768-part object through a single PutObject and the
+            // target rejected the 6 GiB body with EntityTooLarge
+            // (rustfs#6825). The object's own shape is the authority: the
+            // record may only add multipart-ness, never take it away.
+            is_multipart = object_info.is_multipart() || checksum_record_is_multipart;
 
             for (key, value) in checksum_meta.iter() {
                 if key != AMZ_CHECKSUM_TYPE {
@@ -518,6 +525,109 @@ mod tests {
     use time::Duration;
     use uuid::Uuid;
 
+    /// Serialize an object-level checksum record the way
+    /// `complete_multipart_upload` persists it for a **full-object** checksum:
+    /// the record carries the plain algorithm type, without the MULTIPART
+    /// flags that a composite record gets.
+    fn full_object_multipart_checksum_record() -> bytes::Bytes {
+        let checksum_type = rustfs_rio::ChecksumType::from_string_with_obj_type("crc32", "FULL_OBJECT");
+        assert!(checksum_type.is_set(), "crc32 FULL_OBJECT must be a valid checksum type");
+        assert!(checksum_type.full_object_requested());
+
+        let mut combined = Vec::new();
+        let mut checksum = rustfs_rio::Checksum {
+            checksum_type,
+            ..Default::default()
+        };
+        for part in [b"part-one".as_slice(), b"part-two".as_slice()] {
+            let part_checksum = rustfs_rio::Checksum::new_from_data(checksum_type, part).expect("part checksum");
+            combined.extend_from_slice(part_checksum.raw.as_slice());
+            checksum.add_part(&part_checksum, part.len() as i64).expect("add part");
+        }
+
+        checksum.to_bytes(&combined)
+    }
+
+    #[test]
+    fn multipart_object_with_full_object_checksum_keeps_the_multipart_route() {
+        // rustfs#6825: a 768-part upload was replicated with a single
+        // PutObject and rejected by the target with EntityTooLarge. The
+        // object's storage shape says multipart; only the checksum record
+        // looked single-part, and the checksum record must not decide the
+        // transport.
+        let object_info = ObjectInfo {
+            etag: Some("0123456789abcdef0123456789abcdef-768".to_string()),
+            checksum: Some(full_object_multipart_checksum_record()),
+            size: 6 * 1024 * 1024 * 1024,
+            ..Default::default()
+        };
+
+        assert!(object_info.is_multipart(), "the fixture must be a multipart object");
+        let (_, checksum_says_multipart) = object_info
+            .decrypt_checksums(0, &HeaderMap::new())
+            .expect("checksum record must decode");
+        assert!(
+            !checksum_says_multipart,
+            "fixture precondition: a full-object record carries no MULTIPART flag, which is what used to \
+             downgrade the transport"
+        );
+
+        let (_, is_multipart) = replication_put_object_options("STANDARD", &object_info).expect("build put options");
+
+        assert!(
+            is_multipart,
+            "a multipart object must replicate over multipart whatever its checksum record looks like"
+        );
+    }
+
+    #[test]
+    fn checksum_record_never_changes_the_transport_a_single_part_object_needs() {
+        // The mirror of the rustfs#6825 guard: an object stored as one PUT
+        // must keep the single-PUT transport, or its replica's ETag would
+        // change shape and every ETag-based convergence check would re-copy it.
+        let checksum =
+            rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::CRC32, b"whole-object").expect("checksum fixture");
+        let object_info = ObjectInfo {
+            etag: Some("0123456789abcdef0123456789abcdef".to_string()),
+            checksum: Some(checksum.to_bytes(&[])),
+            ..Default::default()
+        };
+
+        assert!(!object_info.is_multipart(), "the fixture must be a single-part object");
+
+        let (_, is_multipart) = replication_put_object_options("STANDARD", &object_info).expect("build put options");
+
+        assert!(!is_multipart, "a single-part object must not be promoted onto the multipart transport");
+    }
+
+    #[test]
+    fn composite_checksum_multipart_object_keeps_the_multipart_route() {
+        // The checksum shape that already worked before rustfs#6825, pinned so
+        // the fix cannot regress it.
+        let mut checksum_type = rustfs_rio::ChecksumType::from_string("crc32");
+        checksum_type
+            .merge(rustfs_rio::ChecksumType::MULTIPART)
+            .merge(rustfs_rio::ChecksumType::INCLUDES_MULTIPART);
+
+        let mut combined = Vec::new();
+        for part in [b"part-one".as_slice(), b"part-two".as_slice()] {
+            let part_checksum =
+                rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::from_string("crc32"), part).expect("part checksum");
+            combined.extend_from_slice(part_checksum.raw.as_slice());
+        }
+        let checksum = rustfs_rio::Checksum::new_from_data(checksum_type, &combined).expect("composite checksum");
+
+        let object_info = ObjectInfo {
+            etag: Some("0123456789abcdef0123456789abcdef-2".to_string()),
+            checksum: Some(checksum.to_bytes(&combined)),
+            ..Default::default()
+        };
+
+        let (_, is_multipart) = replication_put_object_options("STANDARD", &object_info).expect("build put options");
+
+        assert!(is_multipart, "a composite-checksum multipart object must stay on the multipart transport");
+    }
+
     #[test]
     fn replication_action_for_target_head_existing_object_source_newer_null_version_requires_replication() {
         let source = ObjectInfo {

crates/ecstore/src/cluster/rpc/peer_s3_client.rs

@@ -190,7 +190,7 @@ fn install_heal_bucket_pre_mutation_barrier() -> Arc<DeleteBucketEmptyScanBarrie
 }
 
 #[cfg(test)]
-async fn pause_after_delete_bucket_empty_scan() {
+pub(crate) async fn pause_after_delete_bucket_empty_scan() {
     let barrier = DELETE_BUCKET_EMPTY_SCAN_BARRIER
         .lock()
         .expect("empty scan barrier lock should not be poisoned")

crates/ecstore/src/data_movement/mod.rs

@@ -717,8 +717,8 @@ fn is_equivalent_data_movement_part(source: &ObjectPartInfo, target: &ObjectPart
                 == target.checksums.as_ref().filter(|checksums| !checksums.is_empty()))
 }
 
-fn data_movement_parts_by_number(parts: &[ObjectPartInfo]) -> Option<BTreeMap<usize, &ObjectPartInfo>> {
-    let mut parts_by_number = BTreeMap::new();
+pub(crate) fn data_movement_parts_by_number(parts: &[ObjectPartInfo]) -> Option<HashMap<usize, &ObjectPartInfo>> {
+    let mut parts_by_number = HashMap::with_capacity(parts.len());
     for part in parts {
         if parts_by_number.insert(part.number, part).is_some() {
             return None;

crates/ecstore/src/disk/local.rs

@@ -6444,6 +6444,9 @@ impl LocalDisk {
                     .abort_reserved_version_delete(object_dir, rollback_dir, volume, path, "delete_versions_commit_intent", err)
                     .await);
             }
+            if should_fail_after_delete_commit(self.root.as_path(), path) {
+                return Err(DiskError::Unexpected);
+            }
             return Ok(());
         }
 
@@ -6514,6 +6517,10 @@ impl LocalDisk {
                 .await);
         }
 
+        if should_fail_after_delete_commit(self.root.as_path(), path) {
+            return Err(DiskError::Unexpected);
+        }
+
         Ok(())
     }
 

crates/ecstore/src/object_api/types.rs

@@ -19,6 +19,7 @@ use crate::storage_api_contracts::{
         HTTPPreconditions, ObjectLockRetentionOptions, ObjectPreconditionError, ObjectPreconditionPart, ObjectPreconditionState,
     },
 };
+use std::io;
 use std::sync::atomic::{AtomicBool, AtomicU8, Ordering};
 use tokio::sync::{Mutex, Notify, OwnedRwLockReadGuard};
 use tokio_util::sync::CancellationToken;
@@ -678,6 +679,196 @@ pub struct DecommissionCapacityOptions {
     pub(crate) mutation_id: Option<Uuid>,
 }
 
+/// Opaque storage-owned collection point for post-commit tier free-version
+/// cleanup receipts. This type is public only because workspace crates build
+/// [`ObjectOptions`] with struct literals; callers outside `ecstore` must leave
+/// the corresponding option unset.
+#[doc(hidden)]
+#[derive(Clone)]
+pub struct TierFreeVersionReceiptSink {
+    inner: Arc<parking_lot::Mutex<TierFreeVersionReceiptSinkState>>,
+}
+
+struct TierFreeVersionReceiptSinkState {
+    receipts: Option<HashMap<TierFreeVersionReceiptIdentity, TierFreeVersionReceiptPayload>>,
+}
+
+#[derive(PartialEq, Eq, Hash)]
+struct TierFreeVersionReceiptIdentity {
+    bucket: String,
+    logical_name: String,
+    tier: String,
+    remote_name: String,
+    remote_version_state: TierFreeVersionReceiptVersionState,
+    remote_version: String,
+    backend_identity: crate::services::tier::tier::TierDestinationId,
+}
+
+struct TierFreeVersionReceiptPayload {
+    local_free_version_id: Uuid,
+    mod_time: Option<OffsetDateTime>,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq, Hash)]
+enum TierFreeVersionReceiptVersionState {
+    KnownDisabled,
+    SuspendedNull,
+    Exact,
+}
+
+impl TierFreeVersionReceiptSink {
+    /// Only the delete wrapper may originate a sink. The public type exists so
+    /// workspace struct literals can carry it, but external crates cannot
+    /// create an undrainable collector accidentally.
+    pub(crate) fn new() -> Self {
+        Self {
+            inner: Arc::new(parking_lot::Mutex::new(TierFreeVersionReceiptSinkState {
+                receipts: Some(HashMap::new()),
+            })),
+        }
+    }
+}
+
+impl std::fmt::Debug for TierFreeVersionReceiptSink {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        let state = self.inner.lock();
+        f.debug_struct("TierFreeVersionReceiptSink")
+            .field("drained", &state.receipts.is_none())
+            .field("receipt_count", &state.receipts.as_ref().map(HashMap::len).unwrap_or_default())
+            .finish()
+    }
+}
+
+impl TierFreeVersionReceiptVersionState {
+    fn persisted(self) -> rustfs_filemeta::TransitionVersionState {
+        match self {
+            Self::KnownDisabled => rustfs_filemeta::TransitionVersionState::KnownDisabled,
+            Self::SuspendedNull => rustfs_filemeta::TransitionVersionState::SuspendedNull,
+            Self::Exact => rustfs_filemeta::TransitionVersionState::Exact,
+        }
+    }
+}
+
+impl TierFreeVersionReceiptIdentity {
+    fn into_object_info(self, payload: TierFreeVersionReceiptPayload) -> ObjectInfo {
+        let mut metadata = HashMap::with_capacity(2);
+        rustfs_utils::http::metadata_compat::insert_str(
+            &mut metadata,
+            rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
+            rustfs_utils::crypto::hex(self.backend_identity),
+        );
+        ObjectInfo {
+            bucket: self.bucket,
+            name: self.logical_name,
+            mod_time: payload.mod_time,
+            user_defined: Arc::new(metadata),
+            version_id: Some(payload.local_free_version_id),
+            delete_marker: true,
+            transitioned_object: TransitionedObject {
+                name: self.remote_name,
+                version_id: self.remote_version,
+                tier: self.tier,
+                free_version: true,
+                status: String::new(),
+            },
+            transition_version_state: self.remote_version_state.persisted(),
+            ..Default::default()
+        }
+    }
+}
+
+fn tier_free_version_scheduling_receipt_from_source(
+    source: &ObjectInfo,
+    local_free_version_id: Uuid,
+) -> io::Result<Option<(TierFreeVersionReceiptIdentity, TierFreeVersionReceiptPayload)>> {
+    if source.transitioned_object.status != rustfs_filemeta::TRANSITION_COMPLETE
+        || source.transitioned_object.free_version
+        || source.delete_marker
+        || source.bucket.is_empty()
+        || source.name.is_empty()
+        || source.transitioned_object.tier.is_empty()
+        || source.transitioned_object.name.is_empty()
+    {
+        return Ok(None);
+    }
+    if local_free_version_id.is_nil() {
+        return Err(io::Error::new(
+            io::ErrorKind::InvalidInput,
+            "tier free-version receipt has a nil local version identity",
+        ));
+    }
+
+    let remote_version = source.transitioned_object.version_id.as_str();
+    let remote_version_state = match source.transition_version_state {
+        rustfs_filemeta::TransitionVersionState::Unknown => return Ok(None),
+        rustfs_filemeta::TransitionVersionState::KnownDisabled if remote_version.is_empty() => {
+            TierFreeVersionReceiptVersionState::KnownDisabled
+        }
+        rustfs_filemeta::TransitionVersionState::SuspendedNull if remote_version == "null" => {
+            TierFreeVersionReceiptVersionState::SuspendedNull
+        }
+        rustfs_filemeta::TransitionVersionState::Exact if !remote_version.is_empty() && remote_version != "null" => {
+            TierFreeVersionReceiptVersionState::Exact
+        }
+        _ => return Ok(None),
+    };
+
+    let Some(backend_identity) = crate::services::tier::tier::tier_destination_id_from_metadata(&source.user_defined)
+        .map_err(|err| io::Error::new(io::ErrorKind::InvalidData, err))?
+    else {
+        return Ok(None);
+    };
+
+    Ok(Some((
+        TierFreeVersionReceiptIdentity {
+            bucket: source.bucket.clone(),
+            logical_name: decode_dir_object(&source.name),
+            tier: source.transitioned_object.tier.clone(),
+            remote_name: source.transitioned_object.name.clone(),
+            remote_version_state,
+            remote_version: source.transitioned_object.version_id.clone(),
+            backend_identity,
+        },
+        TierFreeVersionReceiptPayload {
+            local_free_version_id,
+            mod_time: source.mod_time,
+        },
+    )))
+}
+
+impl TierFreeVersionReceiptSink {
+    /// Record one committed free-version cleanup target. Cloned options share
+    /// this sink; tuple-equivalent physical copies collapse to one worker task.
+    /// `false` means the source cannot safely identify a destructive cleanup.
+    pub(crate) fn record(&self, source: &ObjectInfo, local_free_version_id: Uuid) -> io::Result<bool> {
+        let Some((identity, payload)) = tier_free_version_scheduling_receipt_from_source(source, local_free_version_id)? else {
+            return Ok(false);
+        };
+        let mut state = self.inner.lock();
+        let receipts = state
+            .receipts
+            .as_mut()
+            .ok_or_else(|| io::Error::new(io::ErrorKind::BrokenPipe, "tier free-version receipt sink was already drained"))?;
+        receipts.entry(identity).or_insert(payload);
+        Ok(true)
+    }
+
+    /// Consume every receipt exactly once. A second drain is a caller bug: it
+    /// could otherwise make two outer wrappers believe they own the same tasks.
+    pub(crate) fn drain(&self) -> io::Result<Vec<ObjectInfo>> {
+        let mut state = self.inner.lock();
+        let receipts = state
+            .receipts
+            .take()
+            .ok_or_else(|| io::Error::new(io::ErrorKind::BrokenPipe, "tier free-version receipt sink was already drained"))?;
+        drop(state);
+        Ok(receipts
+            .into_iter()
+            .map(|(identity, payload)| identity.into_object_info(payload))
+            .collect())
+    }
+}
+
 #[derive(Default, Clone)]
 pub struct ObjectOptions {
     // Use the maximum parity (N/2), used when saving server configuration files
@@ -716,6 +907,12 @@ pub struct ObjectOptions {
     pub skip_rebalancing: bool,
     pub skip_free_version: bool,
 
+    /// Storage-owned, per-request hand-off for committed tier free-version
+    /// cleanup work. The outer delete wrapper installs and drains it; clones
+    /// below that boundary share the same opaque sink.
+    #[doc(hidden)]
+    pub tier_free_version_receipt_sink: Option<TierFreeVersionReceiptSink>,
+
     /// Cooperative cancellation for an owned PutObject before authoritative
     /// rename begins. Storage ignores it after entering the durable commit.
     #[doc(hidden)]
@@ -851,6 +1048,7 @@ impl std::fmt::Debug for ObjectOptions {
             .field("skip_decommissioned", &self.skip_decommissioned)
             .field("skip_rebalancing", &self.skip_rebalancing)
             .field("skip_free_version", &self.skip_free_version)
+            .field("tier_free_version_receipt_sink", &self.tier_free_version_receipt_sink)
             .field("put_object_cancellation", &self.put_object_cancellation.is_some())
             .field("scanner_publication_commit_scope", &self.scanner_publication_commit_scope)
             .field("data_movement", &self.data_movement)
@@ -2622,11 +2820,381 @@ mod tests {
         assert!(default_cloned.parts.is_empty());
     }
 
+    fn transitioned_receipt_source(
+        bucket: &str,
+        object: &str,
+        remote_version: &str,
+        version_state: rustfs_filemeta::TransitionVersionState,
+        identity_hex: Option<&str>,
+    ) -> ObjectInfo {
+        let mut metadata = HashMap::new();
+        if let Some(identity_hex) = identity_hex {
+            rustfs_utils::http::metadata_compat::insert_str(
+                &mut metadata,
+                rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
+                identity_hex.to_string(),
+            );
+        }
+        ObjectInfo {
+            bucket: bucket.to_string(),
+            name: object.to_string(),
+            mod_time: Some(OffsetDateTime::UNIX_EPOCH),
+            user_defined: Arc::new(metadata),
+            transitioned_object: TransitionedObject {
+                name: format!("remote/{object}"),
+                version_id: remote_version.to_string(),
+                tier: "WARM".to_string(),
+                status: TRANSITION_COMPLETE.to_string(),
+                ..Default::default()
+            },
+            transition_version_state: version_state,
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn tier_free_version_receipt_matches_persisted_free_version_worker_fields() {
+        let bucket = "receipt-bucket";
+        let object = "archive/object.bin";
+        let source_version_id = Uuid::from_u128(1);
+        let local_free_version_id = Uuid::from_u128(2);
+        let remote_version_id = Uuid::from_u128(3);
+        let source_mod_time = OffsetDateTime::UNIX_EPOCH + time::Duration::hours(4);
+        let identity_hex = "ab".repeat(32);
+        let mut source_metadata = HashMap::from([
+            ("etag".to_string(), "source-etag".to_string()),
+            ("x-amz-meta-private".to_string(), "must-not-enter-receipt".to_string()),
+        ]);
+        rustfs_utils::http::metadata_compat::insert_str(
+            &mut source_metadata,
+            rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
+            identity_hex.clone(),
+        );
+        let source_file_info = FileInfo {
+            volume: bucket.to_string(),
+            name: object.to_string(),
+            version_id: Some(source_version_id),
+            transition_status: TRANSITION_COMPLETE.to_string(),
+            transitioned_objname: "remote/receipt-object".to_string(),
+            transition_tier: "WARM".to_string(),
+            transition_version_id: Some(remote_version_id),
+            transition_version: Some(remote_version_id.to_string()),
+            transition_version_state: rustfs_filemeta::TransitionVersionState::Exact,
+            mod_time: Some(source_mod_time),
+            size: 8192,
+            data_dir: Some(Uuid::from_u128(4)),
+            metadata: source_metadata,
+            ..Default::default()
+        };
+        let source = ObjectInfo::from_file_info(&source_file_info, bucket, object, true);
+        let mut persisted = FileMeta::new();
+        persisted
+            .add_version(source_file_info)
+            .expect("transitioned receipt source should be persisted");
+        let mut delete_file_info = FileInfo {
+            volume: bucket.to_string(),
+            name: object.to_string(),
+            version_id: Some(source_version_id),
+            mod_time: Some(source_mod_time + time::Duration::minutes(1)),
+            ..Default::default()
+        };
+        delete_file_info.set_tier_free_version_id(&local_free_version_id.to_string());
+        persisted
+            .delete_version(&delete_file_info)
+            .expect("transitioned source delete should create a free-version");
+
+        let encoded = persisted.marshal_msg().expect("free-version metadata should encode");
+        let decoded = FileMeta::load(&encoded).expect("free-version metadata should decode");
+        let persisted_free_version = decoded
+            .get_all_file_info_versions(bucket, object, true)
+            .expect("decoded free-version should produce FileInfo")
+            .versions
+            .into_iter()
+            .find(|version| version.tier_free_version())
+            .expect("decoded metadata should contain the persisted free-version");
+        let persisted_object_info = ObjectInfo::from_file_info(&persisted_free_version, bucket, object, true);
+
+        let sink = TierFreeVersionReceiptSink::new();
+        assert!(
+            sink.record(&source, local_free_version_id)
+                .expect("valid transitioned source should produce a receipt")
+        );
+        let mut receipts = sink.drain().expect("receipt owner should drain exactly once");
+        assert_eq!(receipts.len(), 1);
+        let receipt = receipts.pop().expect("one receipt should be present");
+
+        assert_eq!(receipt.bucket, persisted_object_info.bucket);
+        assert_eq!(receipt.name, persisted_object_info.name);
+        assert_eq!(receipt.version_id, persisted_object_info.version_id);
+        assert_eq!(receipt.mod_time, persisted_object_info.mod_time);
+        assert_eq!(receipt.delete_marker, persisted_object_info.delete_marker);
+        assert_eq!(receipt.transitioned_object.name, persisted_object_info.transitioned_object.name);
+        assert_eq!(
+            receipt.transitioned_object.version_id,
+            persisted_object_info.transitioned_object.version_id
+        );
+        assert_eq!(receipt.transitioned_object.tier, persisted_object_info.transitioned_object.tier);
+        assert_eq!(
+            receipt.transitioned_object.free_version,
+            persisted_object_info.transitioned_object.free_version
+        );
+        assert_eq!(receipt.transitioned_object.status, persisted_object_info.transitioned_object.status);
+        assert_eq!(receipt.transition_version_state, persisted_object_info.transition_version_state);
+        assert_eq!(
+            crate::services::tier::tier::tier_destination_id_from_metadata(&receipt.user_defined)
+                .expect("receipt identity should decode"),
+            crate::services::tier::tier::tier_destination_id_from_metadata(&persisted_object_info.user_defined)
+                .expect("persisted identity should decode")
+        );
+        assert_eq!(
+            receipt.user_defined.len(),
+            2,
+            "receipt should carry only the two compatibility identity keys"
+        );
+        assert_eq!(
+            receipt.user_defined.get("x-rustfs-internal-transition-tier-destination-id"),
+            Some(&identity_hex)
+        );
+        assert_eq!(
+            receipt.user_defined.get("x-minio-internal-transition-tier-destination-id"),
+            Some(&identity_hex)
+        );
+        assert!(!receipt.user_defined.contains_key("x-amz-meta-private"));
+        assert_eq!(receipt.size, 0);
+        assert_eq!(receipt.actual_size, 0);
+        assert!(receipt.parts.is_empty());
+        assert!(receipt.etag.is_none());
+        assert!(receipt.checksum.is_none());
+        assert!(receipt.data_dir.is_none());
+    }
+
+    #[test]
+    fn tier_free_version_receipt_sink_deduplicates_remote_target_and_drains_once() {
+        let identity_hex = "11".repeat(32);
+        let source = transitioned_receipt_source(
+            "bucket",
+            "object",
+            "remote-version",
+            rustfs_filemeta::TransitionVersionState::Exact,
+            Some(&identity_hex),
+        );
+        let other_object = transitioned_receipt_source(
+            "bucket",
+            "other-object",
+            "remote-version",
+            rustfs_filemeta::TransitionVersionState::Exact,
+            Some(&identity_hex),
+        );
+        let sink = TierFreeVersionReceiptSink::new();
+        let clone = sink.clone();
+
+        assert!(
+            sink.record(&source, Uuid::from_u128(10))
+                .expect("first physical receipt should record")
+        );
+        assert!(
+            clone
+                .record(&source, Uuid::from_u128(11))
+                .expect("tuple-equivalent physical receipt should be represented")
+        );
+        assert!(
+            clone
+                .record(&other_object, Uuid::from_u128(12))
+                .expect("a different logical key should retain its own task")
+        );
+
+        let mut receipts = sink.drain().expect("owner should drain shared receipts");
+        receipts.sort_by(|left, right| left.name.cmp(&right.name));
+        assert_eq!(receipts.len(), 2);
+        assert_eq!(receipts[0].name, "object");
+        assert_eq!(receipts[0].version_id, Some(Uuid::from_u128(10)));
+        assert_eq!(receipts[1].name, "other-object");
+        assert_eq!(receipts[1].version_id, Some(Uuid::from_u128(12)));
+        assert_eq!(
+            clone.drain().expect_err("a shared sink must drain only once").kind(),
+            io::ErrorKind::BrokenPipe
+        );
+        assert_eq!(
+            clone
+                .record(&source, Uuid::from_u128(13))
+                .expect_err("recording after drain must fail")
+                .kind(),
+            io::ErrorKind::BrokenPipe
+        );
+    }
+
+    #[test]
+    fn tier_free_version_receipt_identity_covers_every_destructive_dimension() {
+        let identity_hex = "44".repeat(32);
+        let baseline = transitioned_receipt_source(
+            "bucket",
+            "directory/",
+            "remote-version",
+            rustfs_filemeta::TransitionVersionState::Exact,
+            Some(&identity_hex),
+        );
+        let mut encoded_duplicate = baseline.clone();
+        encoded_duplicate.name = "directory__XLDIR__".to_string();
+
+        let mut variants = Vec::new();
+        let mut changed = baseline.clone();
+        changed.bucket = "other-bucket".to_string();
+        variants.push(changed);
+        let mut changed = baseline.clone();
+        changed.name = "other-directory/".to_string();
+        variants.push(changed);
+        let mut changed = baseline.clone();
+        changed.transitioned_object.tier = "COLD".to_string();
+        variants.push(changed);
+        let mut changed = baseline.clone();
+        changed.transitioned_object.name = "remote/other-directory/".to_string();
+        variants.push(changed);
+        let mut changed = baseline.clone();
+        changed.transitioned_object.version_id = "other-remote-version".to_string();
+        variants.push(changed);
+        let mut changed = baseline.clone();
+        changed.transition_version_state = rustfs_filemeta::TransitionVersionState::SuspendedNull;
+        changed.transitioned_object.version_id = "null".to_string();
+        variants.push(changed);
+        let mut changed = baseline.clone();
+        rustfs_utils::http::metadata_compat::insert_str(
+            Arc::make_mut(&mut changed.user_defined),
+            rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TIER_DESTINATION_ID,
+            "55".repeat(32),
+        );
+        variants.push(changed);
+
+        let sink = TierFreeVersionReceiptSink::new();
+        assert!(
+            sink.record(&baseline, Uuid::from_u128(20))
+                .expect("baseline receipt should record")
+        );
+        assert!(
+            sink.record(&encoded_duplicate, Uuid::from_u128(21))
+                .expect("the encoded spelling of one logical key should deduplicate")
+        );
+        for (offset, variant) in variants.iter().enumerate() {
+            assert!(
+                sink.record(variant, Uuid::from_u128(30 + offset as u128))
+                    .expect("each distinct cleanup identity should record")
+            );
+        }
+
+        let receipts = sink.drain().expect("identity matrix should drain once");
+        assert_eq!(receipts.len(), 8, "every destructive identity dimension must prevent deduplication");
+        let baseline_receipt = receipts
+            .iter()
+            .find(|receipt| {
+                receipt.bucket == "bucket"
+                    && receipt.name == "directory/"
+                    && receipt.transitioned_object.tier == "WARM"
+                    && receipt.transitioned_object.name == "remote/directory/"
+                    && receipt.transitioned_object.version_id == "remote-version"
+                    && receipt.transition_version_state == rustfs_filemeta::TransitionVersionState::Exact
+                    && crate::services::tier::tier::tier_destination_id_from_metadata(&receipt.user_defined)
+                        .is_ok_and(|identity| identity == Some([0x44; 32]))
+            })
+            .expect("baseline cleanup identity should remain present");
+        assert_eq!(
+            baseline_receipt.version_id,
+            Some(Uuid::from_u128(20)),
+            "deduplication must retain the first UUID"
+        );
+    }
+
+    #[test]
+    fn tier_free_version_receipt_source_validation_fails_closed() {
+        let identity_hex = "22".repeat(32);
+        for (state, remote_version) in [
+            (rustfs_filemeta::TransitionVersionState::KnownDisabled, ""),
+            (rustfs_filemeta::TransitionVersionState::SuspendedNull, "null"),
+            (rustfs_filemeta::TransitionVersionState::Exact, "opaque-version"),
+        ] {
+            let source = transitioned_receipt_source("bucket", "object", remote_version, state, Some(&identity_hex));
+            assert!(
+                TierFreeVersionReceiptSink::new()
+                    .record(&source, Uuid::new_v4())
+                    .expect("canonical remote-version state should be eligible"),
+                "state={state:?} remote_version={remote_version:?}"
+            );
+        }
+
+        let unknown = transitioned_receipt_source(
+            "bucket",
+            "object",
+            "opaque-version",
+            rustfs_filemeta::TransitionVersionState::Unknown,
+            Some(&identity_hex),
+        );
+        assert!(
+            !TierFreeVersionReceiptSink::new()
+                .record(&unknown, Uuid::new_v4())
+                .expect("unknown remote version state should defer to recovery")
+        );
+        let missing_identity = transitioned_receipt_source(
+            "bucket",
+            "object",
+            "opaque-version",
+            rustfs_filemeta::TransitionVersionState::Exact,
+            None,
+        );
+        assert!(
+            !TierFreeVersionReceiptSink::new()
+                .record(&missing_identity, Uuid::new_v4())
+                .expect("missing durable identity should defer to recovery")
+        );
+        let invalid_exact = transitioned_receipt_source(
+            "bucket",
+            "object",
+            "",
+            rustfs_filemeta::TransitionVersionState::Exact,
+            Some(&identity_hex),
+        );
+        assert!(
+            !TierFreeVersionReceiptSink::new()
+                .record(&invalid_exact, Uuid::new_v4())
+                .expect("conflicting remote state should defer to recovery")
+        );
+
+        let mut conflicting = transitioned_receipt_source(
+            "bucket",
+            "object",
+            "opaque-version",
+            rustfs_filemeta::TransitionVersionState::Exact,
+            Some(&identity_hex),
+        );
+        Arc::make_mut(&mut conflicting.user_defined)
+            .insert("x-minio-internal-transition-tier-destination-id".to_string(), "33".repeat(32));
+        assert_eq!(
+            TierFreeVersionReceiptSink::new()
+                .record(&conflicting, Uuid::new_v4())
+                .expect_err("conflicting identity aliases must fail closed")
+                .kind(),
+            io::ErrorKind::InvalidData
+        );
+
+        let valid = transitioned_receipt_source(
+            "bucket",
+            "object",
+            "opaque-version",
+            rustfs_filemeta::TransitionVersionState::Exact,
+            Some(&identity_hex),
+        );
+        assert_eq!(
+            TierFreeVersionReceiptSink::new()
+                .record(&valid, Uuid::nil())
+                .expect_err("nil local free-version identity must be rejected")
+                .kind(),
+            io::ErrorKind::InvalidInput
+        );
+    }
+
     #[test]
     fn object_options_default_does_not_allocate_lifecycle_delete_all_journal() {
         let mut opts = ObjectOptions::default();
 
         assert!(opts.lifecycle_delete_all_journal().is_none());
+        assert!(opts.tier_free_version_receipt_sink.is_none());
         opts.ensure_lifecycle_delete_all_journal();
         assert!(opts.lifecycle_delete_all_journal().is_some());
     }

crates/ecstore/src/set_disk/ops/object.rs

@@ -217,7 +217,8 @@ use crate::bucket::lifecycle::{
 use crate::bucket::quota::reservation;
 use crate::bucket::replication::{
     DeleteReplicationConfigSnapshot, ReplicationLifecycleBridge, ReplicationStatusType, VersionPurgeStatusType,
-    replication_state_to_filemeta, replication_status_from_filemeta, version_purge_status_to_filemeta,
+    replication_state_to_filemeta, replication_status_from_filemeta, version_purge_status_from_filemeta,
+    version_purge_status_to_filemeta,
 };
 use crate::data_usage::quota_object_size;
 use crate::diagnostics::get::GetObjectFailureReason;
@@ -283,12 +284,95 @@ fn record_transitioned_delete_cleanup_owner(bucket: &str, object: &str, batch: b
     );
 }
 
-async fn acquire_single_tier_delete_lease(
+/// A causal free-version receipt is only valid when the delete request really
+/// removes the locked transitioned source. In particular, replication may turn
+/// an otherwise successful delete into a metadata-only purge-state update, and
+/// a versioned delete without a version ID writes a new delete marker instead
+/// of removing the source selected by `goi`.
+fn transitioned_delete_publishes_free_version(source: &ObjectInfo, delete_request: &FileInfo, skip_free_version: bool) -> bool {
+    if source.delete_marker
+        || source.transitioned_object.status != TRANSITION_COMPLETE
+        || skip_free_version
+        || delete_request.skip_tier_free_version()
+        || delete_request.expire_restored
+        || delete_request.transition_status == TRANSITION_COMPLETE
+        || delete_file_info_version_id(source.version_id) != delete_request.version_id
+    {
+        return false;
+    }
+
+    // Keep this predicate aligned with FileMeta::delete_version's Object
+    // branch: a non-delete-marker request with a nonterminal purge status (or
+    // mark_deleted with no purge status) updates replication metadata in place
+    // and never calls MetaObject::init_free_version.
+    let purge_status = version_purge_status_from_filemeta(delete_request.version_purge_status());
+    let metadata_only = !delete_request.deleted
+        && ((purge_status.is_empty() && delete_request.mark_deleted)
+            || (!purge_status.is_empty() && purge_status != VersionPurgeStatusType::Complete));
+
+    !metadata_only
+}
+
+fn record_committed_tier_free_version_receipt(
+    opts: &ObjectOptions,
     bucket: &str,
     object: &str,
-    opts: &ObjectOptions,
     source: &ObjectInfo,
-) -> Result<Option<TierOperationLease>> {
+    free_version_id: Uuid,
+    batch: bool,
+) {
+    if let Some(sink) = opts.tier_free_version_receipt_sink.as_ref()
+        && let Err(err) = sink.record(source, free_version_id)
+    {
+        warn!(
+            event = EVENT_LIFECYCLE_TRANSITIONED_DELETE_CLEANUP_OWNER,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_SET_DISK,
+            bucket,
+            object,
+            batch,
+            error = ?err,
+            "Failed to retain the in-memory tier free-version scheduling receipt"
+        );
+    }
+    record_transitioned_delete_cleanup_owner(bucket, object, batch);
+}
+
+struct TierFreeVersionReceiptCandidate {
+    source: ObjectInfo,
+    free_version_id: Uuid,
+}
+
+fn committed_tier_free_version_receipt_indices(
+    versions: &[FileInfoVersions],
+    delete_errors: &[Option<Error>],
+    candidates: &HashMap<usize, TierFreeVersionReceiptCandidate>,
+) -> Vec<usize> {
+    if candidates.is_empty() {
+        return Vec::new();
+    }
+
+    let mut committed = Vec::with_capacity(candidates.len());
+    for group in versions {
+        let should_rollback = group
+            .versions
+            .iter()
+            .any(|version| delete_errors.get(version.idx).is_none_or(|error| error.is_some()));
+        if should_rollback {
+            continue;
+        }
+        committed.extend(
+            group
+                .versions
+                .iter()
+                .map(|version| version.idx)
+                .filter(|idx| candidates.contains_key(idx)),
+        );
+    }
+    committed
+}
+
+async fn acquire_single_tier_delete_lease(opts: &ObjectOptions, source: &ObjectInfo) -> Result<Option<TierOperationLease>> {
     let Some(api) = opts.tier_delete_journal_api.as_ref() else {
         return Ok(None);
     };
@@ -308,7 +392,6 @@ async fn acquire_single_tier_delete_lease(
         None => TierConfigMgr::acquire_operation_lease(&api.tier_config_mgr(), &source.transitioned_object.tier).await,
     }
     .map_err(Error::other)?;
-    record_transitioned_delete_cleanup_owner(bucket, object, false);
     Ok(Some(lease))
 }
 
@@ -384,6 +467,168 @@ mod scanner_publication_lease_fence_tests {
     }
 }
 
+#[cfg(test)]
+mod tier_free_version_receipt_eligibility_tests {
+    use super::*;
+    use crate::bucket::replication::ReplicationState;
+
+    fn transitioned_source(version_id: Option<Uuid>) -> ObjectInfo {
+        let mut source = ObjectInfo {
+            version_id,
+            ..Default::default()
+        };
+        source.transitioned_object.status = TRANSITION_COMPLETE.to_string();
+        source.transitioned_object.tier = "WARM".to_string();
+        source.transitioned_object.name = "remote/object".to_string();
+        source
+    }
+
+    fn delete_request(version_id: Option<Uuid>) -> FileInfo {
+        let mut request = FileInfo {
+            version_id,
+            ..Default::default()
+        };
+        request.set_tier_free_version_id(&Uuid::new_v4().to_string());
+        request
+    }
+
+    #[test]
+    fn accepts_exact_transitioned_source_removal_and_suspended_null_replacement() {
+        let version_id = Uuid::new_v4();
+        assert!(transitioned_delete_publishes_free_version(
+            &transitioned_source(Some(version_id)),
+            &delete_request(Some(version_id)),
+            false,
+        ));
+
+        let mut suspended_null_delete = delete_request(None);
+        suspended_null_delete.deleted = true;
+        suspended_null_delete.mark_deleted = true;
+        assert!(transitioned_delete_publishes_free_version(
+            &transitioned_source(Some(Uuid::nil())),
+            &suspended_null_delete,
+            false,
+        ));
+    }
+
+    #[test]
+    fn rejects_new_marker_version_and_non_transitioned_or_delete_marker_sources() {
+        let source_id = Uuid::new_v4();
+        assert!(!transitioned_delete_publishes_free_version(
+            &transitioned_source(Some(source_id)),
+            &delete_request(Some(Uuid::new_v4())),
+            false,
+        ));
+
+        let mut ordinary = transitioned_source(Some(source_id));
+        ordinary.transitioned_object.status.clear();
+        assert!(!transitioned_delete_publishes_free_version(
+            &ordinary,
+            &delete_request(Some(source_id)),
+            false,
+        ));
+
+        let mut delete_marker = transitioned_source(Some(source_id));
+        delete_marker.delete_marker = true;
+        assert!(!transitioned_delete_publishes_free_version(
+            &delete_marker,
+            &delete_request(Some(source_id)),
+            false,
+        ));
+    }
+
+    #[test]
+    fn rejects_skip_restore_and_transition_metadata_updates() {
+        let version_id = Uuid::new_v4();
+        let source = transitioned_source(Some(version_id));
+
+        assert!(!transitioned_delete_publishes_free_version(
+            &source,
+            &delete_request(Some(version_id)),
+            true,
+        ));
+
+        let mut skip_request = delete_request(Some(version_id));
+        skip_request.set_skip_tier_free_version();
+        assert!(!transitioned_delete_publishes_free_version(&source, &skip_request, false));
+
+        let mut restore_request = delete_request(Some(version_id));
+        restore_request.expire_restored = true;
+        assert!(!transitioned_delete_publishes_free_version(&source, &restore_request, false));
+
+        let mut transition_update = delete_request(Some(version_id));
+        transition_update.transition_status = TRANSITION_COMPLETE.to_string();
+        assert!(!transitioned_delete_publishes_free_version(&source, &transition_update, false));
+    }
+
+    #[test]
+    fn rejects_nonterminal_replication_metadata_only_update_but_accepts_complete_purge() {
+        let version_id = Uuid::new_v4();
+        let source = transitioned_source(Some(version_id));
+
+        let mut pending = delete_request(Some(version_id));
+        pending.replication_state_internal = Some(replication_state_to_filemeta(&ReplicationState {
+            version_purge_status_internal: Some("PENDING".to_string()),
+            ..Default::default()
+        }));
+        assert!(!transitioned_delete_publishes_free_version(&source, &pending, false));
+
+        let mut mark_deleted = delete_request(Some(version_id));
+        mark_deleted.mark_deleted = true;
+        assert!(!transitioned_delete_publishes_free_version(&source, &mark_deleted, false));
+
+        let mut complete = delete_request(Some(version_id));
+        complete.replication_state_internal = Some(replication_state_to_filemeta(&ReplicationState {
+            version_purge_status_internal: Some("COMPLETE".to_string()),
+            ..Default::default()
+        }));
+        assert!(transitioned_delete_publishes_free_version(&source, &complete, false));
+    }
+
+    #[test]
+    fn whole_physical_object_group_must_commit_before_any_receipt_is_retained() {
+        let candidate = || TierFreeVersionReceiptCandidate {
+            source: transitioned_source(Some(Uuid::new_v4())),
+            free_version_id: Uuid::new_v4(),
+        };
+        let candidates = HashMap::from([(0, candidate()), (2, candidate())]);
+        let versions = vec![
+            FileInfoVersions {
+                versions: vec![
+                    FileInfo {
+                        idx: 0,
+                        ..Default::default()
+                    },
+                    FileInfo {
+                        idx: 1,
+                        ..Default::default()
+                    },
+                ],
+                ..Default::default()
+            },
+            FileInfoVersions {
+                versions: vec![FileInfo {
+                    idx: 2,
+                    ..Default::default()
+                }],
+                ..Default::default()
+            },
+        ];
+        let one_sibling_failed = vec![None, Some(Error::other("injected quorum failure")), None];
+
+        assert_eq!(
+            committed_tier_free_version_receipt_indices(&versions, &one_sibling_failed, &candidates),
+            vec![2],
+            "a sibling failure must suppress every receipt from the rolled-back xl.meta group"
+        );
+        assert_eq!(
+            committed_tier_free_version_receipt_indices(&versions, &[None, None, None], &candidates),
+            vec![0, 2],
+            "independent fully committed groups should retain their sparse receipts"
+        );
+    }
+}
+
 struct PutObjectCommitCancellation {
     token: CancellationToken,
     armed: bool,
@@ -6977,7 +7222,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         };
         let mut vers_map: HashMap<&String, FileInfoVersions> = HashMap::new();
         let mut tier_reference_leases: Vec<(usize, String, Option<TierDestinationId>)> = Vec::new();
-        let mut transitioned_cleanup_items = vec![false; objects.len()];
+        let mut tier_free_version_receipt_candidates: HashMap<usize, TierFreeVersionReceiptCandidate> = HashMap::new();
 
         for (i, dobj) in objects.iter().enumerate() {
             if del_errs[i].is_some() {
@@ -7069,7 +7314,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             {
                 match tier_destination_id_from_metadata(&goi.user_defined) {
                     Ok(identity) => {
-                        transitioned_cleanup_items[i] = true;
                         tier_reference_leases.push((i, goi.transitioned_object.tier.clone(), identity));
                     }
                     Err(err) => {
@@ -7110,7 +7354,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 ..Default::default()
             };
 
-            vr.set_tier_free_version_id(&Uuid::new_v4().to_string());
+            let tier_free_version_id = Uuid::new_v4();
+            vr.set_tier_free_version_id(&tier_free_version_id.to_string());
 
             // Delete
             // del_objects[i].object_name.clone_from(&vr.name);
@@ -7195,6 +7440,19 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 });
             }
 
+            if opts.tier_free_version_receipt_sink.is_some()
+                && !dobj.synthetic_version_id
+                && transitioned_delete_publishes_free_version(&goi, &vr, opts.skip_free_version)
+            {
+                tier_free_version_receipt_candidates.insert(
+                    i,
+                    TierFreeVersionReceiptCandidate {
+                        source: goi,
+                        free_version_id: tier_free_version_id,
+                    },
+                );
+            }
+
             // Only add to vers_map if we hold the lock
             if locked_objects.contains(&dobj.object_name) {
                 vers_map.insert(&dobj.object_name, v);
@@ -7271,12 +7529,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             }
         }
 
-        for (idx, transitioned) in transitioned_cleanup_items.into_iter().enumerate() {
-            if transitioned && del_errs[idx].is_none() {
-                record_transitioned_delete_cleanup_owner(bucket, &decode_dir_object(&objects[idx].object_name), true);
-            }
-        }
-
         // Keep backend generations pinned through the source mutation, its
         // free-version write quorum, and any local rollback. Ordinary
         // single/batch deletes never transfer cleanup ownership to a journal.
@@ -7388,6 +7640,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
 
         let mut rollback_futures = Vec::new();
+        let committed_receipt_indices =
+            committed_tier_free_version_receipt_indices(&vers, &del_errs, &tier_free_version_receipt_candidates);
         for fi_vers in &vers {
             // delete_versions commits one xl.meta per object group, so rollback must use the same boundary.
             let should_rollback = fi_vers.versions.iter().any(|fi| del_errs[fi.idx].is_some());
@@ -7462,6 +7716,20 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
 
         join_all(rollback_futures).await;
 
+        for idx in committed_receipt_indices {
+            let Some(candidate) = tier_free_version_receipt_candidates.remove(&idx) else {
+                continue;
+            };
+            record_committed_tier_free_version_receipt(
+                &opts,
+                bucket,
+                &decode_dir_object(&objects[idx].object_name),
+                &candidate.source,
+                candidate.free_version_id,
+                true,
+            );
+        }
+
         // TODO(backlog): support partial object deletion for multi-part objects
 
         if dist_erasure {
@@ -7813,7 +8081,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
 
             ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?;
             begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?;
-            let _tier_delete_lease = acquire_single_tier_delete_lease(bucket, object, &opts, &goi).await?;
+            let _tier_delete_lease = acquire_single_tier_delete_lease(&opts, &goi).await?;
             if opts.skip_free_version {
                 fi.set_skip_tier_free_version();
             }
@@ -7823,6 +8091,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             #[cfg(test)]
             pause_delete_object_commit_after_publish(bucket, object).await;
 
+            if opts.tier_free_version_receipt_sink.is_some()
+                && transitioned_delete_publishes_free_version(&goi, &fi, opts.skip_free_version)
+            {
+                record_committed_tier_free_version_receipt(&opts, bucket, object, &goi, find_vid, false);
+            }
+
             let disks = self.disk_inventory().await;
             self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
 
@@ -7855,7 +8129,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
 
         ensure_delete_commit_locks_held(_lock_guard.as_ref(), bucket, object, &opts)?;
         begin_scanner_publication_delete_mutation(scanner_publication_commit_scope.as_ref())?;
-        let _tier_delete_lease = acquire_single_tier_delete_lease(bucket, object, &opts, &goi).await?;
+        let _tier_delete_lease = acquire_single_tier_delete_lease(&opts, &goi).await?;
         if opts.skip_free_version {
             dfi.set_skip_tier_free_version();
         }
@@ -7865,6 +8139,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
         #[cfg(test)]
         pause_delete_object_commit_after_publish(bucket, object).await;
 
+        if opts.tier_free_version_receipt_sink.is_some()
+            && transitioned_delete_publishes_free_version(&goi, &dfi, opts.skip_free_version)
+        {
+            record_committed_tier_free_version_receipt(&opts, bucket, object, &goi, find_vid, false);
+        }
+
         let disks = self.disk_inventory().await;
         self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
 

crates/ecstore/src/store/init.rs

@@ -3611,6 +3611,7 @@ mod tests {
                     assert_eq!(body, multipart_target_body);
 
                     let retry_object = "multipart-retry-object";
+                    let retry_object_etag = "0123456789abcdef0123456789abcdef".to_string();
                     let retry_first_part_size = 5 * 1024 * 1024;
                     let retry_object_mod_time =
                         OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("fixed retry timestamp should be valid");
@@ -3700,6 +3701,7 @@ mod tests {
                             &ObjectOptions {
                                 mod_time: Some(retry_object_mod_time),
                                 want_checksum: Some(retry_object_checksum),
+                                preserve_etag: Some(retry_object_etag.clone()),
                                 ..Default::default()
                             },
                         )
@@ -3732,7 +3734,7 @@ mod tests {
                         part.checksums = Some(HashMap::from([(ChecksumType::CRC32C.to_string(), checksum.encoded.clone())]));
                     }
                     retry_source_info.parts = Arc::new(retry_source_parts);
-                    retry_source_info.etag = Some("0123456789abcdef0123456789abcdef".to_string());
+                    assert_eq!(retry_source_info.etag.as_deref(), Some(retry_object_etag.as_str()));
                     assert!(!retry_source_info.is_multipart());
                     assert!(retry_source_info.parts.iter().all(|part| part.checksums.is_some()));
                     assert_eq!(retry_source_info.checksum.as_deref(), Some(retry_object_checksum_bytes.as_ref()));
@@ -10494,6 +10496,8 @@ mod tests {
         bucket: &str,
         object: &str,
         restore_before_delete: bool,
+        causal_enqueue: bool,
+        delete_with_journal: bool,
     ) {
         let temp_dir = tempfile::tempdir().expect("create transitioned delete store dir");
         let (ctx, store, _shutdown) =
@@ -10557,11 +10561,51 @@ mod tests {
             );
         }
 
-        backend.set_remove_failure(true);
-        store
-            .delete_object_with_tier_delete_journal(bucket, object, ObjectOptions::default())
+        if causal_enqueue {
+            ExpiryState::resize_workers(1, store.clone()).await;
+        }
+        backend.set_remove_failure(!causal_enqueue);
+        if delete_with_journal {
+            store
+                .delete_object_with_tier_delete_journal(bucket, object, ObjectOptions::default())
+                .await
+                .expect("transitioned source journal-wrapper delete should commit");
+        } else {
+            store
+                .delete_object(bucket, object, ObjectOptions::default())
+                .await
+                .expect("transitioned source plain object-layer delete should commit");
+        }
+
+        if causal_enqueue {
+            tokio::time::timeout(Duration::from_secs(30), async {
+                loop {
+                    let metadata_absent = store.pools[0]
+                        .get_disks_by_key(object)
+                        .load_file_info_versions_exact(bucket, object)
+                        .await
+                        .expect("causal free-version cleanup metadata should remain readable")
+                        .is_none();
+                    if metadata_absent && backend.remove_versions().await.len() == 1 {
+                        return;
+                    }
+                    tokio::time::sleep(Duration::from_millis(10)).await;
+                }
+            })
             .await
-            .expect("transitioned source delete should commit");
+            .expect("committed free-version should be cleaned without a recovery scan");
+            assert_eq!(backend.object_count().await, 0, "causal cleanup should remove the remote object");
+            assert_eq!(
+                tier_delete_journal_count(store.clone()).await,
+                0,
+                "ordinary causal cleanup must not create a journal"
+            );
+            store
+                .delete_bucket(bucket, &DeleteBucketOptions::default())
+                .await
+                .expect("bucket delete should succeed after causal free-version cleanup");
+            return;
+        }
 
         let local_versions = store.pools[0]
             .get_disks_by_key(object)
@@ -10640,6 +10684,8 @@ mod tests {
             "transitioned-delete-journal-owner-bucket",
             "transition/archive.bin",
             false,
+            false,
+            true,
         )
         .await;
     }
@@ -10654,6 +10700,40 @@ mod tests {
             "restored-transitioned-delete-journal-owner-bucket",
             "transition/archive.bin",
             true,
+            false,
+            true,
+        )
+        .await;
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn transitioned_delete_causally_enqueues_free_version() {
+        run_transitioned_delete_free_version_owner_case(
+            "transitioned-delete-causal-enqueue",
+            "DELETE-CAUSAL",
+            "transitioned-delete-causal-enqueue-bucket",
+            "transition/archive.bin",
+            false,
+            true,
+            false,
+        )
+        .await;
+    }
+
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn restored_transitioned_delete_causally_enqueues_free_version() {
+        run_transitioned_delete_free_version_owner_case(
+            "restored-transitioned-delete-causal-enqueue",
+            "RESTORE-DELETE-CAUSAL",
+            "restored-transitioned-delete-causal-enqueue-bucket",
+            "transition/archive.bin",
+            true,
+            true,
+            true,
         )
         .await;
     }
@@ -12333,12 +12413,262 @@ mod tests {
                 .is_none(),
             "free-version recovery must remove the exact cleanup owner"
         );
+
+        let causal = "causal.bin";
+        let mut causal_reader = PutObjReader::from_vec(vec![b'c'; 1024 * 1024]);
+        let causal_source = store
+            .put_object(bucket, causal, &mut causal_reader, &ObjectOptions::default())
+            .await
+            .expect("causal batch source should be written");
+        store
+            .transition_object(
+                bucket,
+                causal,
+                &ObjectOptions {
+                    transition: TransitionOptions {
+                        status: TRANSITION_PENDING.to_string(),
+                        tier: tier_name.to_string(),
+                        etag: causal_source.etag.clone().expect("causal batch source should have an etag"),
+                        ..Default::default()
+                    },
+                    mod_time: causal_source.mod_time,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("causal batch source transition should commit");
+        let (_deleted, errors) = store
+            .delete_objects(
+                bucket,
+                vec![
+                    ObjectToDelete {
+                        object_name: causal.to_string(),
+                        ..Default::default()
+                    },
+                    ObjectToDelete {
+                        object_name: causal.to_string(),
+                        ..Default::default()
+                    },
+                ],
+                ObjectOptions::default(),
+            )
+            .await;
+        assert!(
+            errors.iter().all(Option::is_none),
+            "duplicate causal batch deletes should remain idempotent: {errors:?}"
+        );
+        tokio::time::timeout(Duration::from_secs(30), async {
+            loop {
+                let metadata_absent = store.pools[0]
+                    .get_disks_by_key(causal)
+                    .load_file_info_versions_exact(bucket, causal)
+                    .await
+                    .expect("causal batch cleanup metadata should remain readable")
+                    .is_none();
+                if metadata_absent && backend.remove_versions().await.len() >= 2 {
+                    return;
+                }
+                tokio::time::sleep(Duration::from_millis(10)).await;
+            }
+        })
+        .await
+        .expect("batch free-version receipt should converge without another recovery scan");
+        assert_eq!(
+            backend.remove_versions().await.len(),
+            2,
+            "duplicate batch requests must cause only one remote delete for the causal object"
+        );
+
+        crate::bucket::metadata_sys::update_in(
+            &ctx,
+            bucket,
+            BUCKET_VERSIONING_CONFIG,
+            b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
+        )
+        .await
+        .expect("causal batch bucket versioning should be enabled");
+        let versioned_causal = "versioned-causal.bin";
+        let mut versioned_reader = PutObjReader::from_vec(vec![b'v'; 1024 * 1024]);
+        let versioned_source = store
+            .put_object(
+                bucket,
+                versioned_causal,
+                &mut versioned_reader,
+                &ObjectOptions {
+                    versioned: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("versioned causal batch source should be written");
+        let versioned_source_id = versioned_source
+            .version_id
+            .expect("versioned causal batch source should have an identity");
+        store
+            .transition_object(
+                bucket,
+                versioned_causal,
+                &ObjectOptions {
+                    version_id: Some(versioned_source_id.to_string()),
+                    versioned: true,
+                    transition: TransitionOptions {
+                        status: TRANSITION_PENDING.to_string(),
+                        tier: tier_name.to_string(),
+                        etag: versioned_source
+                            .etag
+                            .clone()
+                            .expect("versioned causal batch source should have an etag"),
+                        ..Default::default()
+                    },
+                    mod_time: versioned_source.mod_time,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("versioned causal batch source transition should commit");
+        let (_deleted, errors) = store
+            .delete_objects(
+                bucket,
+                vec![ObjectToDelete {
+                    object_name: versioned_causal.to_string(),
+                    version_id: Some(versioned_source_id),
+                    ..Default::default()
+                }],
+                ObjectOptions::default(),
+            )
+            .await;
+        assert!(
+            errors.iter().all(Option::is_none),
+            "explicit-version causal batch delete should commit: {errors:?}"
+        );
+        tokio::time::timeout(Duration::from_secs(30), async {
+            loop {
+                let metadata_absent = store.pools[0]
+                    .get_disks_by_key(versioned_causal)
+                    .load_file_info_versions_exact(bucket, versioned_causal)
+                    .await
+                    .expect("versioned causal batch cleanup metadata should remain readable")
+                    .is_none();
+                if metadata_absent && backend.remove_versions().await.len() == 3 {
+                    return;
+                }
+                tokio::time::sleep(Duration::from_millis(10)).await;
+            }
+        })
+        .await
+        .expect("explicit-version batch receipt should converge without a recovery scan");
         store
             .delete_bucket(bucket, &DeleteBucketOptions::default())
             .await
             .expect("batch source bucket should be physically empty");
     }
 
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn batch_transitioned_delete_aggregate_error_still_enqueues_committed_free_version() {
+        let temp_dir = tempfile::tempdir().expect("create aggregate-error batch delete store dir");
+        let (ctx, store, shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "batch-transitioned-aggregate-error", &[4, 4]))
+                .await;
+        crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let tier_name = "BATCH-AGGREGATE-ERROR";
+        let backend = register_mock_tier(&ctx.tier_config_mgr(), tier_name).await;
+        let bucket = "batch-transitioned-aggregate-error-bucket";
+        let object = "archive.bin";
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("aggregate-error source bucket should be created");
+        let mut reader = PutObjReader::from_vec(vec![b'a'; 1024 * 1024]);
+        let source = store.pools[0]
+            .put_object(bucket, object, &mut reader, &ObjectOptions::default())
+            .await
+            .expect("aggregate-error source should be written");
+        store.pools[0]
+            .transition_object(
+                bucket,
+                object,
+                &ObjectOptions {
+                    transition: TransitionOptions {
+                        status: TRANSITION_PENDING.to_string(),
+                        tier: tier_name.to_string(),
+                        etag: source.etag.clone().expect("aggregate-error source should have an etag"),
+                        ..Default::default()
+                    },
+                    mod_time: source.mod_time,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("aggregate-error source should transition");
+
+        // Model a data-movement copy: both pools own the same logical source
+        // and exact remote tuple, but each batch delete creates its own local
+        // free-version UUID in the shared request sink.
+        for disk_index in 0..4 {
+            let source_meta = temp_dir
+                .path()
+                .join(format!("pool0/set0/disk{disk_index}/{bucket}/{object}/{STORAGE_FORMAT_FILE}"));
+            let target_meta = temp_dir
+                .path()
+                .join(format!("pool1/set0/disk{disk_index}/{bucket}/{object}/{STORAGE_FORMAT_FILE}"));
+            tokio::fs::create_dir_all(target_meta.parent().expect("target xl.meta should have a parent"))
+                .await
+                .expect("second-pool object directory should be created");
+            tokio::fs::copy(&source_meta, &target_meta)
+                .await
+                .expect("transitioned xl.meta should copy exactly to the second pool");
+        }
+
+        ExpiryState::resize_workers(1, store.clone()).await;
+        let injection = crate::store::object::BatchDeletePoolErrorInjection::install(
+            bucket,
+            1,
+            vec![(object.to_string(), StorageError::ErasureWriteQuorum)],
+        );
+        let (deleted, errors) = store
+            .delete_objects(
+                bucket,
+                vec![ObjectToDelete {
+                    object_name: object.to_string(),
+                    ..Default::default()
+                }],
+                ObjectOptions::default(),
+            )
+            .await;
+        assert_eq!(injection.observed(), 1, "the second pool should inject one post-commit aggregate error");
+        assert_eq!(errors, vec![Some(StorageError::ErasureWriteQuorum)]);
+        assert!(deleted[0].found, "the aggregate error must retain the committed pool result");
+        drop(injection);
+
+        tokio::time::timeout(Duration::from_secs(30), async {
+            loop {
+                let mut metadata_absent = true;
+                for pool in &store.pools {
+                    metadata_absent &= pool
+                        .get_disks_by_key(object)
+                        .load_file_info_versions_exact(bucket, object)
+                        .await
+                        .expect("aggregate-error cleanup metadata should remain readable")
+                        .is_none();
+                }
+                if metadata_absent && backend.remove_count().await == 1 {
+                    return;
+                }
+                tokio::time::sleep(Duration::from_millis(10)).await;
+            }
+        })
+        .await
+        .expect("aggregate failure must not suppress committed receipt dispatch");
+        assert_eq!(backend.object_count().await, 0, "the shared remote object should be removed exactly once");
+        store
+            .delete_bucket(bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("aggregate-error bucket should be physically empty");
+        shutdown.cancel();
+    }
+
     #[cfg(feature = "test-util")]
     #[tokio::test]
     #[serial_test::serial(storage_class_env)]
@@ -12570,6 +12900,217 @@ mod tests {
             .expect("retry should leave the source bucket empty");
     }
 
+    #[cfg(feature = "test-util")]
+    #[tokio::test]
+    #[serial_test::serial(storage_class_env)]
+    async fn batch_transitioned_delete_post_commit_failures_roll_back_without_free_version_receipt() {
+        let temp_dir = tempfile::tempdir().expect("create failed batch delete store dir");
+        let (ctx, store, shutdown) =
+            without_storage_class_env(build_isolated_test_store(temp_dir.path(), "batch-delete-local-failure", &[4])).await;
+        crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let tier_name = "BATCH-DELETE-LOCAL-FAIL";
+        let backend = register_mock_tier(&ctx.tier_config_mgr(), tier_name).await;
+        let bucket = "batch-delete-local-failure-bucket";
+        let object = "archive.bin";
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("failed batch source bucket should be created");
+        crate::bucket::metadata_sys::update_in(
+            &ctx,
+            bucket,
+            BUCKET_VERSIONING_CONFIG,
+            b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
+        )
+        .await
+        .expect("failed batch bucket versioning should be enabled");
+        let mut transitioned_reader = PutObjReader::from_vec(vec![b't'; 1024 * 1024]);
+        let transitioned_source = store
+            .put_object(
+                bucket,
+                object,
+                &mut transitioned_reader,
+                &ObjectOptions {
+                    versioned: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("failed batch transitioned version should be written");
+        let transitioned_version_id = transitioned_source
+            .version_id
+            .expect("failed batch transitioned source should have a version identity");
+        store
+            .transition_object(
+                bucket,
+                object,
+                &ObjectOptions {
+                    version_id: Some(transitioned_version_id.to_string()),
+                    versioned: true,
+                    transition: TransitionOptions {
+                        status: TRANSITION_PENDING.to_string(),
+                        tier: tier_name.to_string(),
+                        etag: transitioned_source
+                            .etag
+                            .clone()
+                            .expect("failed batch transitioned source should have an etag"),
+                        ..Default::default()
+                    },
+                    mod_time: transitioned_source.mod_time,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("failed batch source version should transition");
+        let mut ordinary_reader = PutObjReader::from_vec(vec![b'o'; 1024 * 1024]);
+        let ordinary_source = store
+            .put_object(
+                bucket,
+                object,
+                &mut ordinary_reader,
+                &ObjectOptions {
+                    versioned: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("failed batch ordinary sibling should be written");
+        let ordinary_version_id = ordinary_source
+            .version_id
+            .expect("failed batch ordinary sibling should have a version identity");
+        let delete_requests = || {
+            vec![
+                ObjectToDelete {
+                    object_name: object.to_string(),
+                    version_id: Some(transitioned_version_id),
+                    ..Default::default()
+                },
+                ObjectToDelete {
+                    object_name: object.to_string(),
+                    version_id: Some(ordinary_version_id),
+                    ..Default::default()
+                },
+            ]
+        };
+
+        let set = store.pools[0].get_disks_by_key(object);
+        let disks = set.disks.read().await;
+        assert_eq!(disks.len(), 4, "the rollback fixture must use four disks");
+        // Keep discovery fully online, then make a quorum of disks report an
+        // error only after their batch metadata commit has completed.
+        for disk in disks.iter().take(3) {
+            let disk = disk.as_ref().expect("injected rollback disks should be online");
+            crate::disk::local::set_delete_version_fail_after_commit(disk.path().as_path(), object);
+        }
+        drop(disks);
+        let receipt_sink = crate::object_api::TierFreeVersionReceiptSink::new();
+        let (_deleted, errors) = store
+            .delete_objects(
+                bucket,
+                delete_requests(),
+                ObjectOptions {
+                    tier_free_version_receipt_sink: Some(receipt_sink.clone()),
+                    ..Default::default()
+                },
+            )
+            .await;
+        assert_eq!(
+            errors,
+            vec![Some(StorageError::Unexpected), Some(StorageError::Unexpected),],
+            "three post-commit disk errors must fail batch delete before receipts publish"
+        );
+
+        assert!(
+            receipt_sink
+                .drain()
+                .expect("the test-owned failed-batch sink should drain exactly once")
+                .is_empty(),
+            "a rolled-back physical group must publish no cleanup receipt"
+        );
+        let retained_transitioned = store
+            .get_object_info(
+                bucket,
+                object,
+                &ObjectOptions {
+                    version_id: Some(transitioned_version_id.to_string()),
+                    versioned: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("failed batch delete must restore the transitioned sibling");
+        assert_eq!(retained_transitioned.transitioned_object.status, rustfs_filemeta::TRANSITION_COMPLETE);
+        let retained_ordinary = store
+            .get_object_info(
+                bucket,
+                object,
+                &ObjectOptions {
+                    version_id: Some(ordinary_version_id.to_string()),
+                    versioned: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("failed batch delete must restore the ordinary sibling");
+        assert_ne!(retained_ordinary.transitioned_object.status, rustfs_filemeta::TRANSITION_COMPLETE);
+        let retained_versions = set
+            .load_file_info_versions_exact(bucket, object)
+            .await
+            .expect("rolled-back batch metadata should decode")
+            .expect("rolled-back batch source should remain on disk");
+        assert_eq!(
+            retained_versions
+                .versions
+                .iter()
+                .chain(retained_versions.free_versions.iter())
+                .filter(|version| version.tier_free_version())
+                .count(),
+            0,
+            "failed batch quorum must not retain a free-version owner"
+        );
+        let retained_version_ids = retained_versions
+            .versions
+            .iter()
+            .filter_map(|version| version.version_id)
+            .collect::<std::collections::HashSet<_>>();
+        assert_eq!(
+            retained_version_ids,
+            std::collections::HashSet::from([transitioned_version_id, ordinary_version_id]),
+            "the physical-group rollback must restore both explicit siblings"
+        );
+        assert_eq!(backend.object_count().await, 1, "failed batch commit must retain the remote object");
+        assert_eq!(backend.remove_count().await, 0, "failed batch commit must not dispatch remote cleanup");
+
+        ExpiryState::resize_workers(1, store.clone()).await;
+        let (_deleted, retry_errors) = store
+            .delete_objects(bucket, delete_requests(), ObjectOptions::default())
+            .await;
+        assert!(
+            retry_errors.iter().all(Option::is_none),
+            "retry after disk recovery should commit: {retry_errors:?}"
+        );
+        tokio::time::timeout(Duration::from_secs(30), async {
+            loop {
+                let metadata_absent = set
+                    .load_file_info_versions_exact(bucket, object)
+                    .await
+                    .expect("retry cleanup metadata should remain readable")
+                    .is_none();
+                if metadata_absent && backend.remove_count().await == 1 {
+                    return;
+                }
+                tokio::time::sleep(Duration::from_millis(10)).await;
+            }
+        })
+        .await
+        .expect("successful batch retry should converge without a recovery scan");
+        store
+            .delete_bucket(bucket, &DeleteBucketOptions::default())
+            .await
+            .expect("successful batch retry should leave the bucket empty");
+        shutdown.cancel();
+    }
+
     #[cfg(feature = "test-util")]
     async fn run_multi_pool_same_remote_tuple_delete_case(batch: bool) {
         let temp_dir = tempfile::tempdir().expect("create shared-tuple multi-pool store dir");
@@ -12643,8 +13184,10 @@ mod tests {
         );
         assert_eq!(backend.object_count().await, 1);
 
+        let receipt_sink = crate::object_api::TierFreeVersionReceiptSink::new();
         let mut delete_opts = ObjectOptions {
             tier_delete_journal_api: Some(store.clone()),
+            tier_free_version_receipt_sink: Some(receipt_sink.clone()),
             ..Default::default()
         };
         if batch {
@@ -12772,7 +13315,20 @@ mod tests {
         );
 
         backend.set_remove_failure(false);
-        wait_for_tier_free_version_recovery(store.clone(), &backend, 1).await;
+        let receipts = receipt_sink
+            .drain()
+            .expect("the simulated outer multi-pool wrapper should drain exactly once");
+        assert_eq!(
+            receipts.len(),
+            1,
+            "the same physical key and remote tuple must collapse to one causal task"
+        );
+        assert_eq!(
+            crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_committed_free_versions(&store, receipts).await,
+            1,
+            "the committed shared-tuple task should enter the running worker"
+        );
+        wait_for_expiry_workers_idle(&store).await;
         assert_eq!(backend.remove_count().await, 1, "shared remote tuple should be deleted exactly once");
         for pool_idx in 0..2 {
             assert!(

crates/ecstore/src/store/mod.rs

@@ -97,6 +97,8 @@ pub(crate) struct BucketDeleteDiagnosticBudget {
     deadline: Option<tokio::time::Instant>,
     max_elapsed: Duration,
     entries_remaining: usize,
+    #[cfg(test)]
+    first_io_delay: Option<(Duration, Arc<std::sync::atomic::AtomicBool>)>,
 }
 
 impl BucketDeleteDiagnosticBudget {
@@ -109,9 +111,17 @@ impl BucketDeleteDiagnosticBudget {
             deadline: None,
             max_elapsed: elapsed,
             entries_remaining: entries,
+            #[cfg(test)]
+            first_io_delay: None,
         }
     }
 
+    #[cfg(test)]
+    fn with_first_io_delay(mut self, delay: Duration, started: Arc<std::sync::atomic::AtomicBool>) -> Self {
+        self.first_io_delay = Some((delay, started));
+        self
+    }
+
     fn deadline(&mut self) -> tokio::time::Instant {
         let max_elapsed = self.max_elapsed;
         *self.deadline.get_or_insert_with(|| tokio::time::Instant::now() + max_elapsed)
@@ -137,7 +147,20 @@ impl BucketDeleteDiagnosticBudget {
         if tokio::time::Instant::now() >= deadline {
             return Ok(None);
         }
-        match tokio::time::timeout_at(deadline, future).await {
+        #[cfg(test)]
+        let first_io_delay = self.first_io_delay.take();
+        #[cfg(test)]
+        let timeout_result = tokio::time::timeout_at(deadline, async move {
+            if let Some((delay, started)) = first_io_delay {
+                started.store(true, std::sync::atomic::Ordering::SeqCst);
+                tokio::time::sleep(delay).await;
+            }
+            future.await
+        })
+        .await;
+        #[cfg(not(test))]
+        let timeout_result = tokio::time::timeout_at(deadline, future).await;
+        match timeout_result {
             Ok(result) => result.map(Some),
             Err(_) => Ok(None),
         }
@@ -175,6 +198,22 @@ impl BucketDeleteBlockerKind {
             Self::DiagnosticBudgetExceeded => "diagnostic_budget_exceeded",
         }
     }
+
+    /// Whether the blocking residue is something the caller can still see and
+    /// remove through the S3 API.
+    ///
+    /// A live version or a tier free-version is ordinary: the bucket really is
+    /// not empty, the client can list and delete what is left, and the 409 it
+    /// receives is a complete answer.
+    ///
+    /// The remaining kinds are not. They are on-disk state that no S3 request
+    /// can reach: the caller has drained every version the API will show and
+    /// `DeleteBucket` still refuses, with no way to find out why. That is a
+    /// server-side integrity problem, and it is the reason this classification
+    /// exists — see [`bucket_delete_blocker_level`].
+    pub(crate) const fn is_client_visible(self) -> bool {
+        matches!(self, Self::VisibleVersion | Self::TierFreeVersion)
+    }
 }
 
 impl BucketMetadataLessResidue {
@@ -386,6 +425,8 @@ pub(crate) mod init_format;
 pub(crate) mod list_objects;
 mod multipart;
 mod object;
+#[cfg(any(test, feature = "test-util"))]
+pub use object::DeleteAfterObjectLockSnapshotBarrier;
 pub(crate) use object::{
     DecommissionFixedReadAnchor, ObjectLockDiagGuard, RemoteTuplePublicationCommitGuard, RemoteTuplePublicationFence,
     SourceCleanupMutationFence, tiered_data_movement_source_matches,

crates/ecstore/src/store/object.rs

@@ -14,7 +14,7 @@
 
 use super::*;
 use crate::bucket::lifecycle::{
-    bucket_lifecycle_ops::eval_action_from_lifecycle,
+    bucket_lifecycle_ops::{enqueue_committed_free_versions, eval_action_from_lifecycle},
     get_expiry_configs,
     tier_delete_journal::{
         ActiveTierDeleteDispatch, EVENT_LIFECYCLE_TIER_DELETE_JOURNAL, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_LIFECYCLE,
@@ -39,6 +39,7 @@ use crate::core::pools::{DecommissionCapacityOwner, ensure_decommission_capacity
 use crate::disk::OldCurrentSize;
 use crate::object_api::{
     NamespaceLockFence, ObjectLockConfigSnapshot, ScannerPublicationCommitScopeGuard, ScannerPublicationCommitState,
+    TierFreeVersionReceiptSink,
 };
 use crate::services::notification_sys::acquire_tier_delete_journal_fleet_proof;
 use crate::services::tier::tier::{TierConfigMgr, TierDestinationId, TierOperationLease, tier_destination_id_from_metadata};
@@ -52,6 +53,7 @@ use crate::storage_api_contracts::{
     object::{DeleteAccounting, ObjectIO as _, ObjectOperations as _},
 };
 use parking_lot::Mutex as ParkingMutex;
+use rustfs_filemeta::ObjectPartInfo;
 use rustfs_io_metrics::{
     record_object_lock_diag_acquire_duration, record_object_lock_diag_hold_duration, record_object_lock_diag_slow_acquire,
     record_object_lock_diag_slow_hold,
@@ -70,6 +72,26 @@ const RECURSIVE_DELETE_VERSION_SCAN_PAGE_SIZE: i32 = 1000;
 #[cfg(test)]
 const RECURSIVE_DELETE_VERSION_SCAN_PAGE_SIZE: i32 = 2;
 
+fn install_tier_free_version_receipt_sink(opts: &mut ObjectOptions) -> Option<TierFreeVersionReceiptSink> {
+    if opts.tier_free_version_receipt_sink.is_some() || opts.skip_free_version || opts.delete_prefix {
+        return None;
+    }
+
+    let sink = TierFreeVersionReceiptSink::new();
+    opts.tier_free_version_receipt_sink = Some(sink.clone());
+    Some(sink)
+}
+
+async fn enqueue_recorded_tier_free_versions(store: &ECStore, sink: Option<TierFreeVersionReceiptSink>) -> usize {
+    let Some(sink) = sink else {
+        return 0;
+    };
+    let Ok(receipts) = sink.drain() else {
+        return 0;
+    };
+    enqueue_committed_free_versions(store, receipts).await
+}
+
 fn build_tier_delete_journal_entry(
     bucket: &str,
     object: &str,
@@ -1292,7 +1314,7 @@ fn should_create_delete_marker_for_missing_object(opts: &ObjectOptions) -> bool
     (opts.versioned || opts.version_suspended) && opts.version_id.is_none() && !opts.delete_marker && !opts.data_movement
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 struct DeleteAfterObjectLockSnapshotBarrierState {
     bucket: String,
     arrived: tokio::sync::Notify,
@@ -1301,19 +1323,19 @@ struct DeleteAfterObjectLockSnapshotBarrierState {
     namespace_acquired: AtomicBool,
 }
 
-#[cfg(test)]
-pub(crate) struct DeleteAfterObjectLockSnapshotBarrier {
+#[cfg(any(test, feature = "test-util"))]
+pub struct DeleteAfterObjectLockSnapshotBarrier {
     state: Arc<DeleteAfterObjectLockSnapshotBarrierState>,
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 static DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER: std::sync::OnceLock<
     std::sync::Mutex<Option<Arc<DeleteAfterObjectLockSnapshotBarrierState>>>,
 > = std::sync::OnceLock::new();
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 impl DeleteAfterObjectLockSnapshotBarrier {
-    pub(crate) fn install(bucket: &str) -> Self {
+    pub fn install(bucket: &str) -> Self {
         let state = Arc::new(DeleteAfterObjectLockSnapshotBarrierState {
             bucket: bucket.to_string(),
             arrived: tokio::sync::Notify::new(),
@@ -1330,15 +1352,15 @@ impl DeleteAfterObjectLockSnapshotBarrier {
         Self { state }
     }
 
-    pub(crate) async fn wait_until_paused(&self) {
+    pub async fn wait_until_paused(&self) {
         self.state.arrived.notified().await;
     }
 
-    pub(crate) fn release(&self) {
+    pub fn release(&self) {
         self.state.release.notify_one();
     }
 
-    pub(crate) async fn release_and_wait_until_namespace_pending(&self) {
+    pub async fn release_and_wait_until_namespace_pending(&self) {
         let namespace_pending = self.state.namespace_pending.notified();
         self.release();
         tokio::time::timeout(Duration::from_secs(5), namespace_pending)
@@ -1346,12 +1368,12 @@ impl DeleteAfterObjectLockSnapshotBarrier {
             .expect("delete should proceed to its namespace lock after leaving the snapshot barrier");
     }
 
-    pub(crate) fn namespace_acquired(&self) -> bool {
+    pub fn namespace_acquired(&self) -> bool {
         self.state.namespace_acquired.load(Ordering::Acquire)
     }
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 impl Drop for DeleteAfterObjectLockSnapshotBarrier {
     fn drop(&mut self) {
         self.state.release.notify_one();
@@ -1364,7 +1386,7 @@ impl Drop for DeleteAfterObjectLockSnapshotBarrier {
     }
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 async fn pause_delete_after_object_lock_snapshot(bucket: &str) {
     let state = DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER
         .get_or_init(|| std::sync::Mutex::new(None))
@@ -1376,11 +1398,24 @@ async fn pause_delete_after_object_lock_snapshot(bucket: &str) {
     if let Some(state) = state {
         state.arrived.notify_one();
         state.release.notified().await;
+    }
+}
+
+#[cfg(any(test, feature = "test-util"))]
+fn notify_delete_namespace_pending(bucket: &str) {
+    let state = DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER
+        .get_or_init(|| std::sync::Mutex::new(None))
+        .lock()
+        .expect("delete snapshot barrier mutex should not poison")
+        .as_ref()
+        .filter(|state| state.bucket == bucket)
+        .cloned();
+    if let Some(state) = state {
         state.namespace_pending.notify_one();
     }
 }
 
-#[cfg(test)]
+#[cfg(any(test, feature = "test-util"))]
 fn notify_delete_namespace_acquired(bucket: &str) {
     let state = DELETE_AFTER_OBJECT_LOCK_SNAPSHOT_BARRIER
         .get_or_init(|| std::sync::Mutex::new(None))
@@ -2114,18 +2149,6 @@ fn remote_tuple_publication_object_source_matches(expected: &ObjectInfo, current
     let (Ok(expected_actual_size), Ok(current_actual_size)) = (expected.get_actual_size(), current.get_actual_size()) else {
         return false;
     };
-    let parts_match = expected.parts.len() == current.parts.len()
-        && expected.parts.iter().all(|expected_part| {
-            current
-                .parts
-                .iter()
-                .find(|current_part| current_part.number == expected_part.number)
-                .is_some_and(|current_part| {
-                    current_part.size == expected_part.size
-                        && current_part.actual_size == expected_part.actual_size
-                        && current_part.etag == expected_part.etag
-                })
-        });
 
     expected.data_dir.is_some_and(|data_dir| !data_dir.is_nil())
         && expected.data_dir == current.data_dir
@@ -2133,6 +2156,7 @@ fn remote_tuple_publication_object_source_matches(expected: &ObjectInfo, current
         && expected.delete_marker == current.delete_marker
         && expected.size == current.size
         && expected_actual_size == current_actual_size
+        && expected.etag == current.etag
         && expected.checksum == current.checksum
         && expected.mod_time == current.mod_time
         && expected.storage_class == current.storage_class
@@ -2154,7 +2178,24 @@ fn remote_tuple_publication_object_source_matches(expected: &ObjectInfo, current
         && expected.transitioned_object.free_version == current.transitioned_object.free_version
         && expected.transitioned_object.status == current.transitioned_object.status
         && expected.transition_version_state == current.transition_version_state
-        && parts_match
+        && remote_tuple_publication_parts_match(&expected.parts, &current.parts)
+}
+
+fn remote_tuple_publication_parts_match(expected: &[ObjectPartInfo], current: &[ObjectPartInfo]) -> bool {
+    if expected.len() != current.len() {
+        return false;
+    }
+    let Some(mut current_parts) = crate::data_movement::data_movement_parts_by_number(current) else {
+        return false;
+    };
+
+    expected.iter().all(|expected_part| {
+        current_parts.remove(&expected_part.number).is_some_and(|current_part| {
+            current_part.size == expected_part.size
+                && current_part.actual_size == expected_part.actual_size
+                && current_part.etag == expected_part.etag
+        })
+    })
 }
 
 impl RemoteTuplePublicationFence {
@@ -2557,6 +2598,10 @@ impl ECStore {
         let diag_enabled = is_object_lock_diag_enabled();
         let ns_lock = self.handle_new_ns_lock(bucket, object).await?;
         let acquire_start = Instant::now();
+        #[cfg(any(test, feature = "test-util"))]
+        if matches!(op, "delete_object" | "delete_objects") {
+            notify_delete_namespace_pending(bucket);
+        }
         let guard = ns_lock
             .get_write_lock(get_lock_acquire_timeout())
             .await
@@ -4148,7 +4193,16 @@ impl ECStore {
         opts: ObjectOptions,
         tier_journal_api: Option<Arc<ECStore>>,
     ) -> Result<ObjectInfo> {
-        Box::pin(self.handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api)).await
+        Box::pin(async move {
+            let mut opts = opts;
+            let receipt_sink = install_tier_free_version_receipt_sink(&mut opts);
+            let result = self
+                .handle_delete_object_with_journal_inner(bucket, object, opts, tier_journal_api)
+                .await;
+            enqueue_recorded_tier_free_versions(self, receipt_sink).await;
+            result
+        })
+        .await
     }
 
     async fn handle_delete_object_with_journal_inner(
@@ -4222,7 +4276,7 @@ impl ECStore {
         if opts.delete_prefix && opts.expected_bucket_incarnation_id.is_none() {
             opts.expected_bucket_incarnation_id = current_bucket_incarnation_id;
         }
-        #[cfg(test)]
+        #[cfg(any(test, feature = "test-util"))]
         pause_delete_after_object_lock_snapshot(bucket).await;
 
         if opts.delete_prefix && !opts.delete_prefix_object {
@@ -4236,7 +4290,7 @@ impl ECStore {
         } else {
             None
         };
-        #[cfg(test)]
+        #[cfg(any(test, feature = "test-util"))]
         if _object_lock_guard.is_some() {
             notify_delete_namespace_acquired(bucket);
         }
@@ -4526,6 +4580,25 @@ impl ECStore {
         objects: Vec<ObjectToDelete>,
         opts: ObjectOptions,
         tier_journal_api: Option<Arc<ECStore>>,
+    ) -> (Vec<DeletedObject>, Vec<Option<Error>>, Vec<Option<DeleteAccounting>>) {
+        Box::pin(async move {
+            let mut opts = opts;
+            let receipt_sink = install_tier_free_version_receipt_sink(&mut opts);
+            let result = self
+                .handle_delete_objects_with_journal_and_accounting_inner(bucket, objects, opts, tier_journal_api)
+                .await;
+            enqueue_recorded_tier_free_versions(self, receipt_sink).await;
+            result
+        })
+        .await
+    }
+
+    async fn handle_delete_objects_with_journal_and_accounting_inner(
+        &self,
+        bucket: &str,
+        objects: Vec<ObjectToDelete>,
+        opts: ObjectOptions,
+        tier_journal_api: Option<Arc<ECStore>>,
     ) -> (Vec<DeletedObject>, Vec<Option<Error>>, Vec<Option<DeleteAccounting>>) {
         // encode object name
         let objects: Vec<ObjectToDelete> = objects
@@ -4610,7 +4683,7 @@ impl ECStore {
                 StorageError::BucketNotFound(bucket.to_string()),
             );
         }
-        #[cfg(test)]
+        #[cfg(any(test, feature = "test-util"))]
         if current_bucket_incarnation_id.is_some() {
             pause_delete_after_object_lock_snapshot(bucket).await;
         }
@@ -4618,7 +4691,7 @@ impl ECStore {
             Ok(guards) => guards,
             Err(err) => return return_batch_delete_lock_error_with_accounting(objects.as_slice(), err),
         };
-        #[cfg(test)]
+        #[cfg(any(test, feature = "test-util"))]
         if !_object_lock_guards.is_empty() {
             notify_delete_namespace_acquired(bucket);
         }
@@ -5865,6 +5938,241 @@ mod tests {
         assert!(!ECStore::is_equivalent_decommission_capacity_target(&source, &wrong_bucket));
     }
 
+    fn publication_part(number: usize) -> ObjectPartInfo {
+        ObjectPartInfo {
+            number,
+            size: 100 + number,
+            actual_size: i64::try_from(200 + number).expect("test part size should fit in i64"),
+            etag: format!("part-etag-{number}"),
+            ..Default::default()
+        }
+    }
+
+    fn publication_source(parts: Vec<ObjectPartInfo>) -> ObjectInfo {
+        ObjectInfo {
+            data_dir: Some(Uuid::new_v4()),
+            version_id: Some(Uuid::new_v4()),
+            size: 4096,
+            actual_size: 4096,
+            etag: Some("object-etag".to_string()),
+            checksum: Some(Bytes::from_static(b"object-checksum")),
+            mod_time: Some(time::OffsetDateTime::UNIX_EPOCH),
+            parts: Arc::new(parts),
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn publication_parts_match_is_order_independent_and_bijective() {
+        let ordered = vec![publication_part(1), publication_part(2), publication_part(3)];
+        let mut reversed = ordered.clone();
+        reversed.reverse();
+
+        assert!(remote_tuple_publication_parts_match(&[], &[]));
+        assert!(remote_tuple_publication_parts_match(&ordered, &ordered));
+        assert!(remote_tuple_publication_parts_match(&ordered, &reversed));
+        assert!(!remote_tuple_publication_parts_match(&ordered[..2], &ordered));
+
+        let expected_duplicate = vec![publication_part(1), publication_part(1)];
+        let current_unique = vec![publication_part(1), publication_part(2)];
+        assert!(
+            !remote_tuple_publication_parts_match(&expected_duplicate, &current_unique),
+            "two expected entries must not reuse the same current part"
+        );
+        assert!(!remote_tuple_publication_parts_match(&current_unique, &expected_duplicate));
+        assert!(!remote_tuple_publication_parts_match(&expected_duplicate, &expected_duplicate));
+    }
+
+    #[test]
+    fn publication_parts_match_preserves_exact_part_identity_contract() {
+        let expected = vec![publication_part(1)];
+
+        let mut different_number = expected.clone();
+        different_number[0].number = 2;
+        assert!(!remote_tuple_publication_parts_match(&expected, &different_number));
+
+        let mut different_size = expected.clone();
+        different_size[0].size += 1;
+        assert!(!remote_tuple_publication_parts_match(&expected, &different_size));
+
+        let mut different_actual_size = expected.clone();
+        different_actual_size[0].actual_size += 1;
+        assert!(!remote_tuple_publication_parts_match(&expected, &different_actual_size));
+
+        let mut different_etag = expected.clone();
+        different_etag[0].etag.push_str("-changed");
+        assert!(!remote_tuple_publication_parts_match(&expected, &different_etag));
+
+        let mut ignored_fields = expected.clone();
+        ignored_fields[0].index = Some(Bytes::from_static(b"different-index"));
+        ignored_fields[0].checksums = Some(HashMap::from([("CRC32C".to_string(), "different".to_string())]));
+        ignored_fields[0].mod_time = Some(time::OffsetDateTime::UNIX_EPOCH);
+        assert!(
+            remote_tuple_publication_parts_match(&expected, &ignored_fields),
+            "the publication fence must retain its existing checksum/index/mod-time compatibility contract"
+        );
+
+        let zero_actual_size = vec![ObjectPartInfo {
+            actual_size: 0,
+            ..publication_part(1)
+        }];
+        assert!(
+            !remote_tuple_publication_parts_match(&zero_actual_size, &expected),
+            "the publication fence compares raw part actual sizes without the broader comparator's fallback"
+        );
+    }
+
+    #[test]
+    fn publication_source_match_rejects_object_etag_and_identity_mutations() {
+        let expected = publication_source(vec![publication_part(1)]);
+        assert!(remote_tuple_publication_object_source_matches(&expected, &expected));
+
+        let mut current = expected.clone();
+        current.etag = Some("changed-object-etag".to_string());
+        assert!(!remote_tuple_publication_object_source_matches(&expected, &current));
+
+        let mut current = expected.clone();
+        current.checksum = Some(Bytes::from_static(b"changed-checksum"));
+        assert!(!remote_tuple_publication_object_source_matches(&expected, &current));
+
+        let mut current = expected.clone();
+        current.actual_size += 1;
+        assert!(!remote_tuple_publication_object_source_matches(&expected, &current));
+
+        let mut current = expected.clone();
+        current.data_dir = Some(Uuid::new_v4());
+        assert!(!remote_tuple_publication_object_source_matches(&expected, &current));
+
+        let mut current = expected.clone();
+        current.version_id = Some(Uuid::new_v4());
+        assert!(!remote_tuple_publication_object_source_matches(&expected, &current));
+
+        let mut current = expected.clone();
+        current.mod_time = current.mod_time.map(|mod_time| mod_time + time::Duration::SECOND);
+        assert!(!remote_tuple_publication_object_source_matches(&expected, &current));
+    }
+
+    #[test]
+    fn publication_source_match_preserves_effective_actual_size_compatibility() {
+        let expected = publication_source(vec![publication_part(1)]);
+        let current = ObjectInfo {
+            actual_size: 0,
+            ..expected.clone()
+        };
+
+        assert_eq!(
+            expected.get_actual_size().expect("expected size should be valid"),
+            current.get_actual_size().expect("current size should be valid")
+        );
+        assert!(remote_tuple_publication_object_source_matches(&expected, &current));
+    }
+
+    #[test]
+    fn publication_source_match_rejects_duplicate_parts_through_the_full_fence() {
+        let expected = publication_source(vec![publication_part(1), publication_part(1)]);
+        let current = ObjectInfo {
+            parts: Arc::new(vec![publication_part(1), publication_part(2)]),
+            ..expected.clone()
+        };
+
+        assert!(
+            !remote_tuple_publication_object_source_matches(&expected, &current),
+            "the full source fence must reject the original non-bijective false-positive"
+        );
+    }
+
+    #[test]
+    fn publication_source_match_handles_10_000_reversed_parts_without_payload_cloning() {
+        let parts = Arc::new((1..=10_000).map(publication_part).collect::<Vec<_>>());
+        let mut reversed = parts.as_ref().clone();
+        reversed.reverse();
+        let expected = publication_source(Vec::new());
+        let expected = ObjectInfo {
+            parts: Arc::clone(&parts),
+            ..expected
+        };
+        let current = ObjectInfo {
+            parts: Arc::new(reversed),
+            ..expected.clone()
+        };
+        let expected_parts = Arc::clone(&expected.parts);
+
+        assert!(remote_tuple_publication_object_source_matches(&expected, &current));
+        assert!(Arc::ptr_eq(&expected.parts, &expected_parts));
+    }
+
+    #[tokio::test]
+    async fn publication_commit_guard_rejects_an_etag_only_source_change() {
+        let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
+        let (_first_dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
+        let (_second_dirs, second_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
+        let store =
+            Arc::new(new_prepared_reader_test_store_with_ctx(&[Arc::clone(&first_set), Arc::clone(&second_set)], ctx).await);
+        let bucket = "publication-etag-source-change";
+        let object = "source.bin";
+        for set in [&first_set, &second_set] {
+            set.make_bucket(bucket, &MakeBucketOptions::default())
+                .await
+                .expect("publication ETag test bucket should be created");
+        }
+
+        let mut source_body = PutObjReader::from_vec(b"source body".to_vec());
+        first_set
+            .put_object(bucket, object, &mut source_body, &ObjectOptions::default())
+            .await
+            .expect("publication source should be written");
+        let source = first_set
+            .get_object_info(
+                bucket,
+                object,
+                &ObjectOptions {
+                    include_part_checksums: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("publication source should be readable");
+        let publication = store
+            .acquire_remote_tuple_publication_fence(bucket, 0, &source, false)
+            .await
+            .expect("the source snapshot should produce a publication capability");
+
+        let changed = first_set
+            .put_object_metadata(
+                bucket,
+                object,
+                &ObjectOptions {
+                    eval_metadata: Some(HashMap::from([("etag".to_string(), "changed-etag".to_string())])),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("the source ETag should be updated in place");
+        assert_ne!(source.etag, changed.etag);
+        let unchanged_except_etag = ObjectInfo {
+            etag: source.etag.clone(),
+            ..changed.clone()
+        };
+        assert!(
+            remote_tuple_publication_object_source_matches(&source, &unchanged_except_etag),
+            "the persisted source update fixture must differ only by object ETag"
+        );
+
+        let encoded = encode_dir_object(object);
+        let err = match publication.into_commit_guard(1, bucket, &encoded).await {
+            Ok(_) => panic!("the publication guard must reject an ETag-only source change"),
+            Err(err) => err,
+        };
+        assert!(matches!(err, Error::DataMovementOverwriteErr(_, _, _)));
+        assert!(
+            second_set
+                .get_object_info(bucket, object, &ObjectOptions::default())
+                .await
+                .is_err(),
+            "a rejected publication must not create a target object"
+        );
+    }
+
     #[tokio::test]
     async fn generic_data_movement_put_rejects_transition_ownership_without_capability() {
         let (_dirs, set) = make_local_set_disks(4, 2).await;
@@ -7286,6 +7594,15 @@ mod tests {
         );
         drop(unified_future);
 
+        let batch_future =
+            store.handle_delete_objects_with_journal_and_accounting("bucket", Vec::new(), ObjectOptions::default(), None);
+        let batch_future_size = std::mem::size_of_val(&batch_future);
+        assert!(
+            batch_future_size <= 4 * 1024,
+            "batch delete handler future must remain stack-bounded; measured {batch_future_size} bytes"
+        );
+        drop(batch_future);
+
         let outer_future = store.handle_delete_object("bucket", "object", ObjectOptions::default());
         let outer_future_size = std::mem::size_of_val(&outer_future);
         assert!(

crates/lifecycle/Cargo.toml

@@ -60,7 +60,6 @@ rustfs-storage-api.workspace = true
 s3s = { workspace = true, features = ["minio"] }
 time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
 tracing.workspace = true
-url.workspace = true
 uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnostics"] }
 
 [dev-dependencies]

crates/protocols/src/swift/README.md

@@ -4,23 +4,78 @@ Swift-compatible object storage API implementation for RustFS.
 
 ## Features
 
-This implementation provides **Phase 1 Swift API support** (~25% of full Swift API):
+The lists below are bounded to what `router.rs` / `handler.rs` dispatch and
+what the test suite exercises. A module existing under `src/swift/` does not
+by itself mean the feature is reachable over HTTP.
 
-- ✅ Container CRUD operations (create, list, delete, metadata)
-- ✅ Object CRUD with streaming downloads (upload, get, head, delete)
-- ✅ Keystone token authentication
-- ✅ Multi-tenant isolation with secure SHA256-based bucket prefixing
-- ✅ Server-side object copy (COPY method)
-- ✅ HTTP Range requests for partial downloads (206, 416 responses)
-- ✅ Custom metadata support (X-Object-Meta-*, X-Container-Meta-*)
+### Wired through the router and handler
 
-**Not yet implemented:**
-- ⏳ Account-level operations (statistics, metadata)
-- ⏳ Large object support (multi-part uploads >5GB)
-- ⏳ Object versioning
-- ⏳ Container ACLs and CORS
-- ⏳ Temporary URLs (TempURL)
-- ⏳ XML/plain-text response formats (JSON only)
+- ✅ Account listing (`GET /v1/AUTH_{project}`, JSON) and additive account
+  metadata updates (`POST`, `X-Account-Meta-*` / `X-Remove-Account-Meta-*`)
+- ✅ Container CRUD (create, list, head, update metadata, delete)
+- ✅ Object CRUD with streaming downloads, HTTP Range requests (206 / 416),
+  and server-side copy via the `COPY` method
+- ✅ Keystone token authentication and multi-tenant isolation with
+  SHA256-based bucket prefixing
+- ✅ Custom metadata (`X-Object-Meta-*`, `X-Container-Meta-*`); container and
+  account POSTs are additive, object POSTs replace the set
+- ✅ Container ACLs (`X-Container-Read` / `X-Container-Write`, set and remove on
+  container POST, reported on HEAD). Enforcement is account-level plus
+  referrer checks; per-user grants are not evaluated because credentials
+  carry no user id
+- ✅ CORS: `OPTIONS` preflight on container and object routes, and response
+  header injection driven by `X-Container-Meta-Access-Control-*`
+- ✅ TempURL (`temp_url_sig` / `temp_url_expires` on object GET, HEAD, PUT;
+  key stored as account metadata; optional client-IP restriction)
+- ✅ FormPost (container POST with `multipart/form-data`, signed with the
+  account TempURL key)
+- ✅ Large objects: Static Large Objects (`?multipart-manifest=put|get|delete`)
+  and Dynamic Large Objects (`X-Object-Manifest`)
+- ✅ Bulk operations: `DELETE /v1/AUTH_{project}?bulk-delete` and
+  `PUT /v1/AUTH_{project}/{container}?extract-archive=tar|tar.gz|tar.bz2`
+- ✅ Object versioning in the Swift `X-Versions-Location` style: the previous
+  copy is archived on PUT / DELETE and restored on DELETE
+- ✅ Symlinks (`X-Symlink-Target` on PUT, resolved on GET / HEAD with loop and
+  depth checks)
+- ✅ Container quotas (`X-Container-Meta-Quota-Bytes` / `-Quota-Count`),
+  enforced on object PUT
+- ✅ Static website serving on object GET when `web-index` / `web-listings`
+  container metadata is set
+- ✅ Object expiration headers: `X-Delete-At` / `X-Delete-After` are validated,
+  stored, and returned on GET / HEAD
+
+### Not yet wired, or partially wired
+
+- ⏳ Account `HEAD` returns `501 Not Implemented`; no account-level usage
+  statistics are exposed
+- ⏳ Automatic deletion of expired objects: `expiration_worker.rs` exists but
+  the server never starts it, so objects past `X-Delete-At` are not removed
+- ⏳ Container sync (`sync.rs`): no `X-Container-Sync-*` header handling and no
+  background worker; the module is unit-tested only
+- ⏳ `X-Copy-From` on object PUT (only the `COPY` method is supported)
+- ⏳ `X-History-Location` versioning mode
+- ⏳ Static website index / listing pages at the container root (only the
+  object GET route consults static-web settings)
+- ⏳ XML / plain-text listing formats; the `format=` query parameter is
+  ignored and listings are always JSON
+
+### Test coverage
+
+- Unit tests live next to each module (`acl.rs`, `bulk.rs`, `cors.rs`,
+  `dlo.rs`, `slo.rs`, `tempurl.rs`, `formpost.rs`, `staticweb.rs`,
+  `symlink.rs`, `quota.rs`, `expiration.rs`, `versioning.rs`, `router.rs`,
+  `handler.rs`, and others) and run in the CI `swift` feature lane
+- `crates/protocols/tests/swift_metadata_persistence.rs` runs account,
+  container, ACL, TempURL-key, and versioning metadata writes against a real
+  ECStore and reloads them from disk
+- `crates/protocols/tests/swift_versioning_integration.rs`,
+  `swift_listing_symlink_tests.rs`, `swift_simple_integration.rs`, and
+  `swift_phase4_integration.rs` cover version naming, listing parameters,
+  symlink parsing, and module-level helpers without a server
+- `rustfs/tests/swift_container_integration_test.rs` and
+  `swift_object_integration_test.rs` exercise the HTTP surface end to end but
+  are `#[ignore]` and need a running server (`TEST_RUSTFS_SERVER`); they are
+  not part of CI
 
 ## Enable Feature
 
@@ -42,38 +97,52 @@ cargo build --features full
 
 ## Configuration
 
-Swift API uses Keystone for authentication. Configure the following environment variables:
+Swift API uses Keystone for authentication. The variables below are read by
+`crates/keystone/src/config.rs`; that file is the authoritative list.
 
 | Variable | Description |
 |----------|-------------|
-| `RUSTFS_KEYSTONE_URL` | Keystone authentication endpoint URL |
-| `RUSTFS_KEYSTONE_ADMIN_TENANT` | Admin tenant/project name |
-| `RUSTFS_KEYSTONE_ADMIN_USER` | Admin username |
-| `RUSTFS_KEYSTONE_ADMIN_PASSWORD` | Admin password |
+| `RUSTFS_KEYSTONE_ENABLE` | Set to `true` to enable Keystone authentication (default `false`; nothing else is read while disabled) |
+| `RUSTFS_KEYSTONE_AUTH_URL` | Keystone authentication endpoint URL (required once enabled) |
+| `RUSTFS_KEYSTONE_VERSION` | Keystone API version, `v3` or `v2.0` (default `v3`) |
+| `RUSTFS_KEYSTONE_ADMIN_USER` | Admin username (optional) |
+| `RUSTFS_KEYSTONE_ADMIN_PASSWORD` | Admin password (optional) |
+| `RUSTFS_KEYSTONE_ADMIN_PROJECT` | Admin project name (optional) |
+| `RUSTFS_KEYSTONE_ADMIN_DOMAIN` | Admin domain name (optional) |
+| `RUSTFS_KEYSTONE_VERIFY_SSL` | Verify the Keystone TLS certificate (default `true`) |
+| `RUSTFS_KEYSTONE_ENABLE_CACHE` / `RUSTFS_KEYSTONE_CACHE_SIZE` / `RUSTFS_KEYSTONE_CACHE_TTL` | Token cache toggle, entry count, and TTL in seconds (defaults `true`, `10000`, `300`) |
+| `RUSTFS_KEYSTONE_TENANT_PREFIX` | Prefix bucket names with the tenant hash (default `true`) |
+| `RUSTFS_KEYSTONE_IMPLICIT_TENANTS` | Allow implicit tenant creation (default `true`) |
+| `RUSTFS_KEYSTONE_TIMEOUT` | Keystone request timeout in seconds (default `30`) |
 
 ## API Endpoints
 
 Swift API endpoints follow the pattern: `/v1/AUTH_{project_id}/...`
 
 ### Account Operations
-- `GET /v1/AUTH_{project}` - List containers
-- `HEAD /v1/AUTH_{project}` - Get account metadata (not yet implemented)
-- `POST /v1/AUTH_{project}` - Update account metadata (not yet implemented)
+- `GET /v1/AUTH_{project}` - List containers (JSON)
+- `HEAD /v1/AUTH_{project}` - Get account metadata (returns 501, not yet implemented)
+- `POST /v1/AUTH_{project}` - Update account metadata and TempURL key
+- `DELETE /v1/AUTH_{project}?bulk-delete` - Bulk delete
 
 ### Container Operations
-- `PUT /v1/AUTH_{project}/{container}` - Create container
-- `GET /v1/AUTH_{project}/{container}` - List objects
+- `PUT /v1/AUTH_{project}/{container}` - Create container (`?extract-archive=` for bulk upload)
+- `GET /v1/AUTH_{project}/{container}` - List objects (JSON; `limit`, `marker`, `end_marker`, `prefix`, `delimiter`)
 - `HEAD /v1/AUTH_{project}/{container}` - Get container metadata
-- `POST /v1/AUTH_{project}/{container}` - Update container metadata
+- `POST /v1/AUTH_{project}/{container}` - Update container metadata, ACLs, versioning location; FormPost when `multipart/form-data`
 - `DELETE /v1/AUTH_{project}/{container}` - Delete container
+- `OPTIONS /v1/AUTH_{project}/{container}` - CORS preflight
 
 ### Object Operations
-- `PUT /v1/AUTH_{project}/{container}/{object}` - Upload object
-- `GET /v1/AUTH_{project}/{container}/{object}` - Download object
+- `PUT /v1/AUTH_{project}/{container}/{object}` - Upload object (SLO manifest with `?multipart-manifest=put`, DLO with `X-Object-Manifest`, symlink with `X-Symlink-Target`)
+- `GET /v1/AUTH_{project}/{container}/{object}` - Download object (Range, SLO/DLO assembly, symlink resolution, `?multipart-manifest=get`)
 - `HEAD /v1/AUTH_{project}/{container}/{object}` - Get object metadata
 - `POST /v1/AUTH_{project}/{container}/{object}` - Update object metadata
-- `DELETE /v1/AUTH_{project}/{container}/{object}` - Delete object
+- `DELETE /v1/AUTH_{project}/{container}/{object}` - Delete object (`?multipart-manifest=delete` removes SLO segments)
 - `COPY /v1/AUTH_{project}/{container}/{object}` - Server-side copy
+- `OPTIONS /v1/AUTH_{project}/{container}/{object}` - CORS preflight
+
+Object GET, HEAD, and PUT also accept TempURL query parameters without an auth token.
 
 ## Architecture
 
@@ -96,11 +165,18 @@ Handler   (fallback)
 
 ### Key Components
 
-- **handler.rs** - Main service implementing Tower's Service trait
+- **handler.rs** - Main service implementing Tower's Service trait and method dispatch
 - **router.rs** - URL routing and parsing for Swift paths
 - **container.rs** - Container operations with tenant isolation
 - **object.rs** - Object operations including copy and range requests
-- **account.rs** - Account validation and tenant access control
+- **account.rs** - Account validation, tenant access control, account metadata and TempURL key
+- **acl.rs**, **cors.rs** - Container ACL evaluation and CORS config
+- **slo.rs**, **dlo.rs** - Static and dynamic large objects
+- **tempurl.rs**, **formpost.rs** - Signed URL and form upload validation
+- **bulk.rs** - Bulk delete and archive extraction
+- **versioning.rs**, **symlink.rs**, **quota.rs**, **staticweb.rs**, **expiration.rs** - Per-feature helpers called from the handler
+- **expiration_worker.rs**, **sync.rs** - Background workers that are not started by the server (see above)
+- **metadata_update.rs** - Additive account/container metadata merge
 - **errors.rs** - Swift-specific error types
 - **types.rs** - Data structures for Swift API responses
 
@@ -121,13 +197,15 @@ This ensures:
 
 ## Documentation
 
-See the `docs/` directory for detailed documentation:
+There is no separate Swift reference document in the repository. Use these
+sources instead:
 
-- `SWIFT_API.md` - Complete API reference
-- `TESTING_GUIDE.md` - Manual testing procedures
-- `COMPLETION_ANALYSIS.md` - Protocol coverage tracking
-- `COPY_IMPLEMENTATION.md` - Server-side copy documentation
-- `RANGE_REQUESTS.md` - Range request implementation details
+- Module-level `//!` comments in each `crates/protocols/src/swift/*.rs` file
+  describe the headers and metadata keys that feature reads
+- `crates/protocols/tests/swift_*.rs` and `rustfs/tests/swift_*_integration_test.rs`
+  show the expected request and response shapes
+- `docs/testing/ci-gates.md` describes the CI lane that builds and tests with
+  `--features swift`
 
 ## License
 

crates/scanner/tests/lifecycle_integration_test.rs

@@ -40,14 +40,15 @@ use uuid::Uuid;
 mod storage_api;
 
 use storage_api::lifecycle::{
-    BUCKET_LIFECYCLE_CONFIG, BucketOperations, BucketOptions, BucketVersioningSys, CompletePart, DiskOption, ECStore,
-    EcstoreError, Endpoint, EndpointServerPools, Endpoints, IlmAction, LcEvent, LcEventSrc, ListOperations as _,
-    MakeBucketOptions, MockWarmBackend, MultipartOperations as _, ObjectIO as _, ObjectOperations as _, PoolEndpoints,
-    STORAGE_FORMAT_FILE, TRANSITION_PENDING, TransitionCleanupStoreBarrier, TransitionOptions, assert_transition_meta_consistent,
-    enqueue_transition_for_existing_objects, expire_transitioned_object, free_version_count, get_bucket_metadata,
-    get_global_tier_config_mgr, init_background_expiry, init_bucket_metadata_sys, init_local_disks, is_err_object_not_found,
-    is_err_version_not_found, new_disk, path2_bucket_object_with_base_path, recover_transition_transaction_records,
-    register_mock_tier_util, update_bucket_metadata, wait_for_free_version_absence,
+    BUCKET_LIFECYCLE_CONFIG, BucketOperations, BucketOptions, BucketVersioningSys, CompletePart,
+    DeleteAfterObjectLockSnapshotBarrier, DiskOption, ECStore, EcstoreError, Endpoint, EndpointServerPools, Endpoints,
+    ExpiryState, IlmAction, LcEvent, LcEventSrc, ListOperations as _, MakeBucketOptions, MockWarmBackend,
+    MultipartOperations as _, ObjectIO as _, ObjectOperations as _, PoolEndpoints, STORAGE_FORMAT_FILE, TRANSITION_PENDING,
+    TransitionCleanupStoreBarrier, TransitionOptions, assert_transition_meta_consistent, enqueue_transition_for_existing_objects,
+    expire_transitioned_object, free_version_count, get_bucket_metadata, get_global_tier_config_mgr, init_background_expiry,
+    init_bucket_metadata_sys, init_local_disks, is_err_object_not_found, is_err_version_not_found, new_disk,
+    path2_bucket_object_with_base_path, recover_transition_transaction_records, register_mock_tier_util, update_bucket_metadata,
+    wait_for_free_version_absence,
 };
 
 static GLOBAL_ENV: OnceLock<(Vec<PathBuf>, Arc<ECStore>)> = OnceLock::new();
@@ -601,24 +602,21 @@ mod serial_tests {
     /// persisted free-version recovery -- so no live local metadata ever points
     /// at an already-removed remote version.
     ///
-    /// This test pins the FIXED contract two complementary ways, both
-    /// revert-proof (reverting to remote-first ordering turns them red):
+    /// This test pins the fixed contract with deterministic GET and DELETE
+    /// barriers (reverting to remote-first ordering turns it red):
     ///
-    ///  1. Ordering (deterministic): immediately after
-    ///     `expire_transitioned_object` returns, the remote tier object is still
-    ///     present and the mock recorded **zero** remote `remove` calls --
-    ///     proving the local delete happened with no synchronous remote removal
-    ///     (local-first). Remote-first ordering loses the object and records a
-    ///     `remove`.
-    ///  2. Concurrent GET (user-visible): a tight GET loop runs concurrently
-    ///     with the expiry; every observation must be either a full, correct
-    ///     body (GET won) or a clean object/version-not-found (expiry won). A
-    ///     tier-fetch failure -- the #3491 symptom -- is never tolerated.
+    ///  1. A GET that already resolved the transitioned metadata keeps its read
+    ///     lock and returns the complete remote body while expiry waits.
+    ///  2. Expiry returns after committing the local free-version without
+    ///     waiting for the post-commit worker's remote DELETE. While that DELETE
+    ///     is paused, the durable marker and remote body must both still exist.
+    ///  3. A later GET observes a clean object/version-not-found, never a tier
+    ///     fetch or read-quorum failure.
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
     #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-2)"]
     async fn test_expire_transitioned_object_never_races_concurrent_get() {
-        let (_disk_paths, ecstore) = setup_isolated_test_env(false).await;
+        let (disk_paths, ecstore) = setup_isolated_test_env(false).await;
 
         let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
         let backend = register_mock_tier(&tier_name).await;
@@ -664,59 +662,7 @@ mod serial_tests {
             "the regression must exercise an unversioned remote tier"
         );
 
-        // Concurrent GET loop: hammer GET while the expiry runs. Every outcome
-        // must be a full correct body or a clean not-found -- never a tier-fetch
-        // failure.
-        let get_store = ecstore.clone();
-        let get_bucket = bucket_name.clone();
-        let get_object = object_name.to_string();
-        let expected = payload.clone();
-        let get_loop = tokio::spawn(async move {
-            let mut saw_full_body = 0usize;
-            let mut saw_not_found = 0usize;
-            for _ in 0..400 {
-                match get_store
-                    .get_object_reader(
-                        get_bucket.as_str(),
-                        get_object.as_str(),
-                        None,
-                        http::HeaderMap::new(),
-                        &ObjectOptions::default(),
-                    )
-                    .await
-                {
-                    Ok(mut reader) => {
-                        let mut data = Vec::new();
-                        match reader.stream.read_to_end(&mut data).await {
-                            Ok(_) => {
-                                assert_eq!(
-                                    data, expected,
-                                    "a successful GET during expiry must return the complete, correct body"
-                                );
-                                saw_full_body += 1;
-                            }
-                            Err(err) => {
-                                panic!("GET during expiry streamed a truncated/failed body (expire/GET race regression): {err:?}")
-                            }
-                        }
-                    }
-                    Err(err) => {
-                        let ec: &EcstoreError = &err;
-                        assert!(
-                            is_err_object_not_found(ec) || is_err_version_not_found(ec),
-                            "GET during expiry may only fail with a clean object/version-not-found (expiry won \
-                             the race); a tier-fetch failure is the #3491 regression: {err:?}"
-                        );
-                        saw_not_found += 1;
-                    }
-                }
-                tokio::task::yield_now().await;
-            }
-            (saw_full_body, saw_not_found)
-        });
-
-        // Run the exact expiry action the scanner drives for a transitioned
-        // current version.
+        ExpiryState::resize_workers(1, ecstore.clone()).await;
         let lc_event = LcEvent {
             action: IlmAction::DeleteAction,
             ..Default::default()
@@ -725,39 +671,127 @@ mod serial_tests {
             .bucket_incarnation_id(bucket_name.as_str())
             .await
             .expect("read bucket incarnation");
-        expire_transitioned_object(ecstore.clone(), &oi, &lc_event, &LcEventSrc::Scanner, bucket_incarnation_id)
+
+        // Pause one real tier GET after it has resolved local transition
+        // metadata. The reader still owns the object read lock, so local expiry
+        // cannot commit until this GET finishes.
+        let get_barrier = backend.arm_get_barrier().await;
+        let get_store = ecstore.clone();
+        let get_bucket = bucket_name.clone();
+        let get_object = object_name.to_string();
+        let in_flight_get = tokio::spawn(async move {
+            let mut reader = get_store
+                .get_object_reader(
+                    get_bucket.as_str(),
+                    get_object.as_str(),
+                    None,
+                    http::HeaderMap::new(),
+                    &ObjectOptions::default(),
+                )
+                .await
+                .map_err(|err| format!("in-flight GET failed before streaming: {err:?}"))?;
+            let mut data = Vec::new();
+            reader
+                .stream
+                .read_to_end(&mut data)
+                .await
+                .map_err(|err| format!("in-flight GET returned a failed or truncated stream: {err:?}"))?;
+            Ok::<_, String>(data)
+        });
+        tokio::time::timeout(TRANSITION_WAIT_TIMEOUT, get_barrier.wait_until_paused())
             .await
+            .expect("the in-flight GET should reach the remote read barrier");
+
+        // The next remote DELETE pauses and then fails. A correct local-first
+        // expiry returns while this barrier is still held; synchronous cleanup
+        // (remote-first or local-first) instead times out here.
+        let delete_start_barrier = DeleteAfterObjectLockSnapshotBarrier::install(bucket_name.as_str());
+        let remove_barrier = backend.arm_failing_remove_barrier().await;
+        let expiry_store = ecstore.clone();
+        let expiry_oi = oi.clone();
+        let expiry_event = lc_event.clone();
+        let mut expiry = tokio::spawn(async move {
+            expire_transitioned_object(expiry_store, &expiry_oi, &expiry_event, &LcEventSrc::Scanner, bucket_incarnation_id).await
+        });
+        tokio::time::timeout(TRANSITION_WAIT_TIMEOUT, delete_start_barrier.wait_until_paused())
+            .await
+            .expect("expiry should reach the store delete path while the GET remains paused");
+        delete_start_barrier.release_and_wait_until_namespace_pending().await;
+        assert!(
+            !delete_start_barrier.namespace_acquired() && !expiry.is_finished(),
+            "expiry must wait for the in-flight GET's object read lock before committing the local delete"
+        );
+
+        get_barrier.release();
+        let expiry_outcome = tokio::time::timeout(TRANSITION_WAIT_TIMEOUT, &mut expiry).await;
+        let delete_lock_acquired_after_get = delete_start_barrier.namespace_acquired();
+        drop(delete_start_barrier);
+        let remove_arrival = tokio::time::timeout(TRANSITION_WAIT_TIMEOUT, remove_barrier.wait_until_paused()).await;
+
+        // Snapshot only lock-free observables while the cleanup worker holds
+        // the object write lock. Store API reads wait until the barrier is
+        // released below.
+        let free_version_persisted = free_version_count(&disk_paths[0], bucket_name.as_str(), object_name).await > 0;
+        let remote_present_at_cleanup = backend.contains(&remote_object).await;
+
+        remove_barrier.release();
+        let remove_operation_dropped = if remove_arrival.is_ok() {
+            Some(tokio::time::timeout(TRANSITION_WAIT_TIMEOUT, remove_barrier.wait_until_operation_dropped()).await)
+        } else {
+            None
+        };
+        if expiry_outcome.is_err() && tokio::time::timeout(TRANSITION_WAIT_TIMEOUT, &mut expiry).await.is_err() {
+            expiry.abort();
+            let _ = expiry.await;
+        }
+        let in_flight_get_outcome = tokio::time::timeout(TRANSITION_WAIT_TIMEOUT, in_flight_get).await;
+        let post_expiry_get = tokio::time::timeout(
+            TRANSITION_WAIT_TIMEOUT,
+            ecstore.get_object_reader(bucket_name.as_str(), object_name, None, http::HeaderMap::new(), &ObjectOptions::default()),
+        )
+        .await;
+
+        expiry_outcome
+            .expect("expire_transitioned_object must not wait for asynchronous remote-tier cleanup")
+            .expect("the expiry task should not panic")
             .expect("expire_transitioned_object should succeed");
+        assert!(
+            delete_lock_acquired_after_get,
+            "expiry must acquire the object write lock only after the in-flight GET releases its read lock"
+        );
+        remove_arrival.expect("the post-commit free-version worker should reach the remote DELETE barrier");
+        remove_operation_dropped
+            .expect("the remote DELETE should have reached the barrier")
+            .expect("the injected remote DELETE should finish after release");
+        assert!(
+            free_version_persisted,
+            "the durable free-version marker must exist before asynchronous remote cleanup"
+        );
+        assert!(
+            remote_present_at_cleanup,
+            "the remote object must remain readable until the paused cleanup DELETE is released"
+        );
 
-        // --- Ordering contract (deterministic revert-proof) ----------------
-        // #3491 defers remote cleanup to free-version recovery, so immediately
-        // after expiry the remote object is still present and NO synchronous
-        // remote `remove` was issued. Reverting to remote-first ordering makes
-        // both assertions fail.
+        let in_flight_body = in_flight_get_outcome
+            .expect("the in-flight GET should finish within the test deadline")
+            .expect("the in-flight GET task should not panic")
+            .expect("a GET that wins the expiry race must return a complete body");
         assert_eq!(
-            backend.remove_count().await,
-            0,
-            "expire_transitioned_object must NOT issue a synchronous remote-tier removal (local-first \
-             ordering, #3491); remote cleanup is deferred to free-version recovery"
-        );
-        assert!(
-            backend.contains(&remote_object).await,
-            "remote tier object must still exist immediately after expiry (deferred cleanup, #3491)"
+            in_flight_body, payload,
+            "a GET that resolved transitioned metadata before expiry must return the complete, correct body"
         );
 
-        // Local metadata is gone: the object is atomically unreachable.
-        assert!(
-            wait_for_object_absence(&ecstore, bucket_name.as_str(), object_name, Duration::from_secs(5)).await,
-            "local metadata for the expired transitioned object should be gone"
-        );
-
-        // Drain the concurrent GET loop; its internal asserts already guarantee
-        // no #3491-style tier-fetch failure was ever observed.
-        let (saw_full_body, saw_not_found) = get_loop.await.expect("concurrent GET loop task panicked");
-        assert!(
-            saw_full_body + saw_not_found > 0,
-            "the concurrent GET loop should have observed at least one GET outcome"
-        );
+        match post_expiry_get.expect("the post-expiry GET should finish within the test deadline") {
+            Ok(_) => panic!("the locally expired transitioned object must no longer be readable"),
+            Err(err) => {
+                let ec: &EcstoreError = &err;
+                assert!(
+                    is_err_object_not_found(ec) || is_err_version_not_found(ec),
+                    "a GET after expiry may only fail with a clean object/version-not-found; \
+                     a tier-fetch or read-quorum failure is the #3491 regression: {err:?}"
+                );
+            }
+        }
     }
 
     #[test]
@@ -1469,10 +1503,18 @@ mod serial_tests {
         let stale_remote_object = transitioned.transitioned_object.name.clone();
         assert!(backend.contains(&stale_remote_object).await);
 
-        ecstore
-            .delete_object(bucket_name.as_str(), object_name, ObjectOptions::default())
+        ExpiryState::resize_workers(1, ecstore.clone()).await;
+        let remove_barrier = backend.arm_failing_remove_barrier().await;
+        tokio::time::timeout(
+            Duration::from_secs(5),
+            ecstore.delete_object(bucket_name.as_str(), object_name, ObjectOptions::default()),
+        )
+        .await
+        .expect("DeleteObject must not wait for asynchronous remote-tier cleanup")
+        .expect("Failed to delete transitioned object before scanner fallback");
+        tokio::time::timeout(Duration::from_secs(5), remove_barrier.wait_until_paused())
             .await
-            .expect("Failed to delete transitioned object without expiry workers");
+            .expect("the immediate free-version worker should reach the injected remote DELETE barrier");
 
         assert!(
             free_version_count(&disk_paths[0], bucket_name.as_str(), object_name).await > 0,
@@ -1483,8 +1525,12 @@ mod serial_tests {
             "stale transitioned remote object should still exist before scanner fallback runs"
         );
 
-        init_background_expiry(ecstore.clone()).await;
+        // Queue the scanner fallback while the causal task is still blocked.
+        // Releasing the barrier fails only that first task, so the queued
+        // scanner task can prove durable-marker recovery on a healthy backend.
         scan_object_metadata(&disk_paths[0], bucket_name.as_str(), object_name).await;
+        remove_barrier.release();
+        remove_barrier.wait_until_operation_dropped().await;
 
         assert!(
             backend
@@ -1531,10 +1577,18 @@ mod serial_tests {
         let stale_remote_object = transitioned.transitioned_object.name.clone();
         assert!(backend.contains(&stale_remote_object).await);
 
-        ecstore
-            .delete_object(bucket_name.as_str(), object_name, ObjectOptions::default())
+        ExpiryState::resize_workers(1, ecstore.clone()).await;
+        let remove_barrier = backend.arm_failing_remove_barrier().await;
+        tokio::time::timeout(
+            Duration::from_secs(5),
+            ecstore.delete_object(bucket_name.as_str(), object_name, ObjectOptions::default()),
+        )
+        .await
+        .expect("DeleteObject must not wait for asynchronous remote-tier cleanup")
+        .expect("Failed to delete transitioned object after compensation-driven transition");
+        tokio::time::timeout(Duration::from_secs(5), remove_barrier.wait_until_paused())
             .await
-            .expect("Failed to delete transitioned object after compensation-driven transition");
+            .expect("the immediate free-version worker should reach the injected remote DELETE barrier");
 
         assert!(
             free_version_count(&disk_paths[0], bucket_name.as_str(), object_name).await > 0,
@@ -1545,8 +1599,12 @@ mod serial_tests {
             "stale transitioned remote object should still exist before scanner cleanup runs"
         );
 
-        init_background_expiry(ecstore.clone()).await;
+        // Enqueue the scanner fallback before the first, causal cleanup task is
+        // released into its injected failure. This keeps attribution
+        // deterministic and proves the durable marker drives convergence.
         scan_object_metadata(&disk_paths[0], bucket_name.as_str(), object_name).await;
+        remove_barrier.release();
+        remove_barrier.wait_until_operation_dropped().await;
 
         assert!(
             backend

crates/scanner/tests/storage_api/mod.rs

@@ -15,7 +15,9 @@
 pub(crate) use rustfs_ecstore::api::bucket::lifecycle::transition_transaction::recover_transition_transaction_records;
 pub(crate) use rustfs_ecstore::api::bucket::lifecycle::{
     bucket_lifecycle_audit::LcEventSrc,
-    bucket_lifecycle_ops::{enqueue_transition_for_existing_objects, expire_transitioned_object, init_background_expiry},
+    bucket_lifecycle_ops::{
+        ExpiryState, enqueue_transition_for_existing_objects, expire_transitioned_object, init_background_expiry,
+    },
     lifecycle::{Event as LcEvent, IlmAction, TRANSITION_PENDING, TransitionOptions},
 };
 pub(crate) use rustfs_ecstore::api::bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
@@ -27,6 +29,7 @@ pub(crate) use rustfs_ecstore::api::capacity::path2_bucket_object_with_base_path
 pub(crate) use rustfs_ecstore::api::disk::{DiskOption, STORAGE_FORMAT_FILE, endpoint::Endpoint, new_disk};
 pub(crate) use rustfs_ecstore::api::error::{Error as EcstoreError, is_err_object_not_found, is_err_version_not_found};
 pub(crate) use rustfs_ecstore::api::layout::{EndpointServerPools, Endpoints, PoolEndpoints};
+pub(crate) use rustfs_ecstore::api::object::test_util::DeleteAfterObjectLockSnapshotBarrier;
 pub(crate) use rustfs_ecstore::api::runtime::global_tier_config_mgr as get_global_tier_config_mgr;
 pub(crate) use rustfs_ecstore::api::storage::{ECStore, init_local_disks};
 // Shared lifecycle/tier test utilities (rustfs/backlog#1148 ilm-6). The mock
@@ -45,12 +48,12 @@ pub(crate) mod lifecycle {
     };
 
     pub(crate) use super::{
-        BUCKET_LIFECYCLE_CONFIG, BucketVersioningSys, DiskOption, ECStore, EcstoreError, Endpoint, EndpointServerPools,
-        Endpoints, IlmAction, LcEvent, LcEventSrc, MockWarmBackend, PoolEndpoints, STORAGE_FORMAT_FILE, TRANSITION_PENDING,
-        TransitionCleanupStoreBarrier, TransitionOptions, assert_transition_meta_consistent,
-        enqueue_transition_for_existing_objects, expire_transitioned_object, free_version_count, get_bucket_metadata,
-        get_global_tier_config_mgr, init_background_expiry, init_bucket_metadata_sys, init_local_disks, is_err_object_not_found,
-        is_err_version_not_found, new_disk, path2_bucket_object_with_base_path, recover_transition_transaction_records,
-        register_mock_tier_util, update_bucket_metadata, wait_for_free_version_absence,
+        BUCKET_LIFECYCLE_CONFIG, BucketVersioningSys, DeleteAfterObjectLockSnapshotBarrier, DiskOption, ECStore, EcstoreError,
+        Endpoint, EndpointServerPools, Endpoints, ExpiryState, IlmAction, LcEvent, LcEventSrc, MockWarmBackend, PoolEndpoints,
+        STORAGE_FORMAT_FILE, TRANSITION_PENDING, TransitionCleanupStoreBarrier, TransitionOptions,
+        assert_transition_meta_consistent, enqueue_transition_for_existing_objects, expire_transitioned_object,
+        free_version_count, get_bucket_metadata, get_global_tier_config_mgr, init_background_expiry, init_bucket_metadata_sys,
+        init_local_disks, is_err_object_not_found, is_err_version_not_found, new_disk, path2_bucket_object_with_base_path,
+        recover_transition_transaction_records, register_mock_tier_util, update_bucket_metadata, wait_for_free_version_absence,
     };
 }

docs/architecture/README.md

@@ -1,66 +1,57 @@
 # Architecture Documentation
 
-Durable architecture reference for RustFS: migration guardrails, runtime
-contracts, boundary rules, and support matrices.
+**Use this when:** you need the contract, invariant, or boundary rule that governs a change, and you want the one document that owns it.
+**Source of truth:** the code and the guards. `scripts/check_architecture_migration_rules.sh` enforces the CI-anchored documents below; `scripts/check_doc_paths.sh` fails the pre-commit gate when any doc under `docs/` cites a repository path that no longer exists.
 
 Two rules keep this directory healthy:
 
-1. **Durable reference only.** One-shot implementation plans, task trackers,
-   and PR templates do not belong in the repository — keep them in the issue
-   tracker or your local worktree. When their work closes, delete them rather
-   than archiving them here.
-2. **No copies of other sources of truth.** Crate lists come from
-   `Cargo.toml`, CI steps from `.github/workflows/ci.yml`, code structure from
-   the code. `scripts/check_doc_paths.sh` fails the pre-commit gate when a
-   doc here references a file path that no longer exists.
+1. **Durable reference only.** One-shot plans, task trackers, dated analyses, status snapshots, and PR-scoped notes do not belong in the repository; keep them in the issue tracker or a local worktree and delete them when the work closes.
+2. **No copies of other sources of truth.** Crate lists come from `Cargo.toml`, CI steps from `.github/workflows/`, code structure from the code. Cite a file path plus a symbol name, never a line number, and never paste counts or tables that a command can regenerate.
 
-## Start here
+Every document starts with a `**Use this when:**` line so an agent can decide in one glance whether to read further. The index below repeats those lines.
 
-- [overview.md](overview.md) — migration baseline, phase order, core principles
+## CI-anchored core
 
-## CI-enforced core (required by `scripts/check_architecture_migration_rules.sh`)
+Required headings and strings in these files are asserted by `scripts/check_architecture_migration_rules.sh`; rename a heading only together with the guard.
 
-- [crate-boundaries.md](crate-boundaries.md) — dependency direction, PR types, re-export contracts
-- [runtime-lifecycle.md](runtime-lifecycle.md) — startup/shutdown sequencing, readiness guarantees
-- [readiness-matrix.md](readiness-matrix.md) — request/dependency behavior, probe semantics
-- [storage-control-data-plane.md](storage-control-data-plane.md) — storage API contracts, control-plane boundaries
-- [global-state-crate-split-plan.md](global-state-crate-split-plan.md) — remaining global-state owners and split evaluation
-- [ecstore-module-split-plan.md](ecstore-module-split-plan.md) — ECStore decomposition rules and facade contracts
+| Document | Use this when |
+|---|---|
+| [crate-boundaries.md](crate-boundaries.md) | you add a crate dependency, move code across crates, touch a `storage_api.rs` boundary file, or need the change-type vocabulary the architecture guard enforces |
+| [runtime-lifecycle.md](runtime-lifecycle.md) | moving or reordering anything in `rustfs/src/startup_*.rs`, changing readiness publication, or touching shutdown ordering |
+| [readiness-matrix.md](readiness-matrix.md) | changing what a request surface does before storage or IAM is ready, changing probe semantics, or adding a runtime dependency that readiness must wait for |
+| [storage-control-data-plane.md](storage-control-data-plane.md) | adding a storage API surface, a cluster read model, or a background-service status/reconcile surface, and you need to know which layer owns it |
+| [global-state-crate-split-plan.md](global-state-crate-split-plan.md) | business logic needs runtime state (object store, endpoints, lock clients, lifecycle state, config) and you must pick the right boundary, or you are evaluating a crate split out of ECStore |
+| [global-state-inventory.md](global-state-inventory.md) | you meet a `GLOBAL_*` static or an `OnceLock` and need to know whether it is a runtime ownership handle, an owner-local static, or process-global by design |
+| [ecstore-module-split-plan.md](ecstore-module-split-plan.md) | you add lifecycle or replication logic and need to know which crate it belongs in, plan to move an operation family out of `SetDisks`, or the guard fails on one of the split rules |
+| [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md) | you need something from `rustfs_ecstore` in another crate, you are narrowing a `rustfs_ecstore::api` facade group, or the guard reports a facade bypass |
+| [obs-ecstore-dependency-inventory.md](obs-ecstore-dependency-inventory.md) | adding, removing, or moving any `rustfs_ecstore` or `rustfs_storage_api` reference inside `crates/obs` |
+| [compat-cleanup-register.md](compat-cleanup-register.md) | you add, review, or remove a temporary compatibility path and need the `RUSTFS_COMPAT_TODO` marker format and its removal condition |
+| [overview.md](overview.md) | you need the historical framing of the architecture-migration program or the phase names that other contracts refer to |
 
-## Contracts & invariants
+## Contracts and invariants
 
-- [erasure-coding.md](erasure-coding.md) — normative erasure-coding algorithm and on-disk (`xl.meta`) compatibility contract; the frozen invariants for all user-data read/write, encode/decode, quorum, heal, and decode tolerance
-- [placement-repair-invariants.md](placement-repair-invariants.md)
-- [unified-object-generation.md](unified-object-generation.md) — single per-object generation authority (fencing epoch, transport/encoding/proto/mixed-version contracts)
-- [runtime-capability-contracts.md](runtime-capability-contracts.md)
-- [workload-admission-contracts.md](workload-admission-contracts.md)
-- [background-controller-contract.md](background-controller-contract.md)
-- [config-model-boundary-adr.md](config-model-boundary-adr.md)
-- [ecstore-layout-boundary.md](ecstore-layout-boundary.md)
-- [decommission-compatibility.md](decommission-compatibility.md)
-- [kms-bulk-rekey-contract.md](kms-bulk-rekey-contract.md) — object-side DEK re-wrap job: work unit, idempotency model, exclusion rules, and the never-destroy-old-key-versions constraint
+| Document | Use this when |
+|---|---|
+| [erasure-coding.md](erasure-coding.md) | changing anything under `crates/ecstore/src/erasure/`, `crates/filemeta/`, `crates/ecstore/src/set_disk/`, storage-class or layout code, or any decode, quorum, or heal boundary (normative spec) |
+| [placement-repair-invariants.md](placement-repair-invariants.md) | changing anything that resolves an object to a pool, set, or disk, or that admits scanner or heal work |
+| [heal-concurrency-model.md](heal-concurrency-model.md) | changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that shares the `(bucket, object)` commit surface, or asking whether RustFS needs a persistent healing marker |
+| [unified-object-generation.md](unified-object-generation.md) | adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against the current object version |
+| [decommission-compatibility.md](decommission-compatibility.md) | changing pool decommission or rebalance behavior, its admin API shape, the persisted `PoolMeta` fields, or how tier free versions move between pools |
+| [ecstore-layout-boundary.md](ecstore-layout-boundary.md) | touching endpoint expansion, `FormatV3`, pool/set layout, or moving files between ECStore's internal directories |
+| [runtime-capability-contracts.md](runtime-capability-contracts.md) | changing the read-only observability or topology snapshot contracts in `rustfs-storage-api`, their providers, or the `storage_classes` payload of `GET /rustfs/admin/v4/runtime/capabilities` |
+| [workload-admission-contracts.md](workload-admission-contracts.md) | adding a workload class or snapshot provider, or consuming admission state from a background job |
+| [background-controller-contract.md](background-controller-contract.md) | adding a status snapshot or reconcile surface for a background service, or being tempted to fold several services into a generic controller |
+| [config-model-boundary-adr.md](config-model-boundary-adr.md) | touching the server-config model (`Config`, `KV`, `KVS`) or its persistence, or asking which crate owns which part of server configuration |
+| [admin-route-action-snapshot.md](admin-route-action-snapshot.md) | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → `AdminAction` contract is enforced |
+| [kms-bulk-rekey-contract.md](kms-bulk-rekey-contract.md) | changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch |
 
-## Support matrices (release-facing, keep current)
+## Support and compatibility matrices (release-facing, keep current)
 
-- [s3-compatibility-matrix.md](s3-compatibility-matrix.md)
-- [s3-tables-support-matrix.md](s3-tables-support-matrix.md)
-- [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md)
-- [minio-file-format-compat.md](minio-file-format-compat.md)
+| Document | Use this when |
+|---|---|
+| [s3-compatibility-matrix.md](s3-compatibility-matrix.md) | writing or checking a user-facing S3 compatibility claim, or moving a Ceph s3tests case between lists |
+| [s3-tables-support-matrix.md](s3-tables-support-matrix.md) | writing a release note or client-compatibility statement about S3 Tables / Iceberg REST Catalog (cutover procedure: [../operations/s3-tables-cutover-runbook.md](../operations/s3-tables-cutover-runbook.md)) |
+| [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md) | a client or `mc` call that works against MinIO fails against RustFS and you need to know whether the endpoint is missing, stubbed, or deliberately different |
+| [minio-file-format-compat.md](minio-file-format-compat.md) | deciding whether a MinIO drive set, bucket-metadata blob, or SSE object can be read or imported by a given RustFS build, or before touching a listed version anchor |
 
-## Inventories & baselines (snapshots that feed migration work)
-
-- [global-state-inventory.md](global-state-inventory.md)
-- [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md)
-- [ecstore-config-consumer-inventory.md](ecstore-config-consumer-inventory.md)
-- [obs-ecstore-dependency-inventory.md](obs-ecstore-dependency-inventory.md)
-- [background-services-inventory.md](background-services-inventory.md)
-- [scanner-heal-admission.md](scanner-heal-admission.md)
-- [admin-route-action-snapshot.md](admin-route-action-snapshot.md)
-- [compat-cleanup-register.md](compat-cleanup-register.md)
-
-Historical plans and trackers (rebalance/decommission phases,
-migration-progress ledger, and the one-shot migration snapshots that fed it —
-startup timeline, scheduler baseline, profiling/NUMA capability inventory, KMS
-development defaults inventory) were retired in 2026-07 once the
-architecture-review ledger they served closed out (backlog#660/#665). Planning
-documents are no longer kept in the repository.
+Operations runbooks live in [../operations/](../README.md#operations) and testing references in [../testing/README.md](../testing/README.md).

docs/architecture/admin-route-action-snapshot.md

@@ -1,145 +1,27 @@
 # Admin Route Action Snapshot
 
-This snapshot records the current admin routing and authorization surface before
-directory moves or crate extraction. It is a migration guardrail: later pure
-move PRs must preserve the route, handler, authorization action, public
-exception, and compatibility alias semantics listed here unless the PR is
-explicitly scoped as a behavior change.
+**Use this when:** you add, move, or re-authorize an admin route and need to know where the route → handler → `AdminAction` contract is enforced.
+**Source of truth:** `rustfs/src/admin/route_policy.rs` (the `AdminRouteSpec` matrix, checked by `validate_admin_route_policy_specs`), `rustfs/src/admin/route_registration_test.rs` (registration coverage), `rustfs/src/admin/router.rs` (dispatch and credential checks), `rustfs/src/admin/handlers/*.rs` (handler-level authorization calls).
 
-## Source Of Truth
-
-- Router assembly: `rustfs/src/admin/mod.rs::make_admin_route`
-- Route registration coverage: `rustfs/src/admin/route_registration_test.rs`
-- Runtime dispatch: `rustfs/src/admin/router.rs`
-- Admin auth helpers: `rustfs/src/admin/auth.rs`
-- Handler route/action ownership: `rustfs/src/admin/handlers/*.rs`
-
-The route registration test intentionally covers representative paths for every
-registered route family. This document uses route patterns from the registration
-functions and action names from the handler authorization calls.
+This page is a pointer, not a route table. The machine-checked matrix in `route_policy.rs` lists every admin route with its `AdminAction` and `RouteRiskLevel`; routes that are registered but answered by policy instead of a handler are declared there too through `DeferredRoutePolicyReason`. The `AdminRouteSpec` type lives in `crates/security-governance/src/admin_matrix.rs`.
 
 ## Prefix And Alias Contract
 
-| Prefix | Current behavior | Migration rule |
+| Prefix | Behavior | Rule |
 |---|---|---|
-| `/rustfs/admin` | Canonical admin API prefix used by route registration | Keep as the single registered admin prefix |
-| `/minio/admin` | Compatibility alias accepted by `S3Router::is_match`; dispatch canonicalizes it to `/rustfs/admin` | Do not duplicate registrations; preserve canonicalization |
-| `/iceberg/v1` table catalog prefix | Registered through `table_catalog::register_table_catalog_route` and accepted by `is_admin_path` | Keep outside `/rustfs/admin` and document auth separately |
-| `/health` and `/health/ready` | Public health endpoints when `ENV_HEALTH_ENDPOINT_ENABLE` allows registration | Preserve unauthenticated health bypass |
-| `/profile/cpu` and `/profile/memory` | Registered by health handler but guarded by profile auth | Do not couple to health endpoint enablement |
-
-The compatibility alias is not a second route table. `canonicalize_admin_path`
-maps `/minio/admin/...` to `/rustfs/admin/...` immediately before route lookup.
-
-## Dispatch And Auth Shape
-
-```mermaid
-flowchart LR
-    A["Incoming request"] --> B{"S3Router::is_match"}
-    B -->|"Replication or misc extension"| X["Extension handler"]
-    B -->|"Health path"| H["Public health"]
-    B -->|"OIDC public path"| O["OIDC public handler"]
-    B -->|"POST / STS form"| S["STS handler"]
-    B -->|"Admin or console path"| C{"S3Router::check_access"}
-    C -->|"public exception"| P["No SigV4 required"]
-    C -->|"admin route"| D["Credential required"]
-    D --> E["canonicalize /minio/admin to /rustfs/admin"]
-    E --> F["matchit route lookup"]
-    F --> G["AdminOperation handler"]
-    G --> I["handler-level validate_admin_request"]
-```
-
-Route-level credential presence and handler-level policy authorization are
-separate contracts. The router enforces credential presence for ordinary admin
-routes. Handler rows below record whether the current handler performs a
-precise `AdminAction` or `S3Action` check, or only repeats a credential
-presence check.
+| `/rustfs/admin` | Canonical admin prefix used by `make_admin_route` (`rustfs/src/admin/mod.rs`) | The only registered admin prefix |
+| `/minio/admin` | Compatibility alias accepted by `S3Router::is_match`; `canonicalize_admin_path` rewrites it to `/rustfs/admin` immediately before route lookup (`rustfs/src/admin/router.rs`) | Never register routes twice; preserve canonicalization |
+| `/iceberg/v1` | Table catalog prefix registered by `register_table_catalog_route` (`rustfs/src/admin/handlers/table_catalog/routes.rs`) and accepted by `is_admin_path` | Stays outside `/rustfs/admin`; table actions are authorized per handler |
+| `/health`, `/health/ready` | Public health endpoints, registered only when `ENV_HEALTH_ENDPOINT_ENABLE` allows | Preserve the unauthenticated bypass |
+| `/profile/cpu`, `/profile/memory` | Registered by the health handler but guarded by profile authorization | Never couple to health-endpoint enablement |
 
 ## Public Exceptions
 
-| Method | Path pattern | Handler | Auth contract |
-|---|---|---|---|
-| `GET`, `HEAD` | `/health` | `HealthCheckHandler` | Public when health routes are registered |
-| `GET`, `HEAD` | `/health/ready` | `HealthCheckHandler` | Public when health routes are registered |
-| Registered as `GET`; auth bypass is path-based | `/rustfs/admin/v3/oidc/providers` and `/minio/admin/v3/oidc/providers` | `ListOidcProvidersHandler` | Public OIDC bootstrap path; `check_access` bypasses SigV4 for any method matching this path |
-| Registered as `GET`; auth bypass is path-prefix-based | `/rustfs/admin/v3/oidc/authorize/{provider_id}` and `/minio/admin/v3/oidc/authorize/{provider_id}` | `OidcAuthorizeHandler` | Public OIDC bootstrap path; `check_access` bypasses SigV4 for any method matching this path prefix |
-| Registered as `GET`; auth bypass is path-prefix-based | `/rustfs/admin/v3/oidc/callback/{provider_id}` and `/minio/admin/v3/oidc/callback/{provider_id}` | `OidcCallbackHandler` | Public OIDC bootstrap path; `check_access` bypasses SigV4 for any method matching this path prefix |
-| Registered as `GET`; auth bypass is path-based | `/rustfs/admin/v3/oidc/logout` and `/minio/admin/v3/oidc/logout` | `OidcLogoutHandler` | Public OIDC logout path; `check_access` bypasses SigV4 for any method matching this path |
-| `POST` | `/` with `application/x-www-form-urlencoded` | `AssumeRoleHandle` | Public only for unsigned STS web identity form requests; handler validates JWT/action |
-| Any matched method | `/favicon.ico` and `/rustfs/console...` | Console router | Public only when `console_enabled` is true; router bypasses SigV4 before handing off to the console router |
+Router-level credential checks (`S3Router::check_access`) are bypassed only for:
 
-## Registered Route Families
+- health routes, when they are registered;
+- OIDC bootstrap paths matched by `is_oidc_path` (`providers`, `authorize/{provider_id}`, `callback/{provider_id}`, `logout`); the bypass is path-based, so it applies to any method on those paths;
+- unsigned STS web-identity form posts to `/` with `application/x-www-form-urlencoded`, which the STS handler validates itself;
+- console assets (`/favicon.ico`, `/rustfs/console...`), only while the console is enabled.
 
-All rows with `/rustfs/admin` also accept the `/minio/admin` compatibility alias
-through router canonicalization unless the row explicitly says otherwise.
-
-| Area | Methods and path patterns | Handler ownership | Authorization contract |
-|---|---|---|---|
-| STS and admin probe | `POST /`; `GET /rustfs/admin/v3/is-admin` | `sts.rs`, `is_admin.rs` | STS dispatch validates request action; is-admin checks `AllAdminActions` |
-| User lifecycle | `GET /v3/list-users`; `GET /v3/user-info`; `PUT /v3/add-user`; `PUT /v3/set-user-status`; `DELETE /v3/remove-user` | `user_lifecycle.rs`, `user.rs` | `ListUsersAdminAction`, `GetUserAdminAction`, `CreateUserAdminAction`, `EnableUserAdminAction`, `DeleteUserAdminAction` |
-| Group management | `GET /v3/groups`; `GET /v3/group`; `DELETE /v3/group/{group}`; `PUT /v3/set-group-status`; `PUT /v3/update-group-members` | `group.rs` | `ListGroupsAdminAction`, `GetGroupAdminAction`, `RemoveUserFromGroupAdminAction`, `EnableGroupAdminAction`, `AddUserToGroupAdminAction` |
-| Service accounts | `PUT /v3/add-service-account(s)`; `POST /v3/update-service-account`; `GET /v3/info-service-account`; `GET /v3/temporary-account-info`; `GET /v3/info-access-key`; `GET /v3/list-service-accounts`; `GET /v3/list-access-keys-bulk`; `DELETE /v3/delete-service-account(s)` | `service_account.rs` | create/update/list/temp-info/user-list/remove service account actions as checked in handler context |
-| IAM import/export | `GET /v3/export-iam`; `PUT /v3/import-iam` | `user_iam.rs`, `user.rs` | `ExportIAMAction`, `ImportIAMAction` |
-| IAM policies | `GET /v3/list-canned-policies`; `GET /v3/info-canned-policy`; `PUT /v3/add-canned-policy`; `DELETE /v3/remove-canned-policy`; `PUT /v3/set-user-or-group-policy`; `PUT /v3/set-policy`; `POST /v3/idp/builtin/policy/attach`; `POST /v3/idp/builtin/policy/detach`; `GET /v3/idp/builtin/policy-entities` | `policies.rs` | list/create/get/delete/attach policy actions; policy-entities combines list groups, users, and policies |
-| Account info | `GET /v3/accountinfo` | `account_info.rs` | S3 action checks for account-scoped bucket and object probes |
-| System info | `GET /v3/info`; `GET /v3/storageinfo`; `GET /v3/datausageinfo` | `system.rs` | `ServerInfoAdminAction`, `StorageInfoAdminAction`, `DataUsageInfoAdminAction` plus `ListBucketAction` for data usage |
-| Metrics stream | `GET /v3/metrics` | `metrics.rs` through `system.rs` | Router credential presence plus handler credential check; no handler-level `AdminAction` is currently enforced |
-| System service placeholders | `POST /v3/service`; `GET|POST /v3/inspect-data` | `system.rs` | Currently registered but handler returns `NotImplemented`; migration must preserve this unless behavior changes |
-| Pools | `GET /v3/pools/list`; `GET /v3/pools/status`; `POST /v3/pools/decommission`; `POST /v3/pools/cancel` | `pools.rs` | list/status accept server-info or decommission; decommission/cancel use `DecommissionAdminAction` |
-| Rebalance | `POST /v3/rebalance/start`; `GET /v3/rebalance/status`; `POST /v3/rebalance/stop` | `rebalance.rs` | `RebalanceAdminAction` |
-| Heal | `POST /v3/heal/`; `POST /v3/heal/{bucket}`; `POST /v3/heal/{bucket}/{prefix}`; `POST /v3/background-heal/status`; `GET /v4/heal/replacement-recovery` | `heal.rs` | `HealAdminAction` |
-| Tier | `GET /v3/tier`; `GET /v3/tier-stats`; `GET /v3/tier/{tier}`; `DELETE /v3/tier/{tiername}`; `PUT /v3/tier`; `POST /v3/tier/{tiername}`; `POST /v3/tier/clear` | `tier.rs` | `ListTierAction` for reads/status; `SetTierAction` for add/edit/remove/clear |
-| Quota legacy and bucket-scoped | `PUT /v3/set-bucket-quota`; `GET /v3/get-bucket-quota`; `PUT|GET|DELETE /v3/quota/{bucket}`; `GET /v3/quota-stats/{bucket}`; `POST /v3/quota-check/{bucket}` | `quota.rs` | `SetBucketQuotaAdminAction` for writes; `GetBucketQuotaAction` for bucket-scoped reads/stats/checks |
-| Bucket metadata | `GET /export-bucket-metadata`; `GET /v3/export-bucket-metadata`; `PUT /import-bucket-metadata`; `PUT /v3/import-bucket-metadata` | `bucket_meta.rs` | `ExportBucketMetadataAction`, `ImportBucketMetadataAction` |
-| Server config | `GET /v3/get-config-kv`; `PUT /v3/set-config-kv`; `DELETE /v3/del-config-kv`; `GET /v3/help-config-kv`; `GET /v3/list-config-history-kv`; `DELETE /v3/clear-config-history-kv`; `PUT /v3/restore-config-history-kv`; `GET|PUT /v3/config` | `config_admin.rs` | `ConfigUpdateAdminAction` helper path; read/write handlers preserve current per-handler checks |
-| Scanner | `GET /v3/scanner/status` | `scanner.rs` | `ServerInfoAdminAction` |
-| Notification targets | `GET /v3/target/list`; `GET /v3/target/arns`; `PUT /v3/target/{target_type}/{target_name}`; `DELETE /v3/target/{target_type}/{target_name}/reset` | `event.rs` through `user_policy_binding.rs` | `GetBucketTargetAction` for list/ARNs; `SetBucketTargetAction` for put/delete |
-| Audit targets | `GET /v3/audit/target/list`; `PUT /v3/audit/target/{target_type}/{target_name}`; `DELETE /v3/audit/target/{target_type}/{target_name}/reset` | `audit.rs` | `GetBucketTargetAction` for list; `SetBucketTargetAction` for put/delete |
-| Module switches | `GET|PUT /v3/module-switches` | `module_switch.rs` | `ServerInfoAdminAction` for get; `ConfigUpdateAdminAction` for update |
-| Plugin catalog | `GET /v4/plugins/catalog` | `plugins_catalog.rs` | `ServerInfoAdminAction` |
-| Plugin instances | `GET /v4/plugins/instances`; `GET|PUT|DELETE /v4/plugins/instances/{id}` | `plugins_instances.rs` | read uses `GetBucketTargetAction`; write/delete use `SetBucketTargetAction` |
-| Replication target list | `GET /v3/list-remote-targets` | `replication.rs` | Router credential presence plus handler credential check; no handler-level `AdminAction` is currently enforced |
-| Replication target metrics/mutation | `GET /v3/replicationmetrics`; `PUT /v3/set-remote-target`; `DELETE /v3/remove-remote-target` | `replication.rs` | `GetReplicationMetricsAction` for metrics; `SetBucketTargetAction` for target mutation |
-| Site replication | `PUT /v3/site-replication/add`; `PUT /v3/site-replication/remove`; `GET /v3/site-replication/info`; `GET /v3/site-replication/metainfo`; `GET /v3/site-replication/status`; `POST /v3/site-replication/devnull`; `POST /v3/site-replication/netperf`; `PUT /v3/site-replication/edit`; `PUT /v3/site-replication/peer/join`; `PUT /v3/site-replication/peer/bucket-ops`; `PUT /v3/site-replication/peer/iam-item`; `PUT /v3/site-replication/peer/bucket-meta`; `GET /v3/site-replication/peer/idp-settings`; `PUT /v3/site-replication/peer/edit`; `PUT /v3/site-replication/peer/remove`; `PUT /v3/site-replication/resync/op`; `PUT /v3/site-replication/state/edit` | `site_replication.rs` | add/remove/info/operation/resync actions selected per handler |
-| Admin profiling | `GET /rustfs/admin/debug/pprof/profile`; `GET /rustfs/admin/debug/pprof/status` | `profile_admin.rs`, `profile.rs` | `ProfilingAdminAction` |
-| TLS debug | `GET /rustfs/admin/debug/tls/status` | `tls_debug.rs`, `profile.rs` | `ProfilingAdminAction` via shared profile authorization |
-| KMS legacy management | `POST /v3/kms/create-key`; `POST /v3/kms/key/create`; `GET /v3/kms/describe-key`; `GET /v3/kms/key/status`; `GET /v3/kms/list-keys`; `POST /v3/kms/generate-data-key`; `GET|POST /v3/kms/status`; `GET /v3/kms/config`; `POST /v3/kms/clear-cache` | `kms_management.rs`, `kms_keys.rs` | dedicated `kms:*` actions throughout; `kms:ServiceControl` for the status paths, `kms:Configure` for config, `kms:ClearCache` for cache. No `ServerInfoAdminAction` fallback remains on any KMS route |
-| KMS dynamic control | `POST /v3/kms/configure`; `POST /v3/kms/start`; `POST /v3/kms/stop`; `GET /v3/kms/service-status`; `POST /v3/kms/reconfigure` | `kms_dynamic.rs` | `kms:Configure` for configure/reconfigure; `kms:ServiceControl` for start/stop/service-status |
-| KMS keys | `POST /v3/kms/keys`; `DELETE /v3/kms/keys/delete`; `POST /v3/kms/keys/cancel-deletion`; `GET /v3/kms/keys`; `GET /v3/kms/keys/{key_id}` | `kms_keys.rs` | dedicated `kms:*` actions per handler |
-| OIDC public | `GET /v3/oidc/providers`; `GET /v3/oidc/authorize/{provider_id}`; `GET /v3/oidc/callback/{provider_id}`; `GET /v3/oidc/logout` | `oidc.rs` | Public OIDC exception in `is_oidc_path` |
-| OIDC config | `GET /v3/oidc/config`; `PUT|DELETE /v3/oidc/config/{provider_id}`; `POST /v3/oidc/validate` | `oidc.rs` | `ServerInfoAdminAction` for read/validate; `ConfigUpdateAdminAction` for mutation |
-
-## Table Catalog Routes
-
-The table catalog API is registered by the admin router but is not under
-`/rustfs/admin`. It has its own prefix and Iceberg-style route shape.
-
-| Method | Path pattern | Handler | Authorization action |
-|---|---|---|---|
-| `GET` | `/iceberg/v1/config` | `GET_CONFIG_HANDLER` | `GetTableCatalogAction` |
-| `GET` | `/iceberg/v1/{warehouse}/namespaces` | `LIST_NAMESPACES_HANDLER` | `GetTableNamespaceAction` |
-| `POST` | `/iceberg/v1/{warehouse}/namespaces` | `CREATE_NAMESPACE_HANDLER` | `SetTableNamespaceAction` |
-| `GET` | `/iceberg/v1/{warehouse}/namespaces/{namespace}` | `GET_NAMESPACE_HANDLER` | `GetTableNamespaceAction` |
-| `DELETE` | `/iceberg/v1/{warehouse}/namespaces/{namespace}` | `DROP_NAMESPACE_HANDLER` | `DeleteTableNamespaceAction` |
-| `GET` | `/iceberg/v1/{warehouse}/namespaces/{namespace}/tables` | `LIST_TABLES_HANDLER` | `GetTableAction` |
-| `POST` | `/iceberg/v1/{warehouse}/namespaces/{namespace}/tables` | `CREATE_TABLE_HANDLER` | `CreateTableAction` |
-| `POST` | `/iceberg/v1/{warehouse}/namespaces/{namespace}/register` | `REGISTER_TABLE_HANDLER` | `RegisterTableAction` |
-| `GET` | `/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}` | `LOAD_TABLE_HANDLER` | `GetTableAction` |
-| `POST` | `/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}` | `COMMIT_TABLE_HANDLER` | `CommitTableAction` |
-| `DELETE` | `/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}` | `DROP_TABLE_HANDLER` | `DeleteTableAction` |
-
-## Migration Rules
-
-1. Pure move PRs may move handler modules, but must not change registered
-   methods, patterns, handler ownership, alias canonicalization, or public
-   exception behavior.
-2. If an admin handler is wrapped to cut a dependency direction, the wrapper
-   must preserve the same `AdminAction` or `S3Action` check and keep response
-   compatibility unchanged.
-3. Do not duplicate `/minio/admin` registrations. The alias remains a router
-   canonicalization concern.
-4. Do not move table catalog routes under `/rustfs/admin` during route cleanup.
-5. Registered-but-`NotImplemented` routes are behavior contracts too. Removing
-   or implementing them requires a behavior-change PR type.
-6. Future route matrix automation should compare against this document and
-   `route_registration_test.rs` before crate extraction begins.
+Every other admin route requires credentials at the router and a precise `AdminAction` or `S3Action` check in the handler (metrics routes, for example, authorize `GetMetricsAction`). The MinIO alias contract is specified in [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md).

docs/architecture/background-controller-contract.md

@@ -1,187 +1,55 @@
 # Background Controller Contract
 
-This document defines `BGC-002` for
-[`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660). It turns
-the background service inventory into a shared vocabulary for future read-only
-status work. It does not add a Rust trait, a scheduler, a service registry, or
-any worker start/stop behavior.
+**Use this when:** you add a status snapshot or reconcile surface for a background service (scanner, heal, lifecycle, replication, config reload, capacity, metrics, memory observability, allocator reclaim, auto-tuner), or you are tempted to fold several of them into a generic controller.
+**Source of truth:** the shipped reference surfaces — `MemoryObservabilityReconcilePlan` and `reconcile()` in `rustfs/src/memory_observability.rs`, `AllocatorReclaimControllerSnapshot` and `AllocatorReclaimReconcilePlan` in `rustfs/src/allocator_reclaim.rs`, `MetricsRuntimeReconcilePlan` in `crates/obs/src/metrics/scheduler.rs`. Startup and shutdown ordering is owned by [runtime-lifecycle.md](runtime-lifecycle.md); the plane-level overview is in [storage-control-data-plane.md](storage-control-data-plane.md).
 
-## Scope
+There is no `BackgroundController` trait, scheduler, or service registry. Each service exposes its own typed snapshot and reconcile plan; this page fixes the vocabulary and the rules those surfaces follow.
 
-- PR type: `docs-only`.
-- Baseline: `upstream/main` at
-  `f9a5e6d7e67322ac6f626b6f437a5e722fbe22e2`.
-- Applies to future controller work for scanner, heal, lifecycle, replication,
-  dynamic config reload, capacity, metrics, memory observability, allocator
-  reclaim, and auto-tuning.
-- Out of scope: worker creation, worker shutdown, queue resizing, storage
-  writes, readiness changes, peer signaling changes, scheduler replacement, and
-  crate splitting.
+## Vocabulary
 
-## Contract Vocabulary
-
-| Term | Meaning | BGC-002 boundary |
+| Term | Meaning | Boundary |
 |---|---|---|
-| Desired | Static intent from env, persisted config, module switches, feature flags, bucket config, or admin configuration. | Read only. Do not normalize or mutate config while collecting desired state. |
-| Current | Observed local runtime state such as configured, disabled, running, degraded, stopping, or unknown. | Read only. Do not infer state by starting probes that create storage or network side effects. |
-| Status | Human-readable and machine-checkable snapshot of runtime counters, worker counts, queue pressure, last successful cycle, last error, cancellation source, and shutdown handle shape. | Side-effect-free. Missing status surfaces must be reported as `unknown`, not guessed. |
-| Reconcile | Future comparison between desired, current, and status that can produce a recommendation. | No action in `BGC-002`; future reconcile must not start or stop workers until a tested pilot PR allows it. |
-| Side effects | Writes, deletes, queue admission, target activation, external I/O, metrics emission, readiness publication, peer signal, or config reload fanout. | Must be declared before any controller migration touches that service. |
+| Desired | Static intent from env, persisted config, module switches, feature flags, bucket config, or admin configuration. | Read only; collecting desired state never normalizes or mutates config. |
+| Current | Observed local runtime state: configured, disabled, running, degraded, stopping, or unknown. | Read only; never inferred by probes that create storage or network side effects. |
+| Status | Machine-checkable snapshot of counters, worker counts, queue pressure, last cycle, last error, cancellation source, and shutdown-handle shape. | Side-effect-free; a missing surface is reported as `unknown`, never guessed. |
+| Reconcile | Comparison of desired, current, and status that yields a plan. | Shipped plans only report; the only worker mutation they may request is `none`. |
+| Side effects | Writes, deletes, queue admission, target activation, external I/O, metrics emission, readiness publication, peer signals, config reload fanout. | Declared per service before any controller touches it. |
 
 ## State Model
 
-Future status snapshots should use the narrowest state that the current code can
-prove:
+Snapshots use the narrowest state the code can prove:
 
 | State | Meaning | Notes |
 |---|---|---|
-| NotConfigured | No valid desired source exists for this service. | Use when config/module switches/features make the service absent. |
-| Disabled | Desired source exists and explicitly disables the service. | Do not use for missing config. |
-| Starting | Startup was requested and has not reached steady state. | Only expose when current code has a start boundary. |
-| Running | The service is active according to existing runtime state. | Do not use merely because config is enabled. |
-| Degraded | The service is active but current status exposes known error, partial, or stalled state. | Do not introduce new failure classification in docs-only work. |
-| Stopping | Shutdown was requested and the service has not fully exited. | Only expose where shutdown can be observed. |
-| Stopped | The service was started before and is now fully stopped. | Do not confuse with `Disabled` or `NotConfigured`. |
-| Unknown | Current code lacks a safe status surface. | Preferred over speculative status. |
-
-## Lifecycle Boundary
-
-```mermaid
-flowchart LR
-    D["Desired source"]
-    C["Current runtime state"]
-    S["Read-only status snapshot"]
-    R["Future reconcile recommendation"]
-    W["Workers and side effects"]
-
-    D --> S
-    C --> S
-    S --> R
-    R -. "future tested pilot only" .-> W
-```
-
-`BGC-002` stops at the read-only contract. The arrow from reconcile to workers is
-intentionally dotted because this PR does not allow any implementation to start,
-stop, resize, or reconfigure workers.
-
-## Service Boundaries
-
-| Service area | Desired source | Current/status inputs | Side effects to preserve |
-|---|---|---|---|
-| Data scanner | Scanner env and runtime scanner config. | Admin scanner status, scanner metrics, scanner cancellation token, checkpoint/yield/alert counters. | Data usage cache updates, lifecycle evaluation, replication heal admission, scanner heal admission, alerts, and scanner metrics. |
-| Heal/AHM | Heal enablement and scanner-driven heal admission. | Heal manager global channel, active task atomics, queue length atomics, AHM cancellation token. | Heal queue consumption, heal storage writes, and channel close semantics. |
-| Lifecycle expiry/transition | Bucket lifecycle config and scanner event source. | Lifecycle worker counts, active tasks, queue send timeouts, transition stats, expiry/transition queues. | Object deletes, transition queueing, stale multipart cleanup, and lifecycle metrics. |
-| Replication pool | Bucket/site replication config and resync admin requests. | Global replication stats, worker pool sizes, queue counters, persisted resync state, per-bucket cancel tokens. | Object replication, delete replication, queue resizing by channel close, persisted resync metadata, and admin-triggered cancel paths. |
-| Dynamic config reload | Persisted server config, admin config calls, and peer snapshot signals. | Last local reload result, per-subsystem reload errors, peer reload signal result. | Scanner/heal runtime config updates, audit reload, notification reload, peer signaling, and config snapshot fanout. |
-| Capacity manager | Local disk inventory and capacity feature state. | Capacity manager cache age, scheduled refresh state, last refresh result, runtime summary loop. | Global capacity cache refresh and runtime summary metrics/logging. |
-| Metrics runtime | Observability metrics feature state and collector configuration. | Collector intervals, last collection result, cancellation token state, collector grouping. | Metrics collection and emission only. |
-| Memory observability | Observability feature state and memory sampling config. | Sampler loop state, last sample time, last sample error, runtime cancellation token. | Memory metric emission. This is the preferred first BGC-003 status candidate. |
-| Allocator reclaim | Allocator reclaim env/config and backend support. | Enabled flag, idle streak, active request gauge, scanner/heal activity gauges, last reclaim result. | Backend-specific allocator reclaim and metrics. |
-| Auto-tuner | `RUSTFS_AUTOTUNER_ENABLED` and tuning inputs. | Last tuning attempt, last tuning error, 60-second loop state. | Runtime concurrency tuning. Treat as behavior-sensitive. |
-
-The following areas stay outside the first controller migrations:
-
-- deferred IAM recovery, because it can publish readiness;
-- optional protocol servers, because they already have protocol shutdown handles;
-- ECStore endpoint monitor and disk health monitor, because they are storage-
-  adjacent and can affect disk state;
-- notification and audit runtime coupling, because live streams, replay, target
-  activation, and reload behavior need dedicated preservation tests.
+| NotConfigured | No valid desired source exists. | Config, module switches, or features make the service absent. |
+| Disabled | A desired source exists and explicitly disables the service. | Not for missing config. |
+| Starting | Start requested, steady state not reached. | Only where a start boundary exists. |
+| Running | Active according to existing runtime state. | Not merely because config is enabled. |
+| Degraded | Active with known error, partial, or stalled status. | No new failure classification is invented for a snapshot. |
+| Stopping | Shutdown requested, not fully exited. | Only where shutdown is observable. |
+| Stopped | Started earlier, now fully stopped. | Distinct from `Disabled` and `NotConfigured`. |
+| Unknown | No safe status surface exists. | Preferred over speculation. |
 
 ## Read-Only Snapshot Requirements
 
-Any future `BGC-003` status implementation must satisfy all of these:
+- Status collection never starts, stops, resizes, or wakes a worker.
+- Status collection never writes storage data, object metadata, target state, queue entries, persisted config, or resync metadata.
+- Status collection never publishes readiness or peer reload signals.
+- Missing fields are `unknown` or omitted with a documented reason.
+- Cancellation source and shutdown-handle shape are reported separately from desired enabled/disabled state.
+- Repeated `reconcile` calls over the same snapshot return the same plan.
+- Scanner, heal, lifecycle, and replication status must not hide their queue and admission coupling.
 
-- status collection must not start, stop, resize, or wake a worker;
-- status collection must not write storage data, object metadata, target state,
-  queue entries, persisted config, or resync metadata;
-- status collection must not publish readiness or peer reload signals;
-- missing fields must be represented as `unknown` or omitted with a documented
-  reason;
-- cancellation source and shutdown handle shape must be reported separately from
-  desired enabled/disabled state;
-- scanner, heal, lifecycle, and replication status must not hide their queue and
-  admission coupling.
+## Coupling Notes
 
-## BGC-003 Snapshot Pilot
+The services below share state or shutdown contracts and must not be folded into a generic controller without service-specific preservation tests:
 
-The first read-only snapshot is memory observability status. It reports the
-service name, whether observability metrics currently enable the sampler, the
-configured sampler interval, runtime-token cancellation state, and the absence
-of a dedicated shutdown handle.
-
-This snapshot intentionally does not define an admin route, scheduler, service
-registry, worker start/stop path, readiness signal, peer signal, storage write,
-or metrics emission change.
-
-## BGC-004 Controller Pilot
-
-The first controller pilot is also memory observability. It converts the
-existing desired inputs and status snapshot into a typed reconcile plan. The
-pilot reports desired state, current state, and worker mutation intent.
-
-The only allowed worker mutation for this pilot is `none`. Repeated reconcile
-calls must return the same plan for the same snapshot and must not request a
-worker start, stop, resize, wakeup, storage write, readiness signal, peer
-signal, or metrics emission.
-
-## BGC-005 Allocator Reclaim Status And Controller Surface
-
-The second low-risk controller/status surface is allocator reclaim. It reports
-the service name, desired enablement, configured force flag, backend-specific
-effective force, idle interval settings, runtime-token cancellation state, and
-the absence of a dedicated shutdown handle.
-
-The only allowed worker mutation for this surface is `none`. Reconcile output is
-read-only and must not start, stop, resize, wake, or otherwise drive the
-allocator reclaim loop. Existing backend-specific force handling, idle-streak
-logic, metrics emission, and runtime-token shutdown behavior remain owned by the
-current loop.
-
-## BGC-006 Metrics Runtime Status And Controller Surface
-
-The third low-risk controller/status surface is metrics runtime. It reports the
-service name, observability metrics enablement, collector task count, configured
-collector intervals, replication bandwidth zero-tombstone cycle count,
-runtime-token cancellation state, and the absence of a dedicated shutdown
-handle.
-
-The only allowed worker mutation for this surface is `none`. Reconcile output is
-read-only and must not start, stop, resize, wake, or otherwise drive metrics
-collector tasks. Existing collector grouping, interval parsing, metrics
-emission, replication bandwidth tombstone handling, and runtime-token shutdown
-behavior remain owned by the current loops.
-
-## Future Reconcile Rules
-
-Future reconcile work is allowed only after a read-only status snapshot exists.
-The first reconcile pilot must:
-
-- choose one low-risk service;
-- compare desired/current/status without side effects;
-- prove idempotence under repeated calls;
-- prove no duplicate workers are created;
-- preserve existing shutdown order and cancellation source;
-- include rollback guidance that removes the pilot without changing existing
-  worker behavior.
-
-Memory observability is the recommended first candidate because it already has a
-simple runtime cancellation loop and no storage writes. Scanner, heal,
-replication, lifecycle, disk health, deferred IAM recovery, and auto-tuning must
-wait for focused preservation tests.
-
-## Verification Expectations
-
-For this docs-only contract:
-
-- architecture migration guard scripts must pass;
-- layer dependency and metrics reference guards must pass;
-- no Rust source, Cargo metadata, CI workflow, Makefile, or runtime config file
-  may change.
-
-For the next implementation PRs:
-
-- add focused tests before changing behavior;
-- do not modify production logic only to make tests pass;
-- keep compatibility comments searchable with `RUSTFS_COMPAT_TODO(<task-id>)`
-  whenever temporary old paths are retained for later deletion.
+- Scanner implies heal: the loop started by `init_data_scanner` (`rustfs/src/startup_lifecycle.rs`) enqueues heal work, so scanner status must separate scheduler state from work-source accounting.
+- Heal/AHM owns its own token: `create_ahm_services_cancel_token` and `init_heal_manager` run in `rustfs/src/startup_background.rs`; `shutdown_ahm_services` runs in `rustfs/src/startup_shutdown.rs`. Heal admission and channel-close semantics stay intact.
+- Replication has two shutdown contracts: the pool started by `init_background_replication` (`rustfs/src/startup_storage.rs`) stops workers by closing channels, while resync started by `init_resync` (`rustfs/src/startup_bucket_metadata.rs`) uses cancellation tokens, and admin-triggered resync uses per-bucket tokens.
+- Lifecycle expiry, transition, and stale-multipart cleanup are started by `ECStore::init` (`init_background_expiry`, `init_background_stale_multipart_upload_cleanup` in `crates/ecstore/src/store/init.rs`), which binds the runtime token through `bind_background_cancel_token`; the scanner is their event source, so they are not a separate periodic controller.
+- Notification and audit share a runtime pattern but not a lifecycle: `init_event_notifier` and `start_audit_system` (`rustfs/src/startup_audit.rs`), `shutdown_event_notifier` and `stop_audit_system` (`rustfs/src/startup_shutdown.rs`). Live event streams stay separate from target-delivery enablement.
+- Dynamic config reload is admin-triggered fanout (`apply_dynamic_config_for_subsystem`, `signal_dynamic_config_reload`, `signal_config_snapshot_reload` in `rustfs/src/admin/service/config.rs`), not a loop; per-subsystem validation and error boundaries are preserved.
+- Capacity refresh tasks are owned through `CapacityBackgroundTasks` returned by `init_capacity_management_managed` (`rustfs/src/capacity/capacity_integration.rs`, called from `rustfs/src/startup_entrypoint.rs`); scheduled interval defaults and singleflight refresh stay unchanged.
+- Storage-adjacent monitors (`monitor_and_connect_endpoints` in `crates/ecstore/src/core/sets.rs`, `enable_health_check` in `crates/ecstore/src/disk/disk_store.rs`) change disk state and stay outside controller work.
+- Deferred IAM recovery (`spawn_iam_recovery_task`, `rustfs/src/startup_iam.rs`) publishes readiness; optional protocol servers already own `ShutdownHandle`s; the auto-tuner (`init_auto_tuner` in `rustfs/src/init.rs`) changes runtime concurrency. All three stay outside generic controllers.

docs/architecture/background-services-inventory.md

@@ -1,89 +0,0 @@
-# Background Services Inventory
-
-This document records the current background service surface before
-BackgroundController work. It is a behavior-preservation inventory only; it does
-not define a new scheduler, controller framework, or shutdown contract.
-
-## Scope
-
-- Related migration task: `BGC-001`.
-- PR type: `docs-only`.
-- Baseline: `upstream/main` at
-  `03eb10b07f5f968c531151ae667dfe218050493d`.
-- Out of scope: changing startup order, shutdown order, readiness, storage
-  writes, heal admission, scanner scheduling, replication queues, config reload
-  behavior, metrics intervals, or worker counts.
-
-## Startup And Shutdown Owners
-
-| Area | Startup owner | Shutdown owner | Current cancellation source |
-|---|---|---|---|
-| Main runtime token | `rustfs/src/main.rs::run` creates `ctx` after HTTP listeners start and before ECStore creation. | `rustfs/src/main.rs::handle_shutdown` calls `ctx.cancel()` before service-specific shutdown. | Shared `tokio_util::sync::CancellationToken`. |
-| Scanner | `rustfs/src/main.rs::run` calls `init_data_scanner(ctx.clone(), store.clone())` after successful startup log and global init time. | Main shutdown calls `ctx.cancel()`; if scanner was enabled it also calls `shutdown_background_services()`. | Scanner loop receives the main runtime token. |
-| Heal/AHM | Main creates `create_ahm_services_cancel_token()` before scanner/heal feature checks and calls `init_heal_manager(...)` when heal or scanner is enabled. | Main shutdown calls `shutdown_ahm_services()` when heal or scanner was enabled. | Global AHM token plus channel/worker-local state. |
-| Replication pool | Main calls `init_background_replication(store.clone())` after global config init, then `pool.init_resync(ctx.clone(), buckets.clone())` after bucket listing. | No direct main shutdown call for the replication pool; resync receives the main runtime token. | Resync routine uses the main runtime token; per-bucket resync uses registered cancel tokens. |
-| Lifecycle expiry/transition | `ECStore::init` calls `init_background_expiry(self.clone())` and `init_background_stale_multipart_upload_cleanup(self.clone())`. | Expiry workers read `get_background_services_cancel_token()` and fall back to a private token if none exists. Stale multipart cleanup exits when the weak ECStore reference cannot upgrade. | `ECStore::init` binds the main runtime token into the instance context with `bind_background_cancel_token(ctx)` before expiry starts, so the private-token fallback is a defensive path rather than the normal one. |
-| Notification runtime | Main calls `init_event_notifier()` after buffer profile init. | Main shutdown calls `shutdown_event_notifier().await`. | Notification runtime owns target/replay shutdown internally. |
-| Audit runtime | Main calls `start_audit_system().await`. | Main shutdown calls `stop_audit_system().await`. | Audit runtime owns target/replay shutdown internally. |
-| Metrics and memory loops | Main calls `init_metrics_runtime(ctx.clone())`, `init_memory_observability(ctx.clone())`, and `init_auto_tuner(ctx.clone())` when observability metrics are enabled. | Main shutdown only cancels the shared runtime token. | Shared runtime token. |
-| Allocator reclaim | Main calls `init_allocator_reclaim(ctx.clone())` unconditionally. | Main shutdown cancels the shared runtime token. | Shared runtime token. |
-| Capacity manager | Main calls `init_capacity_management().await` before HTTP listener startup and ECStore creation. | No direct main shutdown call. | Current scheduled capacity and metrics loops do not receive a shutdown token. |
-| Optional protocol servers | Main calls feature-gated FTP, FTPS, WebDAV, and SFTP init functions. | Main shutdown calls each stored `ShutdownHandle` and waits for all protocol shutdown futures. | Per-protocol broadcast shutdown handles. |
-| Deferred IAM recovery | `bootstrap_or_defer_iam_init(...)` may spawn a deferred recovery loop. | Main shutdown cancels the shared runtime token. | Shared runtime token. |
-
-## Service Inventory
-
-| Service | Trigger and workers | Side effects | Status and metrics | Migration notes |
-|---|---|---|---|---|
-| Capacity background refresh | `rustfs/src/capacity/capacity_integration.rs::init_capacity_management` delegates to `init_capacity_management_for_local_disks`, then `crates/object-capacity/src/capacity_manager.rs::start_background_task` spawns a scheduled refresh loop and a runtime summary loop. | Refreshes global capacity cache from local disks and logs runtime summaries. | Uses the object-capacity manager state and log summaries; no explicit shutdown status surface is exposed here. | Add read-only status before any controller migration. A future controller must not change scheduled interval defaults or singleflight refresh behavior. |
-| ECStore endpoint monitor | `crates/ecstore/src/core/sets.rs::new` spawns `monitor_and_connect_endpoints`. | Monitors endpoint connectivity and reconnect behavior for erasure sets. | Logs monitor start, cancellation, and exit. | This is storage-adjacent and must stay outside broad controller movement until storage shutdown semantics are explicitly covered. |
-| Local disk health monitor | `crates/ecstore/src/store/init.rs::init` enables disk health checks after store initialization; `crates/ecstore/src/disk/disk_store.rs::enable_health_check` spawns writable and recovery monitors. | Periodically probes disk writability, can create test objects named `health-check-*`, and updates disk runtime health state. | Disk info includes runtime health metrics and waiting counts. | Do not merge this with scanner/heal controller work; probes affect disk health semantics. |
-| Data scanner | `crates/scanner/src/scanner.rs::init_data_scanner` configures scanner defaults, applies runtime config, waits the initial scanner delay, then loops `run_data_scanner`. | Updates data usage cache, scans buckets/sets, evaluates lifecycle rules, queues replication heal, queues scanner heal, and emits scanner alerts. | Scanner runtime config/status is exposed through admin scanner status; scanner metrics record ILM, replication admission, heal admission, checkpoints, yields, and alerts. | Scanner implies heal because scanner can enqueue heal requests. Future controller status must separate scheduler state from scanner work-source accounting. |
-| Heal/AHM | `crates/heal/src/lib.rs::init_heal_manager` starts `HealManager`, initializes the shared heal channel, and spawns `HealChannelProcessor`. | Consumes heal requests from the global heal channel and drives heal work through the configured heal storage API. | Global active-task and queue-length atomics track current heal pressure. | Keep heal admission and channel semantics intact. Controller work should first expose queue/active status and shutdown state. |
-| Bucket replication pool | `crates/ecstore/src/bucket/replication/replication_pool.rs::init_background_replication` creates global replication stats and the global pool; pool resizing spawns regular, large-object, and failed-object workers. | Replicates object and delete operations, updates queue stats, and maintains replication worker pools. | Replication stats expose active worker counts and queue accounting. | Worker resize behavior currently closes channels to stop workers. Do not replace this with a generic controller until queue close semantics are captured by tests. |
-| Bucket replication resync | Main calls `get_global_replication_pool().init_resync(ctx.clone(), buckets.clone())`; the pool spawns `start_resync_routine`. Admin site-replication handlers can start or cancel per-bucket resync with dedicated tokens. | Loads persisted resync state, starts bucket resync, persists status, and can cancel per-target resync. | Admin site-replication status surfaces resync state. | Preserve the split between startup resync and admin-triggered resync operations. |
-| Lifecycle expiry and transition | `ECStore::init` calls `init_background_expiry(self.clone())`. Scanner evaluates lifecycle events and queues expiry/transition work through `apply_expiry_rule` and `apply_transition_rule`. | Deletes expired objects, queues transitions, updates lifecycle stats, and accounts scanner ILM actions only when work is queued. | Lifecycle state tracks worker counts, active tasks, queue send timeouts, compensation tasks, and transition stats. | This is not a separate periodic controller today; scanner is the main event source for object lifecycle evaluation. |
-| Stale multipart cleanup | `ECStore::init` calls `init_background_stale_multipart_upload_cleanup(self.clone())`. | Periodically deletes stale multipart upload data. | Logs cleanup passes when objects are deleted. | Current loop has no explicit cancellation token and exits when ECStore is dropped. Future controller work needs an explicit lifecycle decision before changing it. |
-| Notification runtime | `rustfs/src/server/event.rs::init_event_notifier` initializes live event stream support even when notification targets are disabled; when enabled, it loads server config and activates targets. Config reload uses `NotificationConfigManager::reload_config`. | Installs ECStore event dispatch hook, activates notification targets, manages replay/runtime target state, and supports live event streams. | Notification module state is refreshed from persisted module switches; target health is available through runtime target status. | Keep live event stream support separate from target delivery enablement. Reload must remain admin-triggered and peer-signaled. |
-| Audit runtime | `rustfs/src/server/audit.rs::start_audit_system` starts audit only when module switches and configured targets allow it. `AuditSystem::reload_config` replaces runtime targets. | Dispatches audit events to configured targets and manages replay workers. | Audit observability records config reloads and target delivery metrics. | Do not couple audit lifecycle to notification lifecycle even though the runtime patterns are similar. |
-| Dynamic config reload | Admin config handlers call `apply_dynamic_config_for_subsystem`, then `signal_dynamic_config_reload` or `signal_config_snapshot_reload` through the global notification system. | Applies scanner/heal runtime config, audit reloads, notification reloads, and peer reload signals. | Logs local and peer reload failures. Audit reload increments audit config reload metrics. | This is admin-triggered fanout, not a background scheduler. Controller work should preserve per-subsystem validation and error boundaries. |
-| Metrics runtime | `crates/obs/src/metrics/scheduler.rs::init_metrics_runtime` spawns multiple interval loops for cluster, bucket, node, resource, audit, notification, and replication bandwidth metrics. | Periodically collects and reports metrics. | Reports through the metrics runtime, logs cancellation warnings, and exposes a typed read-only status snapshot plus a no-op reconcile plan for enablement, collector task count, intervals, replication bandwidth tombstone cycles, cancellation source, and shutdown handle shape. | Keep intervals and collector grouping stable. The current controller surface does not mutate workers. |
-| Memory observability | `rustfs/src/memory_observability.rs::init_memory_observability` spawns a token-cancelled sampler. | Periodically records memory snapshots. | Emits memory observability metrics and exposes a read-only status snapshot plus a no-op reconcile plan for metrics enablement, interval, cancellation source, and shutdown handle shape. | This is the first low-risk pilot for controller status because it already has a simple token loop and the pilot does not mutate workers. |
-| Allocator reclaim | `rustfs/src/allocator_reclaim.rs::init_allocator_reclaim` spawns a token-cancelled reclaim loop when enabled. | Observes reclaimable work and may run allocator reclaim after idle intervals. | Emits reclaim enabled/backend counters, active-request gauges, scanner/heal activity gauges, and reclaim result counters. Exposes a typed read-only status snapshot plus a no-op reconcile plan for enablement, backend, effective force, intervals, cancellation source, and shutdown handle shape. | A controller must preserve idle-streak logic and backend-specific force behavior. The current controller surface does not mutate workers. |
-| Auto-tuner | `rustfs/src/init.rs::init_auto_tuner` optionally spawns a 60-second loop when `RUSTFS_AUTOTUNER_ENABLED` is true. | Tunes concurrency manager settings from performance metrics. | Logs iteration success/failure. | Treat as behavior-sensitive; a future controller needs explicit rollback because it can change runtime concurrency. |
-| Update check | `rustfs/src/init.rs::init_update_check` spawns one async task with a 30-second timeout when update checks are enabled. | Performs version check network I/O and logs available updates. | Logs result only. | This is a one-shot task, not a controller candidate for the first BGC PRs. |
-| Deferred IAM recovery | `rustfs/src/startup_iam.rs::spawn_iam_recovery_task` retries IAM init with backoff and finalizes readiness when successful. | Can initialize IAM later, initialize AppContext if needed, mark `IamReady`, and publish `FullReady`. | Readiness state reflects deferred recovery progress. | Keep this lifecycle-critical path separate from generic background controllers. |
-| Optional protocol servers | `rustfs/src/init.rs` starts FTP, FTPS, WebDAV, and SFTP with per-protocol `ShutdownHandle`s when features and config enable them. | Serve protocol traffic in background tasks. | Shutdown logs per protocol. | Protocol servers already have explicit handles; do not fold them into BGC until the service registry owns shutdown ordering. |
-
-## Current Gaps To Preserve Before Controller Work
-
-- ECStore background-service cancellation has a public global token API, but this
-  inventory found no current startup call to create that token. Lifecycle expiry
-  workers therefore use their fallback token when no global token exists.
-- Capacity manager loops do not receive the main runtime cancellation token.
-- Replication worker pools stop some workers by closing channels, while resync
-  uses cancellation tokens. These are different shutdown contracts.
-- Scanner, lifecycle, replication, and heal are coupled by work queues and
-  metrics. Moving one without status snapshots for the others risks hiding work
-  admission failures.
-- Dynamic config reload is admin-triggered and peer-signaled, not a periodic
-  background loop.
-
-## BGC-002 Contract Inputs
-
-These inputs are formalized in
-[`background-controller-contract.md`](background-controller-contract.md).
-
-Future controller contract work should start with a read-only shape:
-
-- `desired`: enabled/disabled plus static config source.
-- `current`: started, stopped, running, degraded, or disabled.
-- `status`: worker counts, queue lengths, last cycle/reload time, and last error.
-- `shutdown`: cancellation source and whether the service has an explicit stop
-  handle.
-- `side_effects`: storage writes, target activation, external I/O, metrics, and
-  readiness changes.
-
-The first pilot should use a service with an existing simple cancellation loop
-and no storage writes, such as memory observability. Scanner, heal, replication,
-lifecycle, and disk health must wait for focused preservation tests.

docs/architecture/compat-cleanup-register.md

@@ -1,8 +1,7 @@
 # Compatibility Cleanup Register
 
-Use this file to track temporary compatibility code introduced by architecture
-migration PRs. Entries are required only for compatibility paths that are planned
-for later deletion.
+**Use this when:** you add, review, or remove a temporary compatibility path (fallback, wrapper, re-export, legacy codec) and need the required marker and its removal condition.
+**Source of truth:** the `RUSTFS_COMPAT_TODO(<id>)` source markers, matched in both directions against `## Open Items` by `scripts/check_architecture_migration_rules.sh`. Entries exist only for compatibility paths planned for later deletion.
 
 ## Required Source Marker
 
@@ -14,7 +13,7 @@ for later deletion.
 
 - `tokio-tar-extension-limits` bounded archive parser hardening: Snowball extraction depends on per-entry and cumulative GNU long-name, GNU long-link, and PAX extension limits; physical-entry, GNU sparse-map, and sparse-continuation limits; cancellation-safe sparse parsing; and fused entry streams after parser errors. The released tokio-tar API does not provide this complete boundary. Keep the reviewed fork pin until astral-sh/tokio-tar#118 is merged and one published tokio-tar release contains every listed capability with the Snowball regression fixtures passing against that release.
 - `backlog-2102` rc.2/rc.3 empty scanner usage floor recovery: old DeleteBucket cleanup could synthesize an empty incomplete v2 usage primary/backup before leadership added an epoch, while newer scanners require a durable authoritative baseline identity. New scanners recognize only that exact serialized empty-fence shape, preserve its epoch through a CAS-protected recovery marker, and rebuild namespace coverage without treating zero usage as authoritative. Remove this recovery path and marker after rc.2 and rc.3 are no longer supported direct-upgrade sources.
-- `s3gate-metadata-xml` persisted bucket XML migration: mixed-version site-replication peers, retained `.metadata.bin` objects, and backup archives can all carry XML written by the s3s codec, so the gateway migration must keep the legacy codec available until every stored form has crossed a verified rewrite boundary. Remove the legacy s3s parser and serializer only after the minimum supported direct-upgrade release reads and writes every persisted XML configuration family through the gateway codec, the four-way D1-D5 gate has remained clean for one full support window, every supported mixed-version site-replication topology has completed its writer upgrade, and migration tooling has verified or rewritten every retained bucket metadata object and restorable backup archive.
+- `s3gate-metadata-xml` persisted bucket XML migration: mixed-version site-replication peers, retained `.metadata.bin` objects, and backup archives can all carry XML written by the s3s codec, so the gateway migration must keep the legacy codec available until every stored form has crossed a verified rewrite boundary. Remove the legacy s3s parser and serializer only after the minimum supported direct-upgrade release reads and writes every persisted XML configuration family through the gateway codec, every supported mixed-version site-replication topology has completed its writer upgrade, and migration tooling has verified or rewritten every retained bucket metadata object and restorable backup archive.
 - `rustfs-6339` legacy bucket policy ID casing: earlier RustFS releases persisted the top-level policy identifier as "ID", while current writes use the S3-compatible "Id" spelling. Readers accept both spellings so retained bucket metadata remains usable after upgrade. Remove the legacy alias after migration tooling has rewritten every retained bucket policy using "ID".
 - `table-publication-fence-v1` table publication fencing: nodes that predate table and table-bucket publication fences can mutate live files while a new node is publishing a catalog pointer. New nodes retain exact object guards until the operator confirms that every serving node uses the new fences. Fleet confirmation also requires non-overlapping active warehouse prefixes and lifecycle workers that exclude table buckets. Remove the exact live-file fallback and the fleet-confirmation gate after the minimum supported RustFS release acquires table fences for registered-table mutations and table-bucket fences for unresolved-prefix mutations.
 - `table-catalog-strong-snapshot-v1` durable strong catalog snapshot compatibility: version 1 writes continue during mixed-version rollout until operators confirm that every serving node reads version 2, and version 1 table/view identifier collisions remain available only for cleanup. Remove version 1 writes and collision cleanup after the minimum supported RustFS release reads version 2 and every retained durable strong snapshot is collision-free and has been upgraded to version 2.
@@ -29,7 +28,7 @@ for later deletion.
 - `rustfs-5416-zero-retry-delay` startup retry-delay validation: releases before bounded topology convergence accept RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY values of 0 or 0ms. New servers replace those values with the safe nonzero default so a direct upgrade neither fails startup nor enters a busy loop. Reject zero after the minimum supported direct-upgrade release validates or rewrites this setting before rollout.
 - `scanner-usage-v2` persisted scanner usage migration: pre-v2 scanners write `.usage.json`, so upgraded clusters read that primary/backup pair only while `.usage.v2.json` is absent and continue removing deleted buckets from legacy copies that still exist. The additive usage_snapshot_complete field in `.usage.v2.json` must remain optional while mixed-version clusters are supported; a missing field means the snapshot is not authoritative. The legacy read also feeds the degraded quota-admission baseline (issue #5716): while no authoritative usage exists, quota checks admit against the pre-discard sizes of the last loaded snapshot, including a legacy one. Remove the legacy object fallback and cleanup only after every supported direct-upgrade source writes `.usage.v2.json`; the baseline then feeds from incomplete v2 snapshots alone.
 - `ns-scanner-rpc-v3` namespace scanner capability and activity handshake: old peers and legacy internode transports lack the authenticated startup-epoch handshake. The oldest peers send an empty activity request and receive a field-empty protocol-0 response. Protocol v4 binds the challenge and response topology but cannot authenticate distributed dirty-usage state. Protocol v5 binds the request version, acknowledgement target and generation, and the response dirty-usage state, but predates set-scoped scanner cache locks. Protocol v6 additionally fences scanner cache lock-domain changes. Current protocol v7 binds the storage-owned movement generation and publication-blocked state, so distributed scanner cycles publish usage only after every peer reports a complete v7 activity proof; v6 responses remain readable but are treated as unverified for publication. Servers retain protocol-0, protocol-v4, and protocol-v6 codecs alongside the current v7 codec for rolling upgrades, while protocol-v5 peers are treated as previous-version peers that cannot safely participate in the new cache lock domain. Scanner selection treats HTTP 404/405/426 and the legacy MethodNotAllowed default as an explicit lack of remote scanner v3 support and assigns those disks to coordinator-driven workers; transient capability failures remain incomplete and do not activate the fallback. Remove the coordinator fallback after the minimum supported RustFS peer version implements namespace scanner protocol v3, remove protocol-0 activity requests and responses after every supported peer implements authenticated scanner activity protocol v4, remove the protocol-v4 activity codec after every supported peer implements protocol v5, and remove protocol-v5 previous-version rejection after every supported peer implements protocol v6; future protocol revisions must keep the same dual-version server/codec window before changing the advertised version.
-- `#4648` walk-dir stream completion capability: old clients can append fallback output to an already-used metacache writer after a terminal body error, so servers emit terminal walk errors only to clients that sign the `walk_dir_stream_completion=error-v1` query capability and its request-body digest. Remove the legacy clean-EOF path after the minimum supported RustFS peer version always advertises this capability.
+- `rustfs-4648` walk-dir stream completion capability: old clients can append fallback output to an already-used metacache writer after a terminal body error, so servers emit terminal walk errors only to clients that sign the `walk_dir_stream_completion=error-v1` query capability and its request-body digest. Remove the legacy clean-EOF path after the minimum supported RustFS peer version always advertises this capability.
 - `heal-rpc-auth-v2` internode gRPC authentication: servers temporarily accept legacy prefix signatures so old peers remain available during rolling upgrades. Remove the legacy fallback after the minimum supported RustFS peer version sends v2 authentication on every internode gRPC request.
 - `put-file-auth-epoch-strict` internode put_file epoch compatibility: rc.2 peers can cache a remote put_file capability before that remote node restarts, then continue sending v1 authenticated uploads with the old server epoch; those peers cannot recover from the 409 conflict used by newer clients to trigger a re-probe. Servers temporarily accept signed, non-nil stale put_file epochs while legacy put_file auth remains non-strict so mixed-version rolling upgrades can finish multipart/object writes. Remove the stale-epoch fallback after the minimum supported RustFS peer version re-probes put_file capability after server-epoch conflicts and legacy put_file auth is no longer accepted.
 - `disk-mutation-body-digest` internode mutating disk RPCs: servers temporarily accept mutating disk RPCs (RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete, DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes) that carry no signature-bound canonical body digest, so peers from releases that predate body-digest signing remain available during rolling upgrades. Accepted digestless mutations increment the internode body-digest fallback counter; that counter must read zero fleet-wide across a release window before RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT is enabled. Because body-bound requests now consume replay-cache nonces on the receiver, deploy the raised RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY default fleet-wide before enabling strict mode, and watch the internode replay-cache overflow counter for undersized capacity during the rollout. Remove the digestless fallback after the minimum supported RustFS peer version body-binds every mutating disk RPC.

docs/architecture/config-model-boundary-adr.md

@@ -1,185 +1,49 @@
 # Config Model Boundary ADR
 
-Related issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
-
-Task: `CFG-002`
+**Use this when:** you touch the server-config model (`Config`, `KV`, `KVS`) or its persistence, or you need to know which crate owns which part of server configuration.
+**Source of truth:** `crates/config/src/server_config.rs` (model, default registration, process-global snapshot) and `crates/ecstore/src/config/` (`ConfigSys`, persistence, migration, storage-class runtime state).
 
 ## Decision
 
-Use the existing `crates/config` package (`rustfs-config`) as the target owner
-for the pure server-config model. Do not create a new config-model crate for
-the first extraction.
+`rustfs-config` (`crates/config`) owns the pure server-config model and the process-global server-config snapshot. ECStore keeps config persistence, migration, default-registration wiring, startup initialization, and storage-class runtime state. There is no separate config-model crate, and `rustfs_ecstore::config` does not re-export the model or the snapshot accessors.
 
-The next model extraction PR should introduce the model under:
-
-```text
-crates/config/src/server_config.rs
-```
-
-The exported path should be:
-
-```rust
-rustfs_config::server_config::{Config, KV, KVS}
-```
-
-The extraction kept the existing path available through a temporary
-compatibility re-export:
-
-```rust
-rustfs_ecstore::config::{Config, KV, KVS}
-```
-
-That re-export included `RUSTFS_COMPAT_TODO(CFG-004)` and a matching entry in
-[`compat-cleanup-register.md`](compat-cleanup-register.md) until the model
-consumers were migrated. The CFG-004 cleanup removed this old model path after
-code scans showed consumers import the model directly from `rustfs-config`.
-
-Follow-up `CFG-008` moved the process-global server-config snapshot accessors
-to `rustfs_config::server_config` after the model path stabilized. Its temporary
-`rustfs_ecstore::config::{get_global_server_config, set_global_server_config}`
-compatibility re-export was removed after in-repo runtime consumers migrated to
-the `rustfs-config` owner.
+Import path: `rustfs_config::server_config::{Config, KV, KVS}`. The model sits behind the `server-config-model` feature of `rustfs-config` (`crates/config/Cargo.toml`), which enables `serde` and `serde_json`.
 
 ## Why `rustfs-config`
 
-`rustfs-config` is already the lowest RustFS crate for configuration constants
-and subsystem identifiers used by ECStore, notify, audit, targets, scanner, IAM,
-and admin code. The current `ecstore::config::{Config, KV, KVS}` model already
-uses `rustfs-config` constants, so moving the pure model upward to
-`rustfs-config` cuts the wrong dependency direction without adding another crate.
+- It is already the lowest RustFS crate for configuration constants and subsystem identifiers used by ECStore, notify, audit, targets, scanner, IAM, and admin code, and the model needs only those constants.
+- Moving the model upward removes the wrong-direction dependency (outer crates importing ECStore for a plain data type) without adding another crate or a second config namespace.
 
-Creating a new crate now would add a second config namespace before consumers
-are migrated. That would increase re-export and compatibility surface while not
-removing any storage or runtime dependency by itself.
+## Ownership
 
-## Allowed Dependencies
+| Item | Owner | Notes |
+|---|---|---|
+| `KV`, `KVS`, `Config` and their methods (`get_value`, `set_defaults`, `marshal`, `unmarshal`, `merge`) | `crates/config/src/server_config.rs` | Pure data model with serde roundtrip |
+| `DEFAULT_KVS`, `register_default_kvs` | `crates/config/src/server_config.rs` | Registration surface; ECStore still calls it from `init()` in `crates/ecstore/src/config/mod.rs` |
+| `GLOBAL_SERVER_CONFIG`, `get_global_server_config`, `set_global_server_config` | `crates/config/src/server_config.rs` | Process-global snapshot accessors |
+| `ConfigSys`, `init()`, `try_migrate_server_config` | `crates/ecstore/src/config/mod.rs` | Startup order and caller unchanged |
+| `read_config_without_migrate`, `save_server_config`, other config-object helpers | `crates/ecstore/src/config/com.rs` | Persistence over the object store |
+| `GLOBAL_STORAGE_CLASS` and storage-class parsing | `crates/ecstore/src/config/mod.rs`, `crates/ecstore/src/config/storageclass.rs` | Storage behavior stays in ECStore |
 
-The server-config model module may use only:
+## Allowed Dependencies Of The Model Module
 
-- `std::collections::HashMap`
-- `std::sync::{LazyLock, OnceLock, RwLock}` for the default `KVS` registration
-  surface and process-global server-config snapshot
-- `serde` for `KV` and `KVS` serialization compatibility
-- `serde_json` for `Config::marshal` and `Config::unmarshal`
-- existing `rustfs-config` constants and subsystem modules
+- `std::collections::HashMap` and `std::sync::{LazyLock, OnceLock, RwLock}` for `DEFAULT_KVS` and `GLOBAL_SERVER_CONFIG`;
+- `serde` for `KV`/`KVS` and `serde_json` for `Config::marshal` / `Config::unmarshal`, gated by `server-config-model`;
+- existing `rustfs-config` constants and subsystem modules.
 
-If `serde` and `serde_json` are added to `rustfs-config`, they should be attached
-only to a model feature such as `server-config-model` unless the implementation
-PR proves that making them non-optional is simpler and harmless for downstream
-builds.
+## Forbidden Dependencies Of The Model Module
 
-## Forbidden Dependencies
+- `rustfs-ecstore`, `rustfs`, storage-api traits, or object persistence helpers;
+- notify, audit, targets, IAM, scanner, KMS, or admin handler crates;
+- async runtimes, HTTP/router crates, object-store crates, or runtime lifecycle state;
+- `ConfigSys`, `read_config_without_migrate`, `save_server_config`, or any `com.rs` helper.
 
-The model module must not depend on:
+## Shape Preservation
 
-- `rustfs-ecstore`
-- `rustfs`
-- `StorageAPI` or object persistence helpers
-- notify, audit, targets, IAM, scanner, KMS, or admin handler crates
-- async runtimes, HTTP/router crates, object-store crates, or runtime lifecycle
-  state
-- unrelated runtime global state outside the process-global server-config
-  snapshot
-- `ConfigSys`, `read_config_without_migrate`, `save_server_config`, or any
-  `com.rs` persistence helper
+Persisted server-config JSON must keep decoding unchanged:
 
-## Boundary Split
-
-Move in the first extraction:
-
-- `KV`
-- `KVS`
-- `Config`
-- `DEFAULT_KVS`
-- `register_default_kvs`
-- `Config::new`
-- `Config::get_value`
-- `Config::set_defaults`
-- `Config::marshal`
-- `Config::unmarshal`
-- `Config::merge`
-
-Keep in `ecstore`:
-
-- `ConfigSys`
-- `init_global_config_sys`
-- `try_migrate_server_config`
-- `read_config_without_migrate`
-- `save_server_config`
-- generic `com.rs` config-object helpers
-- storage-class runtime global state
-
-Keep default registration wiring in `ecstore::config::init` until a later PR
-extracts a dedicated default-registration contract. The values may be registered
-through the moved `rustfs_config::server_config::register_default_kvs`, but the
-startup order and caller remain unchanged.
-
-Move in `CFG-008`:
-
-- `GLOBAL_SERVER_CONFIG`
-- `get_global_server_config`
-- `set_global_server_config`
-
-The temporary ECStore compatibility re-export for these accessors was removed
-after code scans showed in-repo consumers use `rustfs_config::server_config`
-directly.
-
-## Required Shape Preservation
-
-The extraction PR must preserve:
-
-- `KV { key, value, hidden_if_empty }`
-- `#[serde(default, alias = "hiddenIfEmpty")]` on `KV::hidden_if_empty`
-- `KVS(pub Vec<KV>)`
-- `Config(pub HashMap<String, HashMap<String, KVS>>)`
-- `KVS::new`, `get`, `lookup`, `is_empty`, `keys`, `insert`, and `extend`
-- `Config::new`, `get_value`, `set_defaults`, `marshal`, `unmarshal`, and
-  `merge`
-- `Config::new()` default application after `ecstore::config::init()`
-- existing persisted server-config JSON shape
-- existing target, notify, audit, scanner, OIDC, and admin interpretation of
-  `Config` and `KVS`
-
-## Next PR Requirements
-
-`CFG-003` should be a pure model extraction or narrow `api-extraction` PR. It
-must not migrate consumers, change persistence helpers, or alter runtime
-behavior.
-
-`CFG-004` kept the old `rustfs_ecstore::config::*` path as a temporary
-compatibility shim, registered its removal condition, and removed the shim after
-all in-repo consumers migrated.
-
-`CFG-005` should migrate external consumers one group at a time after the model
-and compatibility path are stable.
-
-`CFG-008` moves only the global server-config snapshot accessors to
-`rustfs-config` and migrates in-repo direct consumers. It must not move
-`ConfigSys`, storage-class global state, persistence helpers, default
-registration wiring, startup order, or storage behavior.
-
-## Verification Gate
-
-Before pushing an extraction PR, run:
-
-- serde roundtrip tests for old and new paths
-- tests for `hiddenIfEmpty` alias compatibility
-- tests for `KVS` insertion, lookup, extension, and keys behavior
-- tests for `Config::new`, `set_defaults`, `marshal`, `unmarshal`, and `merge`
-- a cleanup scan proving in-repo consumers no longer use the old
-  `rustfs_ecstore::config::{Config, KV, KVS}` model path before removing the
-  compatibility shim
-- `cargo tree -p rustfs-config --edges normal`
-- `cargo tree -p rustfs-ecstore --edges normal`
-- `./scripts/check_layer_dependencies.sh`
-- `./scripts/check_architecture_migration_rules.sh`
-- `cargo fmt --all --check`
-- `make pre-commit`
-
-## Non-Goals
-
-- No consumer migration in `CFG-002`.
-- No code movement in `CFG-002`.
-- No new crate in `CFG-002`.
-- No `com.rs` or `StorageAPI` movement in the first model extraction.
-- No global server-config state migration until the model path is stable.
+- `KV { key, value, hidden_if_empty }` with `#[serde(default, alias = "hiddenIfEmpty")]` on `hidden_if_empty`;
+- `KVS(pub Vec<KV>)` and `Config(pub HashMap<String, HashMap<String, KVS>>)`;
+- `KVS::{get, lookup, is_empty, keys, insert, extend}` and `Config::{get_value, set_defaults, marshal, unmarshal, merge}` keep their semantics;
+- `Config::new()` applies the defaults registered by `ecstore::config::init()`;
+- target, notify, audit, scanner, OIDC, and admin code keep interpreting `Config` and `KVS` the same way.

docs/architecture/crate-boundaries.md

@@ -1,7 +1,7 @@
 # Crate Boundaries And Migration Guardrails
 
-These rules apply to architecture-migration PRs linked to
-[`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660).
+**Use this when:** you add a crate dependency, move code across crates, touch a `storage_api.rs` boundary file, or need the change-type vocabulary the architecture guard enforces.
+**Source of truth:** `scripts/check_architecture_migration_rules.sh` (the enumerated rules; this file is its boundary document) and `scripts/check_layer_dependencies.sh` (layer and edge checks). Extend those guards instead of adding a parallel system.
 
 ## PR Types
 
@@ -22,275 +22,51 @@ Do not mix directory movement, security tightening, and behavior changes in one
 
 ## Dependency Direction
 
-Contract crates must stay below implementation crates. Initial forbidden edges:
+Contract crates stay below implementation crates. Forbidden edges:
 
-- `storage-api -> ecstore`
-- `security-governance -> rustfs`
-- `extension-schema -> rustfs`
-- `extension-schema -> ecstore`
-
-`rustfs-storage-api` may only expose storage-facing replication status/state
-contracts through `crates/storage-api/src/replication.rs` while the underlying
-wire types still live in `rustfs-filemeta`. This keeps the temporary dependency
-centralized until those wire contracts can move without introducing a
-`rustfs-replication` / `rustfs-storage-api` cycle.
-
-Leaf crates carry exactly one adjudicated allowed edge:
-`io-metrics -> rustfs-s3-ops` (transitively `rustfs-s3-types`). Both are pure
-contract crates — types and enums only, no I/O, no global state, no non-contract
-internal dependencies — so `io-metrics` reuses the `S3Operation` vocabulary
-instead of copying it. `madmin` is no longer counted a leaf: since #6166 it is
-the SigV4-signed admin SDK client and deliberately depends on `rustfs-signer`;
-the guard pins its internal dependency surface to exactly that edge so it cannot
-quietly grow storage-side dependencies. The leaf-crate allowlist in
-`scripts/check_architecture_migration_rules.sh` fails any other `rustfs-*`
-dependency in `config`, `credentials`, `crypto`, `io-metrics`, or `madmin`, in
-either TOML spelling (`rustfs-x = ...` or `rustfs-x.workspace = true`).
-Adjudicated in
-[`rustfs/backlog#1834`](https://github.com/rustfs/backlog/issues/1834); a further
-leaf exception must meet the pure-contract criterion — types and enums only, no
-I/O, no globals, no non-contract internal dependencies — and land its guard
-allowlist entry alongside the dependency.
-
-Dependency direction also applies to compile-time source reads:
-`include_str!`/`include!` of a `.rs` file must not resolve outside the
-including crate's own directory (`scripts/check_layer_dependencies.sh`
-enforces this). A source-text tripwire belongs in the crate that owns the
-asserted file; shared expectations move into a contract surface such as
-`rustfs_protos::compat_manifest` and are asserted by each owning crate.
-
-Existing migration checks live in:
-
-- `scripts/check_layer_dependencies.sh`
-- `scripts/check_architecture_migration_rules.sh`
-
-Extend these guardrails instead of adding a parallel system.
-
-## Required Architecture Documents
-
-The migration guard must keep these baseline documents present and anchored to
-their required sections:
-
-- `docs/architecture/overview.md`: Baseline, Core Principle, Phase Order.
-- `docs/architecture/runtime-lifecycle.md`: Startup And Readiness, Shutdown
-  Lifecycle Boundary, AppContext Foundation.
-- `docs/architecture/storage-control-data-plane.md`: Storage API Contracts,
-  Cluster Control Plane, Background Controllers.
-- `docs/architecture/crate-boundaries.md`: PR Types, Dependency Direction,
-  Required Architecture Documents.
-- `docs/architecture/readiness-matrix.md`: Request Behavior Matrix, Runtime
-  Dependency Matrix, Probe Semantics.
-- `docs/architecture/global-state-crate-split-plan.md`: Remaining Global
-  Owners, Runtime Source Boundaries, Fallback Removal Plan, Crate Split
-  Evaluation.
-
-## Pre-Push Expert Review
-
-Before pushing any PR branch, record three expert reviews in the task notes:
-
-| Expert | Required focus |
+| Edge | Why |
 |---|---|
-| Quality/architecture | Structure, naming, dependency direction, PR type, scope, and over-abstraction risk |
-| Migration preservation | Startup order, readiness, quorum, reader semantics, AppContext/global fallback, notify/audit lifecycle, IAM/KMS boundaries, and compatibility |
-| Testing/verification | Focused tests, regression tests, commands run, missing coverage, and whether tests are forcing business-logic drift |
+| `storage-api -> ecstore` | Storage contracts must not depend on the storage implementation |
+| `security-governance -> rustfs` | Governance contracts stay below the binary crate |
+| `extension-schema -> rustfs` | The extension schema is consumed by the binary, never the reverse |
+| `extension-schema -> ecstore` | The extension schema must not reach storage internals |
 
-Push is allowed only when all three experts return `pass` or
-`pass-with-nonblocking-follow-up`. Any `blocker` prevents push until the issue is
-fixed and the relevant review is repeated.
+- `rustfs-storage-api` exposes storage-facing replication status/state contracts only through `crates/storage-api/src/replication.rs`, so its temporary dependency on `rustfs-filemeta` wire types stays centralized and no `rustfs-replication` / `rustfs-storage-api` cycle appears.
+- Leaf crates (`config`, `credentials`, `crypto`, `io-metrics`, `madmin`) may not depend on other `rustfs-*` crates, in either TOML spelling, except the adjudicated edges pinned in the guard's leaf allowlist: `io-metrics -> rustfs-s3-ops` (pure contract crates sharing the `S3Operation` vocabulary) and `madmin -> rustfs-signer` (the SigV4-signed admin SDK client). A new leaf exception must be a pure contract dependency (types and enums only, no I/O, no globals, no non-contract internal dependencies) and land together with its allowlist entry.
+- Compile-time source reads follow the same direction: `include_str!` / `include!` of a `.rs` file must not resolve outside the including crate (`scripts/check_layer_dependencies.sh`). Shared source-text expectations belong in a contract surface such as `rustfs_protos::compat_manifest` (`crates/protos/src/compat_manifest.rs`) and are asserted by each owning crate.
 
-## Temporary Compatibility Code
+## ECStore Access Boundary
 
-Temporary compatibility code that must be removed later must include a searchable
-source comment and a cleanup-register entry.
+Outer crates reach ECStore only through `rustfs_ecstore::api`, and only from one local boundary file per owner (`storage_api.rs`). Boundary files and facade groups are inventoried in [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md).
 
-Use this source-comment format:
-
-```rust
-// RUSTFS_COMPAT_TODO(API-005): keep old ecstore::store_api path during storage-api migration. Remove after all consumers use rustfs-storage-api.
-```
-
-Rules:
-
-- Add the marker only to temporary compatibility paths, not permanent APIs.
-- Include the task ID in the marker.
-- State why the compatibility path exists and when it can be removed.
-- Use this for temporary re-exports, wrappers, fallbacks, legacy action mappings,
-  and old endpoint compatibility layers.
-- Delete compatibility layers in their own cleanup PR.
-
-## Config Model First
-
-`ecstore::config::{Config, KV, KVS}` should move before extension config adapters
-or config-schema work. First inventory consumers, then decide whether existing
-`crates/config` is enough or whether a smaller model crate is required.
-
-The current decision is recorded in
-[`config-model-boundary-adr.md`](config-model-boundary-adr.md): use the existing
-`rustfs-config` package for the pure server-config model and global
-server-config snapshot accessors, while ECStore keeps config persistence,
-storage-class global state, default wiring, and startup initialization.
-
-The old `rustfs_ecstore::config::{Config, KV, KVS, register_default_kvs,
-get_global_server_config, set_global_server_config}` compatibility path must
-not be restored after the Phase 1a cleanup. Consumers use
-`rustfs_config::server_config` for the moved model and accessors; ECStore public
-facades must not re-export those symbols.
+- Inside a boundary file, raw `rustfs_ecstore::api::...` paths are centralized behind local `ecstore_*` module aliases; code outside the boundary sees local type aliases, constants, traits, or wrapper functions, never the raw facade path.
+- Non-trait ECStore surfaces (metadata, object-lock, lifecycle journal, monitor, notification types) stay behind local aliases; boundary function signatures do not expose raw ECStore facade types once narrowed. Object and error aliases anchor on storage-api associated object types and a local `StorageError`.
+- Outer consumers use `rustfs-storage-api` operation traits (`ObjectIO`, `ObjectOperations`, `ListOperations`, `MultipartOperations`, `HealOperations`, `NamespaceLocking`) and generic list responses (`ListObjectsV2Info`, `ListObjectVersionsInfo`, `ObjectInfoOrErr`) directly; ECStore keeps concrete aliases only for internal implementation and compatibility.
+- Bucket lifecycle, replication, versioning, object-lock, restore-request, disk, RPC peer client, and warm-backend trait methods are reached through owner-local compatibility traits or wrapper functions, not by importing ECStore traits outside the boundary.
+- The old `StorageAPI` aggregate facade must not reappear in production `crates/ecstore/src` or `rustfs/src` code.
+- Facade-covered ECStore root modules (layout, `endpoints`, `disks_layout`, bitrot, erasure, object DTO/reader, event, list, batch processor, `global`) stay crate-private; public access goes through the matching `rustfs_ecstore::api::*` group.
+- Cluster control-plane read models stay owned by the crate-private `cluster` module and are published through `rustfs_ecstore::api::cluster`; pool-state, local-node storage, and peer-health projections are read-only.
+- RustFS startup internals are crate-private: only `startup_entrypoint` is a public startup module of the `rustfs` library (`rustfs/src/lib.rs`), and items inside the other `startup_*` modules use crate visibility.
+- The observability dependency baseline is [obs-ecstore-dependency-inventory.md](obs-ecstore-dependency-inventory.md); observability extraction updates it together with the guard.
 
 ## Loss-Prevention Coverage
 
-Architecture migration checks must keep public contract re-exports and ECStore
-compatibility coverage from silently drifting during cleanup PRs.
+The guard pins specific public re-export lines (its `require_source_line` entries) so contract surfaces cannot silently disappear during cleanup. The canonical lists are the guard script and the owning files, not this page:
 
-Required `rustfs-storage-api` public re-exports:
+- `crates/storage-api/src/lib.rs`: admin, bucket, capability, error, multipart, observability, object, and topology contract re-exports;
+- `crates/concurrency/src/lib.rs`: workload admission contract re-exports;
+- `rustfs/src/lib.rs`: `pub mod startup_entrypoint;`.
 
-- `pub use admin::{DiskSetSelector, StorageAdminApi};`
-- `pub use bucket::{BucketInfo, BucketOperations, BucketOptions, DeleteBucketOptions, MakeBucketOptions, SRBucketDeleteOp};`
-- `pub use capability::{CapabilitySnapshotError, CapabilityState, CapabilityStatus};`
-- `pub use error::{StorageErrorCode, StorageResult};`
-- `pub use multipart::{CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartUploadResult, PartInfo};`
-- `pub use observability::{MemorySamplingState, ObservabilitySnapshot, ObservabilitySnapshotProvider, PlatformSupport, UserspaceProfilingCapability};`
-- `pub use object::{HTTPPreconditions, HTTPRangeError, HTTPRangeSpec, ObjectLockRetentionOptions};`
-- `pub use object::{ExpirationOptions, TransitionedObject};`
-- `pub use object::{HealOperations, MultipartOperations, NamespaceLocking, ObjectIO, ObjectOperations};`
-- `pub use object::{ListObjectVersionsInfo, ListObjectsInfo, ListObjectsV2Info, ListOperations, ObjectInfoOrErr};`
-- `pub use object::{ObjectPreconditionError, ObjectPreconditionPart, ObjectPreconditionState};`
-- `pub use object::{VersionMarker, WalkOptions, WalkVersionsSortOrder};`
-- `pub use topology::{DiskCapabilities, TopologyCapabilities, TopologyDisk, TopologyLabels, TopologyPool, TopologySet, TopologySnapshot, TopologySnapshotProvider};`
+ECStore keeps compile-time coverage for `StorageAdminApi`, `HealOperations`, and the separate `NamespaceLocking` operation group (`crates/ecstore/tests/ecstore_contract_compat_test.rs`), and its internal consumers use the `rustfs-storage-api` lifecycle DTOs `ExpirationOptions` and `TransitionedObject` directly.
 
-Required `rustfs-concurrency` public workload admission contract re-exports:
+## Temporary Compatibility Code
 
-- `pub use workload::{AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider, WorkloadClass};`
+Every temporary compatibility path carries a `RUSTFS_COMPAT_TODO(<id>)` source marker with a removal condition and a matching entry in [compat-cleanup-register.md](compat-cleanup-register.md); the guard enforces the match in both directions. Compatibility layers are deleted in their own cleanup change, never bundled with new migration logic.
 
-ECStore must keep compile-time coverage for `StorageAdminApi`, `HealOperations`,
-and the separate `NamespaceLocking` operation group.
+## Config Model
 
-The old `StorageAPI` aggregate facade must not reappear in production
-`crates/ecstore/src` or `rustfs/src` code after the storage operation groups
-have been made explicit.
+The server-config model (`Config`, `KV`, `KVS`) and the global server-config snapshot accessors are owned by `rustfs_config::server_config`; ECStore keeps persistence, storage-class state, and startup wiring, and its public facades must not re-export those symbols. See [config-model-boundary-adr.md](config-model-boundary-adr.md).
 
-Outer RustFS/IAM consumers must use `rustfs-storage-api` generic list response
-contracts directly for `ListObjectsV2Info`, `ListObjectVersionsInfo`, and
-`ObjectInfoOrErr`; ECStore keeps the concrete aliases only for internal
-implementation and compatibility.
+## Required Architecture Documents
 
-Outer RustFS/scanner consumers must use `rustfs-storage-api` operation traits
-directly for `ObjectIO`, `ObjectOperations`, `ListOperations`,
-`MultipartOperations`, `HealOperations`, and `NamespaceLocking`; ECStore keeps
-the concrete compatibility traits only for internal implementation and
-downstream compatibility.
-Outer consumers must not import ECStore directly outside compatibility
-boundaries except for temporary trait imports needed for method resolution or
-local test trait implementations. Non-trait ECStore surfaces must stay behind
-local aliases, constants, or wrapper functions.
-
-Outer compatibility boundary modules must use `rustfs_ecstore::api` for ECStore
-public facade surfaces such as layout, storage owner, admin, metrics,
-notification, capacity, bucket/config helpers, disk/error contracts, global
-state accessors, RPC constants/clients, reader helpers, tier helpers, and
-rebalance status contracts. Any non-ECStore `storage_compat.rs` import from
-`rustfs_ecstore` must route through the `rustfs_ecstore::api` facade.
-The legacy ECStore root `endpoints` and `disks_layout` compatibility modules
-must remain crate-private; public layout access goes through
-`rustfs_ecstore::api::layout`.
-Facade-covered ECStore root modules must remain crate-private after this
-boundary is established; outer crates should use `rustfs_ecstore::api::*`
-instead of legacy root module paths. This includes storage/layout surfaces as
-well as remaining bitrot, erasure coding, object DTO/reader, event, list, and
-batch processor root modules once their facade groups exist.
-ECStore root `global` re-exports must also stay removed once consumers use
-`rustfs_ecstore::api::global` or crate-internal `crate::global` paths.
-RustFS root `storage_compat.rs` must expose bucket metadata and quota contracts
-as explicit aliases only. Broad `metadata`, `metadata_sys`, and `quota` module
-passthroughs are reserved to narrower app/admin/storage compatibility
-boundaries that still need module-local owner cleanup.
-Root runtime storage config initialization and disk endpoint contracts must also
-stay explicit aliases. The root compatibility boundary must not restore `com`,
-bare `init`, or grouped `endpoint::Endpoint` passthroughs.
-RustFS root `storage_compat.rs` must not re-export ECStore API symbols directly;
-remaining root runtime compatibility symbols must be local type aliases,
-constants, traits, or wrapper functions so ownership stays visible at the
-boundary.
-RustFS admin `storage_compat.rs` must expose config IO and default
-initialization through explicit aliases. The admin compatibility boundary must
-not restore broad `com` or bare `init` passthroughs.
-RustFS admin and app `storage_compat.rs` bucket-facing compatibility contracts
-must stay explicitly whitelisted. They must not restore broad bucket module,
-client object API, client transition API, or storage-class module passthroughs
-once a local compatibility boundary has narrowed them to specific aliases.
-RustFS storage `storage_compat.rs` must expose bucket metadata, object-lock,
-policy, replication, tagging, versioning, object API, and test-only
-storage-class config contracts through explicit aliases. The storage
-compatibility boundary must not restore broad `metadata`, `metadata_sys`,
-`object_lock`, `policy_sys`, `replication`, `tagging`, `utils`, `versioning`,
-`versioning_sys`, `object_api_utils`, or `com` passthroughs.
-RustFS storage owner `storage_compat.rs` must not re-export ECStore API symbols
-directly except temporary trait imports needed for method resolution. Remaining
-storage-owner compatibility symbols must be local constants, type aliases, or
-wrapper functions so storage-owned global state and helper access stays visible
-at the boundary.
-RustFS app, admin, and storage outer `storage_compat.rs` object and error
-facade aliases must stay anchored on storage-api associated object types and
-local `StorageError` aliases. They must not reintroduce raw
-`rustfs_ecstore::api::object::{ObjectInfo,ObjectOptions}` or
-`rustfs_ecstore::api::error::{Error,Result}` references.
-Outer compatibility function signatures must also use local aliases for ECStore
-metadata, object-lock, lifecycle journal, monitor, and notification facade
-types. The boundary may define the local alias, but call signatures must not
-expose the raw ECStore facade path once narrowed.
-The RustFS storage owner compatibility boundary must keep raw ECStore facade
-paths centralized behind local `ecstore_*` module aliases rather than scattering
-`rustfs_ecstore::api::...` references through its aliases and wrappers.
-The RustFS app/admin storage compatibility boundaries must likewise route raw
-ECStore facade access through their local `ecstore_*` module aliases instead of
-scattering `rustfs_ecstore::api::...` paths through compatibility wrappers.
-Peripheral consumer storage compatibility boundaries must follow the same
-pattern. IAM, heal, scanner, notify, observability, Swift, S3 Select, test, and
-fuzz storage compatibility modules keep raw ECStore facade access centralized
-behind local `ecstore_*` module aliases.
-RustFS root runtime and e2e storage compatibility boundaries must follow the
-same pattern, keeping raw ECStore facade access centralized behind local
-`ecstore_*` module aliases.
-Outer bucket lifecycle, replication, versioning, object-lock, and
-restore-request trait method access must stay behind local compatibility traits
-or wrapper functions. Non-compat sources must not import those ECStore bucket
-API traits directly after the wrapper boundary is established. Disk, RPC peer
-client, and warm-backend method-resolution access must follow the same pattern:
-non-compat sources use owner-local compatibility traits or test aliases instead
-of importing ECStore traits directly.
-Scanner, notify, observability, and e2e `storage_compat.rs` boundaries must
-also stay narrow. Scanner must not restore grouped bucket compatibility exports
-for target, lifecycle, metadata, replication, or versioning modules. Notify
-must not restore broad `config`/`global` module imports. Observability must
-consume data usage through a local DTO projection instead of re-exporting the
-ECStore data-usage loader. The e2e harness must not restore grouped RPC
-passthroughs.
-Test and fuzz `storage_compat.rs` harnesses must also stay narrow. Heal and
-scanner test harnesses must expose ECStore contracts through direct aliases or
-local wrappers, and fuzz harnesses must wrap bucket utility entrypoints instead
-of restoring grouped ECStore passthrough exports.
-External ECStore API facade imports must stay inside local `storage_api`
-boundary files after the external runtime, test, and fuzz consumers have been
-narrowed. IAM, heal, scanner, notify, observability, Swift, S3 Select, e2e, and
-fuzz code must not reintroduce direct `rustfs_ecstore::api::...` references
-outside those boundary files.
-The observability ECStore dependency baseline is tracked in
-[`obs-ecstore-dependency-inventory.md`](obs-ecstore-dependency-inventory.md);
-future observability extraction PRs must update that inventory with the guard.
-
-ECStore ClusterControlPlane read models must stay owned by the crate-private
-`cluster` module. Public access goes through `rustfs_ecstore::api::cluster` so
-outer crates cannot depend on ECStore root control-plane internals.
-Pool-state, local-node storage, and peer-health status projections are part of
-the same facade boundary and must remain read-only until a later controller
-slice explicitly wires dynamic health or membership behavior.
-
-RustFS startup internals must stay crate-private after the startup owner split.
-Only `startup_entrypoint` remains a public startup module for the binary
-entrypoint; IAM bootstrap, optional runtime, and profiling startup shims must
-not be re-exported as public library modules. Items inside crate-private
-startup modules must also use crate visibility rather than bare public
-visibility.
-
-ECStore internal consumers must use `rustfs-storage-api` lifecycle helper DTOs
-directly for `ExpirationOptions` and `TransitionedObject`; ECStore keeps the
-old lifecycle paths only as downstream compatibility re-exports.
+The guard requires the documents and section headings listed in its `require_source_contains` entries (`scripts/check_architecture_migration_rules.sh`); the directory index is [README.md](README.md).

docs/architecture/decommission-compatibility.md

@@ -1,47 +1,31 @@
 # Decommission Compatibility Scope
 
-This note records the current RustFS decommission contract for admin/API
-compatibility reviews.
+**Use this when:** you change pool decommission or rebalance behavior, its admin API shape, the persisted `PoolMeta` decommission fields, or how tier free versions move between pools.
+**Source of truth:** `crates/ecstore/src/core/pools.rs` (queue, recovery, cleanup predicates), `crates/ecstore/src/services/rebalance/worker.rs` (rebalance predicates), `rustfs/src/admin/handlers/pools.rs` plus the `pools/*` rows of `rustfs/src/admin/route_policy.rs` (admin surface), `crates/ecstore/src/data_movement/` and `crates/ecstore/src/set_disk/` (free-version movement).
 
 ## Current Contract
 
-RustFS supports queued multi-pool decommission start requests on multi-pool
-deployments.
-
-The admin handler accepts the request shape used by the MinIO-compatible admin
-API, including comma-separated pool targets. An empty target list is rejected.
-Single-pool deployments reject decommission because there is no destination pool.
-On multi-pool deployments, one or more valid target pools are accepted as a
-single queued operation.
+RustFS supports queued multi-pool decommission start requests on multi-pool deployments. The admin handler accepts the MinIO-compatible request shape, including comma-separated pool targets. An empty target list is rejected; single-pool deployments reject decommission because there is no destination pool; on multi-pool deployments one or more valid target pools are accepted as a single queued operation.
 
 ### Request Semantics
 
-`POST /v3/pools/decommission` with comma-separated pool targets is treated as a
-queue submission:
+`POST /v3/pools/decommission` with comma-separated pool targets is a queue submission:
 
 - validate all requested pool identifiers before mutating metadata;
 - reject duplicate target pools in the same request;
 - reject active or queued target pools;
-- reject completed decommission targets because completion means the pool can be
-  removed from the deployment configuration;
+- reject completed decommission targets, because completion means the pool can be removed from the deployment configuration;
 - allow failed or canceled targets to be retried;
 - persist queued metadata before starting workers;
 - start only the local-leader prefix of the queue on the receiving node.
 
-The local-leader-prefix rule keeps the active worker on the leader for the pool
-being moved while still allowing a request to contain later targets whose leaders
-are different nodes. Later queued targets are recovered or promoted by the
-leader that owns that target.
+The local-leader-prefix rule keeps the active worker on the leader for the pool being moved while still allowing a request to contain later targets whose leaders are different nodes. Later queued targets are recovered or promoted by the leader that owns that target.
 
-Admin start, cancel, and clear requests may arrive on any cluster node. When the
-target pool first endpoint is remote, RustFS forwards the operation over the
-authenticated internode RPC channel to that first endpoint. The receiving node
-still enforces the local-leader rule before mutating decommission state.
+Start, cancel (`POST /v3/pools/cancel`), and clear (`POST /v3/pools/clear`) requests may arrive on any cluster node. When the target pool's first endpoint is remote, RustFS forwards the operation over the authenticated internode RPC channel to that endpoint; the receiving node still enforces the local-leader rule before mutating decommission state.
 
 ### Persisted Metadata Shape
 
-The queue is persisted in pool metadata and decoded with the rest of
-`PoolMeta`. Each pool entry can distinguish:
+The queue is persisted in pool metadata and decoded with the rest of `PoolMeta`. Each pool entry can distinguish:
 
 - `active`: at most one pool currently moving data;
 - `queued`: validated pools waiting for the active entry to finish;
@@ -49,310 +33,112 @@ The queue is persisted in pool metadata and decoded with the rest of
 - `failed`: pools whose worker reached terminal failure;
 - `canceled`: pools canceled before or during execution.
 
-Legacy metadata without queue fields decodes as a non-queued decommission entry,
-preserving restart behavior for already deployed clusters.
+Legacy metadata without queue fields decodes as a non-queued decommission entry, preserving restart behavior for already deployed clusters.
 
 ### Serial Scheduling And Recovery
 
 Only one queued entry may own a decommission worker at a time. Startup recovery:
 
 - loads pool metadata before rebalance recovery;
-- resumes the first local non-terminal active/queued entry;
-- skips a durably completed prefix and promotes the next queued entry only after
-  successful completion;
-- treats failed or canceled terminal entries as an automatic-promotion barrier,
-  leaving later queued pools visible but stopped until an operator retries,
-  clears, or otherwise resolves the terminal entry;
-- keeps queued pools out of active worker scheduling until promotion, while still
-  making their future state visible in admin status.
+- computes the resumable entries with `resumable_decommission_queue_indices` (`crates/ecstore/src/core/pools.rs`): every pool that has decommission state and is not terminal (`complete`, `failed`, or `canceled`). Terminal predecessors are skipped, not treated as barriers, so a queued pool behind a failed or canceled attempt is still resumable (`test_resumable_decommission_queue_indices_skip_terminal_predecessors`);
+- starts workers only for the local-leader prefix of those entries; later queued pools stay out of worker scheduling until promotion while their state remains visible in admin status.
 
-Promotion is persisted before worker execution. If cancellation is already
-requested immediately after promotion, RustFS persists a canceled terminal state
-instead of leaving the promoted pool active without a worker.
+Promotion is persisted before worker execution. If cancellation is already requested immediately after promotion, RustFS persists a canceled terminal state instead of leaving the promoted pool active without a worker.
 
 ### Cancel Semantics
 
 Cancel separates active and queued behavior:
 
-- canceling the active entry requests worker cancellation and persists terminal
-  metadata;
+- canceling the active entry requests worker cancellation and persists terminal metadata;
 - canceling a queued entry marks that entry canceled before it becomes active;
-- failed or canceled terminal entries can be cleared explicitly when the operator
-  chooses to abandon the decommission attempt;
-- peer reload failures during cancel must be surfaced in status and logs.
+- failed or canceled terminal entries can be cleared explicitly (`POST /v3/pools/clear`) when the operator abandons the decommission attempt;
+- peer reload failures during cancel are surfaced in status and logs.
 
-Cancel requests can be accepted on non-leader nodes as remote cancel intent; the
-leader observes the pending cancel and applies it to the active worker.
+Cancel requests can be accepted on non-leader nodes as remote cancel intent; the leader observes the pending cancel and applies it to the active worker.
 
 ### Status Response Shape
 
-`GET /v3/pools/list` and `GET /v3/pools/status?pool=...` expose per-pool
-machine-readable decommission state. The `status` field can report `active`,
-`running`, `queued`, `complete`, `failed`, or `canceled`.
+`GET /v3/pools/list` and `GET /v3/pools/status?pool=...` expose per-pool machine-readable decommission state. The `status` field can report `active`, `running`, `queued`, `complete`, `failed`, or `canceled`.
 
 When decommission metadata is present, `decommissionInfo` includes:
 
 - queue and terminal flags: `queued`, `complete`, `failed`, `canceled`;
-- progress counters: `objectsDecommissioned`,
-  `objectsDecommissionedFailed`, `bytesDecommissioned`, and
-  `bytesDecommissionedFailed`;
+- progress counters: `objectsDecommissioned`, `objectsDecommissionedFailed`, `bytesDecommissioned`, and `bytesDecommissionedFailed`;
 - current location: `bucket`, `prefix`, and `object`;
 - queue/history lists: `queuedBuckets` and `decommissionedBuckets`;
-- `waitingReason`, currently `queued` for queued entries and
-  `waiting_for_worker` when metadata exists but no worker has started.
+- `waitingReason`: `queued` for queued entries and `waiting_for_worker` when metadata exists but no worker has started.
 
-This makes queued pools and stalled metadata visible without requiring operators
-to inspect pool metadata files directly.
+This makes queued pools and stalled metadata visible without requiring operators to inspect pool metadata files directly.
 
 ## MinIO Divergence Decisions
 
-This section records the current product decisions for behavior that is close to
-MinIO but not always byte-for-byte identical.
+Behavior that is close to MinIO but not byte-for-byte identical. Changing either decision requires an operator compatibility note and updated characterization tests.
 
 ### Empty Delete Markers
 
-MinIO decommission documentation states that empty delete markers, meaning delete
-markers with no successor object versions, are not transitioned to another pool.
+MinIO decommission documentation states that empty delete markers (delete markers with no successor object versions) are not transitioned to another pool. RustFS follows that behavior for decommission when the bucket has no replication configuration: a lone remaining delete marker is cleanup-only metadata and is skipped. When replication is configured, RustFS keeps the delete marker eligible for movement so delete-marker replication and purge state are not lost.
 
-RustFS follows that behavior for decommission when the bucket has no replication
-configuration: a lone remaining delete marker is treated as cleanup-only metadata
-and is skipped. When replication is configured, RustFS intentionally keeps the
-delete marker eligible for movement so delete-marker replication and purge state
-are not lost.
-
-RustFS rebalance uses the same predicate as decommission: skip only a lone delete
-marker without replication. This is intentional even though MinIO's public
-documentation calls out the decommission case more explicitly than the rebalance
-case.
-
-Regression guards:
-
-- `should_skip_decommission_delete_marker_characterizes_empty_marker_without_replication`
-- `should_skip_decommission_delete_marker_characterizes_replication_configured`
-- `test_should_skip_rebalance_delete_marker_characterizes_empty_marker_without_replication`
-- `test_should_skip_rebalance_delete_marker_characterizes_replication_configured`
+Rebalance uses the same predicate as decommission (`should_skip_decommission_delete_marker` in `crates/ecstore/src/core/pools.rs`, `should_skip_rebalance_delete_marker` in `crates/ecstore/src/services/rebalance/worker.rs`), even though MinIO's public documentation calls out the decommission case more explicitly than the rebalance case.
 
 ### Lifecycle-Expired Versions During Cleanup
 
-MinIO decommission ignores versions that are already expired by lifecycle rules.
-RustFS follows that decommission behavior by allowing safely expired versions to
-count toward source cleanup completion.
-
-RustFS rebalance is intentionally stricter. Expired versions do not prove that a
-target pool received an equivalent version, so rebalance cleanup requires actual
-rebalance completion for the source entry instead of treating lifecycle-expired
-versions as moved.
-
-Regression guards:
-
-- `test_should_cleanup_decommission_source_entry_accepts_migrated_and_safely_expired_versions`
-- `test_should_cleanup_decommission_source_entry_accepts_versions_only_safely_expired_by_lifecycle`
-- `test_should_cleanup_rebalance_source_entry_rejects_versions_only_expired_by_lifecycle`
-
-No migration step is required for these decisions because this note documents the
-current RustFS behavior. Changing either decision later requires an operator
-compatibility note and updated characterization tests.
+MinIO decommission ignores versions already expired by lifecycle rules. RustFS applies the same rule to decommission and rebalance: a source entry is cleanup-complete when moved versions plus safely expired versions equal the total version count (`should_cleanup_decommission_source_entry` in `crates/ecstore/src/core/pools.rs`, `should_cleanup_rebalance_source_entry` in `crates/ecstore/src/services/rebalance/worker.rs`). Versions retained by object lock or pending replication are not counted as safely expired by the callers, so an entry with such versions is retained. Both predicates accept an entry whose versions are all lifecycle-expired (`test_should_cleanup_decommission_source_entry_accepts_versions_only_safely_expired_by_lifecycle`, `test_should_cleanup_rebalance_source_entry_accepts_versions_only_expired_by_lifecycle`).
 
 ## Tier Free Versions During Decommission
 
-A tier free version is an internal xl.meta record (`rustfs_filemeta::FREE_VERSION`,
-flagged `XL_FLAG_FREE_VERSION`) shaped like a delete marker. It is created by
-`MetaObject::init_free_version` when a version whose remote transition completed is
-deleted locally: the visible version is removed and the record keeps the remote-tier
-identity (tier, object name, version id, state, destination id) needed for an
-idempotent remote delete. Free versions are not user-visible versions; `num_versions`
-and all listing/GET paths exclude them.
+A tier free version is an internal xl.meta record (`rustfs_filemeta::FREE_VERSION`, flagged `XL_FLAG_FREE_VERSION`) shaped like a delete marker. It is created by `MetaObject::init_free_version` when a version whose remote transition completed is deleted locally: the visible version is removed and the record keeps the remote-tier identity (tier, object name, version id, state, destination id) needed for an idempotent remote delete. Free versions are not user-visible versions; `num_versions` and all listing/GET paths exclude them.
 
 ### Lifecycle And Consumers
 
-Creation: a local delete that removes a version whose transition status is
-`complete` normally appends the record via `MetaObject::delete_version` →
-`init_free_version`. User-facing single and batch deletes always retain that
-historical owner when they actually remove a transitioned source; they do not
-create a tier journal, probe a fleet capability, or issue a peer mutation RPC.
-`TransitionVersionState::Unknown` and incomplete destination identities remain
-on the same conservative free-version path. Delete-marker creation on an Enabled
-bucket remains unchanged and does not schedule remote deletion.
+Creation: a local delete that removes a version whose transition status is `complete` normally appends the record via `MetaObject::delete_version` → `init_free_version` (skipped only when `skip_tier_free_version` is set, as on data-movement copies). User-facing single and batch deletes always retain that historical owner when they actually remove a transitioned source; they do not create a tier journal, probe a fleet capability, or issue a peer mutation RPC. `TransitionVersionState::Unknown` and incomplete destination identities stay on the same conservative free-version path. Delete-marker creation on an Enabled bucket is unchanged and does not schedule remote deletion.
 
-Recursive prefix/delete-all cannot preserve per-object markers across its
-physical directory purge, so it requires a v6 recoverable journal for every
-transitioned visible source plus a durable dispatch manifest for the complete
-operation. It fails closed before mutation on legacy metadata or any existing
-hidden tier free-version under the prefix. Its internal streaming walk
-discovers logical keys, then exact-loads every key from its authoritative set in
-every pool, including free versions; the S3 listing merge is never treated as a
-complete physical-owner inventory. Tier-operation leases remain held from that
-preflight through journal prepare and physical deletion. Once physical deletion
-starts, any error is mutation-ambiguous: authorized/dispatched journals remain
-for recovery to commit owners only after all physical sets prove both the source
-and exact free-version identity absent; uncertain owners are retained.
-If a retry discovers a later transitioned source after the manifest reached
-`DispatchAuthorized`, it replays only the manifest's immutable predecessor set,
-completes that operation, and leaves the newcomer for a successor dispatch.
-Operators may retry after the legacy free-version worker has durably completed
-remote and local cleanup. Journal-less internal deletes and older nodes retain
-their established marker behavior.
+Recursive prefix/delete-all cannot preserve per-object markers across its physical directory purge, so it requires a v6 recoverable journal for every transitioned visible source plus a durable dispatch manifest for the whole operation. It fails closed before mutation on legacy metadata or on any existing hidden tier free-version under the prefix. Its internal streaming walk discovers logical keys, then exact-loads every key from its authoritative set in every pool, including free versions; the S3 listing merge is never treated as a complete physical-owner inventory. Tier-operation leases stay held from that preflight through journal prepare and physical deletion. Once physical deletion starts, any error is mutation-ambiguous: authorized/dispatched journals remain for recovery to commit owners only after all physical sets prove both the source and the exact free-version identity absent; uncertain owners are retained. If a retry discovers a later transitioned source after the manifest reached `DispatchAuthorized`, it replays only the manifest's immutable predecessor set, completes that operation, and leaves the newcomer for a successor dispatch. Operators may retry after the legacy free-version worker has durably completed remote and local cleanup. Journal-less internal deletes and older nodes keep their established marker behavior.
 
-Consumption while the record exists: the background recovery loop started by
-`init_background_expiry` (spawned by `spawn_tier_free_version_recovery_once`,
-enabled by default) scans disks for pending records and re-enqueues them; the
-usage scanner does the same; the lifecycle worker then deletes the remote tier
-object idempotently and only afterwards removes the local record. Heal walks
-include free-version records in metadata healing. Transition planning,
-replication, restore, GET, listings, and usage aggregation never depend on
-them.
+Consumption while the record exists: the background recovery loop started by `init_background_expiry` (spawned by `spawn_tier_free_version_recovery_once`, enabled by default) scans disks for pending records and re-enqueues them; the usage scanner does the same; the lifecycle worker then deletes the remote tier object idempotently and only afterwards removes the local record. Heal walks include free-version records in metadata healing. Transition planning, replication, restore, GET, listings, and usage aggregation never depend on them.
 
 ### Decommission Handling
 
-The exact decommission inventory loader (`load_file_info_versions_exact` via
-`get_all_file_info_versions`) keeps free-version records inline in `versions`.
-The migration loop handles them before lifecycle expiry and delete-marker
-shortcuts. It selects a target pool using the free-version-aware lookup, then
-writes the original free record to every target disk with the normal metadata
-write quorum. The free-version marker, local version id, transition identity,
-transition state, and destination id are preserved at the FileInfo/metadata
-boundary.
+The exact decommission inventory loader (`load_file_info_versions_exact` via `get_all_file_info_versions`) keeps free-version records inline in `versions`. The migration loop handles them before lifecycle expiry and delete-marker shortcuts. It selects a target pool using the free-version-aware lookup, then writes the original free record to every target disk with the normal metadata write quorum. The free-version marker, local version id, transition identity, transition state, and destination id are preserved at the FileInfo/metadata boundary.
 
-The source record is physically removed only after the target write quorum has
-committed and the source cleanup preflight still matches the exact inventory.
-If the lifecycle worker has already completed the remote delete and removed the
-source record before decommission acquires the source lock, decommission records
-that identity as already consumed and treats the missing source record as safe.
-If target capacity, metadata validation, lock fencing, or quorum fails, the
-source record remains and the entry records `state = "free_version_retained"`
-with reason `tier_free_version_migration_failed`; the worker retries the
-operation on a later pass. A target record with the same version id is accepted
-only when its free-version identity matches; a conflicting ordinary version or
-different free record is an overwrite error. This makes retries idempotent and
-prevents a free record from replacing a user-visible version.
+The source record is physically removed only after the target write quorum has committed and the source cleanup preflight still matches the exact inventory. If the lifecycle worker has already completed the remote delete and removed the source record before decommission acquires the source lock, decommission records that identity as already consumed and treats the missing source record as safe. If target capacity, metadata validation, lock fencing, or quorum fails, the source record remains and the entry records `state = "free_version_retained"` with reason `tier_free_version_migration_failed`; the worker retries the operation on a later pass. A target record with the same version id is accepted only when its free-version identity matches; a conflicting ordinary version or different free record is an overwrite error. This makes retries idempotent and prevents a free record from replacing a user-visible version.
+
+`TransitionVersionState::Unknown` records are migrated unchanged rather than discarded; the lifecycle worker retains them if remote identity validation cannot make a delete request. Only an authorized recursive prefix/delete-all v6 transaction may use a per-source journal as the sole retry source; ordinary single/batch deletes never take that path, and a journal discovered alongside an older or fallback free-version never authorizes dropping the xl.meta record.
 
 ### Remote-Tuple Publication Fence
 
-Cross-pool capability v3 includes a commit-late publication contract for every
-path that can copy an existing transition tuple to a new physical owner. This
-capability version is independent of the tier-mutation RPC protocol version.
-A mixed fleet whose minimum cross-pool capability is below v3 cannot authorize
-journal-v6 remote deletion.
+Cross-pool capability v3 adds a commit-late publication contract for every path that can copy an existing transition tuple to a new physical owner. This capability version is independent of the tier-mutation RPC protocol version; a mixed fleet whose minimum cross-pool capability is below v3 cannot authorize journal-v6 remote deletion.
 
-Data movement captures a non-cloneable, process-local source capability before
-copying, but it does not hold a namespace write lock or tier-operation lease
-while reading a large body or uploading multipart parts. `NewMultipartUpload`
-and `UploadPart` are staging only. Immediately before single-PUT rename,
-Multipart Complete, or a pure-remote/free-version metadata quorum write, the
-final consumer acquires the exact tier generation (when a remote tuple exists),
-then fixed/source/target write domains in stable order. The fixed domain is used
-only for a real remote-tuple decommission publisher; an ordinary local object
-keeps the lighter source/target commit scope.
+Data movement captures a non-cloneable, process-local source capability before copying, but it does not hold a namespace write lock or tier-operation lease while reading a large body or uploading multipart parts (`NewMultipartUpload` and `UploadPart` are staging only). Immediately before single-PUT rename, Multipart Complete, or a pure-remote/free-version metadata quorum write, the final consumer acquires the exact tier generation (when a remote tuple exists), then the fixed/source/target write domains in stable order. The fixed domain is used only for a real remote-tuple decommission publisher; an ordinary local object keeps the lighter source/target commit scope.
 
-While that owned scope is held, the publisher re-reads the exact source pool and
-compares version, data directory, modification time, ETag, checksums, transition
-tuple, transition-version state, and destination identity. A missing or changed
-source, changed/revoked tier generation, bucket incarnation change, or lost lock
-fails before target rename. The scope remains owned through rename quorum and
-the existing rename-tail guard handoff. Consequently, recovery-first ordering
-cannot delete the remote object and then have a stale restored-transitioned
-rebalance recreate its tuple; publisher-first ordering makes recovery wait and
-rescan the newly committed owner.
+While that owned scope is held, the publisher re-reads the exact source pool and compares version, data directory, modification time, ETag, checksums, transition tuple, transition-version state, and destination identity. A missing or changed source, a changed or revoked tier generation, a bucket incarnation change, or a lost lock fails before target rename. The scope stays owned through rename quorum and the rename-tail guard handoff, so recovery-first ordering cannot delete the remote object and then let a stale restored-transitioned rebalance recreate its tuple, and publisher-first ordering makes recovery wait and rescan the newly committed owner.
 
-Full cross-key S3 Copy is not an ownership-sharing operation: it materializes
-local data and strips transition, destination, transaction, and free-version
-keys. Same-key metadata/version-only updates preserve the existing protected
-state. Admin heal keeps the legacy `nolock` request field for wire compatibility
-but ignores it as lock authority; final heal writes enter the normal locked
-path. Restore similarly ignores ambient `ObjectOptions.no_lock`, acquires its
-own commit-late PUT/Complete lock, validates the restore operation id, and keeps
-an exact tier generation lease through the local commit.
+Full cross-key S3 Copy is not an ownership-sharing operation: it materializes local data and strips transition, destination, transaction, and free-version keys. Same-key metadata/version-only updates preserve the protected state. Admin heal keeps the legacy `nolock` request field for wire compatibility but ignores it as lock authority; final heal writes enter the normal locked path. Restore likewise ignores ambient `ObjectOptions.no_lock`, acquires its own commit-late PUT/Complete lock, validates the restore operation id, and keeps an exact tier generation lease through the local commit.
 
-### Reference-Audit Result
+### Tier Mutation Protocol And Journal v6 Rollout
 
-After migration, user-facing GET/list/transition/replication/restore paths still
-exclude the record. Recovery, usage scanning, lifecycle tier cleanup, and heal
-continue to see a legacy/fallback record when they request free versions, so an
-unresolved remote delete remains actionable on the target pool. Only an
-authorized recursive prefix/delete-all v6 transaction may instead use a
-per-source journal as the sole retry source; ordinary single/batch deletes never
-take that path. A journal discovered
-alongside an older or fallback free-version does not authorize dropping the
-record. In particular, `Unknown` transition state records are migrated unchanged
-rather than discarded: the lifecycle worker retains them if remote identity
-validation cannot make a delete request.
+Tier edit/remove/clear reference proof uses the internal walk with `include_free_versions = true`, in addition to persisted journal and transition-transaction checks. Protocol v3 peer Prepare blocks new reference creators and drains existing tier-operation leases before this proof; protocol v4 preserves that state machine and adds a signed failure classification. Abort carries the canonical Prepare intent, so a peer can create an identity-bound `Aborted` tombstone even when Abort overtakes Prepare; a delayed matching Prepare then converges on `Aborted` instead of reinstalling the block, and a conflicting intent with the same mutation id fails closed. The tombstone stays durable until intent expiry plus the configured clock-skew allowance, including across reload and coordinator-record cleanup. After expiry, a missing-record replay of the original signed Prepare is rejected and cannot recreate a peer-only runtime fence. Abort checks an existing same-identity terminal record before consulting mutable current-config proof, and recovery reconstructs the original Prepared revision for Abort fanout.
 
-Tier edit/remove/clear reference proof uses the internal walk with
-`include_free_versions = true`, in addition to persisted journal and transition
-transaction checks. Protocol v3 peer Prepare blocks new reference creators and
-drains existing tier-operation leases before this proof; protocol v4 preserves
-that state machine and adds a signed failure classification. Abort carries the
-canonical Prepare intent, so a peer can create an identity-bound `Aborted`
-tombstone even when Abort overtakes Prepare. A delayed matching Prepare then
-converges on `Aborted` instead of reinstalling the block; a conflicting intent
-with the same mutation id fails closed. The tombstone remains durable until the
-intent expiry plus the configured clock-skew allowance, including across reload
-and coordinator-record cleanup. After
-expiry, a missing-record replay of the original signed Prepare is rejected and
-cannot recreate a peer-only runtime fence. Abort checks an existing same-identity
-terminal record before consulting mutable current-config proof, and recovery
-reconstructs the original Prepared revision for Abort fanout.
+A new server accepts both v3 and v4 requests and selects the matching canonical response proof. During a mixed rollout an older v3 server rejects a v4 request with an authenticated, byte-exact unsupported-version status before dispatch; the v4 coordinator treats only that exact rejection as definitely-not-installed, fails the admin mutation, and does not send the peer an incompatible Abort. There is deliberately no automatic v3 retry: `Unimplemented`, near-text, timeouts, missing or unknown failure classes, and other ambiguous outcomes still receive Abort and retain the coordinator retry record if Abort cannot be proven. Operators must pause and drain tier edit/remove/clear operations before starting a rolling upgrade, leave them disabled while any v3-only peer remains, and resume only after every topology member advertises the v4-capable release. Ordinary object I/O and free-version cleanup stay available; `xl.meta` is unchanged by a rejected mutation.
 
-A new server accepts both v3 and v4 requests and selects the matching canonical
-response proof. During a mixed rollout, an older v3 server rejects a v4 request
-with an authenticated, byte-exact unsupported-version status before dispatch;
-the v4 coordinator treats only that exact rejection as definitely not installed,
-fails the admin mutation, and does not send the peer an incompatible Abort.
-There is deliberately no automatic v3 retry. `Unimplemented`, near-text,
-timeouts, missing/unknown failure classes, and other ambiguous outcomes still
-receive Abort and retain the coordinator retry record if Abort cannot be proven.
-Operators must pause and drain tier edit/remove/clear operations before starting
-the rolling upgrade, leave them disabled while any v3-only peer remains, and
-resume only after every topology member advertises the v4-capable release.
-Ordinary object I/O and free-version cleanup remain available; xl.meta is
-unchanged by the rejected mutation.
+Sole-owner transactions use journal v6: v5-and-older readers reject and retain those records, so an old recovery worker cannot bypass the all-pool proof. Older nodes may keep creating fallback free-versions until the rollout is homogeneous. Do not downgrade every v6-aware recovery worker while any v6 record remains; drain the journal first or keep at least one v6-aware worker until cleanup converges.
 
-Sole-owner transactions use journal v6: v5-and-older readers reject and retain
-those records, so an old recovery worker cannot bypass the all-pool proof. Older
-nodes may continue to create fallback free-versions until the rollout is
-homogeneous. A deployment must not downgrade every v6-aware recovery worker
-while any v6 record remains; drain the journal first or keep at least one v6-aware
-worker until cleanup converges.
+### Disposition Events
 
-Each migrated record emits `state = "free_version_migrated"` with reason
-`tier_free_version_migrated`. A record consumed before migration emits
-`state = "free_version_consumed"` with reason
-`tier_free_version_already_consumed`. Each failed record emits the retained state
-and failure reason above. The entry also emits a disposition summary with
-migrated, consumed, retained, and total counts. The final decommission sweep uses
-the exact loader, counts free records still present, and emits one retained
-record/reason for each unresolved free version before failing the sweep. This
-makes successful migration, completed cleanup, and retained cleanup obligations
-visible instead of silently omitting free records.
+Free versions remain internal, so no S3-visible version or admin response field is added. The structured `decommission_entry` events are the operational status surface:
 
-No new S3-visible version or admin response field is needed: free versions remain
-internal and are never counted as user-visible versions. The structured
-`decommission_entry` events are the operational status surface for the
-free-version disposition; the existing decommission item/failed counters still
-report the enclosing object migration result.
+| Outcome | `state` | `reason` |
+|---|---|---|
+| Record migrated to the target pool | `free_version_migrated` | `tier_free_version_migrated` |
+| Record consumed by the lifecycle worker before migration | `free_version_consumed` | `tier_free_version_already_consumed` |
+| Migration failed, source retained for retry | `free_version_retained` | `tier_free_version_migration_failed` |
 
-Regression guard:
+The entry also emits a disposition summary with migrated, consumed, retained, and total counts. The final decommission sweep uses the exact loader, counts free records still present, and emits one retained record/reason per unresolved free version before failing the sweep. The existing decommission item/failed counters still report the enclosing object migration result.
 
-- `decommission_tier_free_version_preserves_remote_identity`
-- `decommission_tier_free_version_resume_requires_write_quorum`
-- `decommission_tier_free_version_commit_rejects_lost_fence`
-- `test_decommission_cleanup_preflight_accepts_migrated_free_version_consumed_from_source`
-- `decommission_entry_skips_cleanup_only_marker_when_free_version_is_present`
-- `decommission_entry_rejects_subquorum_free_version_conflict_and_retains_source`
+## Regression Guards
 
-## Regression Guard
+Test names drift; locate the current guards instead of copying them:
 
-The queued multi-pool contract is guarded by:
-
-- `test_contextualized_decommission_start_request_allows_multiple_target_pools`
-- `test_decommission_start_local_leader_allows_remote_queued_pool`
-- `test_local_decommission_queue_prefix_stops_at_remote_leader`
-- `test_decommission_peer_target_returns_none_for_local_first_endpoint`
-- `test_pool_meta_queued_decommission_is_not_suspended_until_promoted`
-- `test_pool_meta_promoted_queued_decommission_can_be_canceled`
-- `test_first_resumable_decommission_queue_indices_stops_at_failed_or_canceled_state`
-- `test_first_resumable_decommission_queue_indices_allows_after_completed_prefix`
-- `admin_pool_list_item_exposes_queued_decommission_state`
-
-These tests live in `crates/ecstore/src/core/pools.rs` and
-`rustfs/src/app/admin_usecase.rs`.
+```bash
+rg -n 'fn [a-z_]*decommission[a-z_]*\(' crates/ecstore/src/core/pools.rs crates/ecstore/src/set_disk/mod.rs crates/ecstore/src/data_movement/mod.rs crates/ecstore/src/store/init.rs rustfs/src/admin/handlers/pools.rs rustfs/src/app/admin_usecase.rs
+rg -n 'fn test_should_[a-z_]*rebalance[a-z_]*\(' crates/ecstore/src/services/rebalance/rebalance_unit_tests.rs
+```

docs/architecture/ecstore-api-facade-inventory.md

@@ -1,76 +1,65 @@
 # ECStore API Facade Inventory
 
-This inventory records the current `rustfs_ecstore::api` compatibility surface
-before any ECStore split PR removes or narrows re-exports. It is a planning and
-guardrail document only. It must not be used as approval to move lifecycle,
-replication, or `SetDisks` runtime behavior.
+**Use this when:** you need something from `rustfs_ecstore` in another crate, you are narrowing a `rustfs_ecstore::api` facade group, or the architecture guard reports a facade bypass.
+**Source of truth:** `crates/ecstore/src/api/mod.rs` (facade groups), the boundary files listed below, and the facade rules in `scripts/check_architecture_migration_rules.sh`.
+
+The broad `rustfs_ecstore::api` facade is a compatibility boundary, not an architecture target. It shrinks monotonically and only through guarded changes; it is never approval to move lifecycle, replication, or `SetDisks` runtime behavior.
 
 ## Facade Group Inventory
 
-| Facade group | Current role | Shrink posture |
+| Facade group | Role | Shrink posture |
 |---|---|---|
 | `storage`, `layout`, `error`, `runtime`, `cluster`, `rpc` | Compatibility spine for storage, topology, runtime handles, cluster control, and internode calls. | Keep until replacement contracts compile in downstream boundary files. |
-| `bucket` | Domain facade consumed through owner-local `storage_api` boundaries. The public API keeps compatibility paths but exposes explicit submodules and symbol lists instead of whole bucket owner modules. | Keep explicit lists aligned with owner-boundary consumers; do not restore whole-module passthroughs. |
-| `client`, `config`, `disk`, `tier` | Compatibility paths consumed through owner-local `storage_api` boundaries. The public API keeps existing path names but exposes explicit nested submodules and symbol lists instead of whole owner modules. | Keep explicit lists aligned with owner-boundary consumers; do not restore whole-module passthroughs. |
-| `data_usage`, `capacity`, `notification`, `metrics`, `rebalance` | Domain and service facades still consumed through owner-local `storage_api` boundaries. | Narrow one group at a time after explicit aliases or wrappers exist. |
-| `set_disk`, `object`, `rio`, `bitrot`, `erasure`, `compression`, `cache`, `store_list` | Low-level object IO, reader, erasure, cache, and migration helper compatibility. | Keep stable while `SetDisks` remains the shared state carrier. |
-| `admin`, `event`, `global` | Admin, event hook, and legacy global compatibility. | Keep `global` limited to bootstrap writes and lifecycle controls; read-only runtime access must use runtime-source contracts. |
+| `bucket` | Domain facade consumed through owner-local `storage_api` boundaries; explicit submodules and symbol lists, never whole bucket owner modules. | Keep lists aligned with boundary consumers; never restore whole-module passthroughs. |
+| `config`, `disk`, `tier` | Compatibility paths with explicit nested submodules and symbol lists. | Same as `bucket`. |
+| `data_usage`, `capacity`, `notification`, `metrics`, `rebalance` | Domain and service facades consumed through owner-local boundaries. | Narrow one group at a time after explicit aliases or wrappers exist. |
+| `set_disk`, `object`, `object_api_utils`, `rio`, `bitrot`, `erasure`, `compression`, `cache`, `store_list` | Low-level object IO, reader, erasure, cache, and migration helper compatibility. | Keep stable while `SetDisks` remains the shared state carrier. |
+| `admin`, `event`, `global` | Admin, event hook, and bootstrap-global compatibility. | `global` is limited to bootstrap writes and lifecycle controls; read-only runtime access goes through `runtime`. |
+
+The S3 client is no longer a facade group: it lives in `crates/s3-client` (`rustfs_s3_client`). Regenerate the group list with:
+
+```bash
+rg -n '^pub mod ' crates/ecstore/src/api/mod.rs
+```
 
 ## External Consumer Boundaries
 
-External `rustfs_ecstore::api` imports must stay in these local boundary files:
+External `rustfs_ecstore::api` imports stay in these local boundary files:
 
-| Boundary file | Current facade families |
+| Boundary file | Facade families consumed |
 |---|---|
-| `rustfs/src/storage/storage_api.rs` | Broad RustFS storage owner bridge for admin, explicit bucket facade submodules, capacity, client, compression, cluster, config, data usage, disk, error, event, global bootstrap controls, runtime-source getters, layout, metrics, notification, rebalance, rio, rpc, set disk, storage, and tier. Replication pool/stat handles are projected into RustFS-local wrapper types here. |
-| `crates/scanner/src/storage_api.rs` | Scanner bridge for bucket lifecycle, replication, metadata, capacity, config, data usage, disk, error, runtime, set disk, storage, and tier. Replication queue config, admission, and heal object DTOs are projected into scanner-local types here. |
-| `crates/obs/src/metrics/storage_api.rs` | Metrics bridge for bucket bandwidth, lifecycle, replication, quota, capacity, data usage, error, runtime, and storage. |
-| `crates/iam/src/storage_api.rs` | IAM bridge for config, error, notification, runtime, and storage. |
-| `crates/heal/src/heal/storage_api.rs` | Heal bridge for data usage, disk, error, runtime, and storage. |
-| `crates/notify/src/storage_api.rs` | Notification bridge for config, runtime, and storage. |
-| `crates/protocols/src/swift/storage_api.rs` | Swift bridge for bucket metadata, bucket metadata system, error, runtime, and storage. |
-| `crates/s3select-api/src/storage_api.rs` | S3 Select bridge for error, runtime, set disk, and storage. |
-| `crates/e2e_test/src/storage_api.rs` | E2E harness bridge for bucket targets, disk walking, and RPC helpers. |
-| `crates/heal/tests/*/storage_api.rs`, `crates/scanner/tests/storage_api/mod.rs`, `fuzz/fuzz_targets/*_storage_api.rs` | Test and fuzz bridges for the same compatibility seams under test. |
+| `rustfs/src/storage/storage_api.rs` | Broad storage-owner bridge: admin, bucket submodules, capacity, compression, cluster, config, data usage, disk, error, event, global bootstrap controls, runtime getters, layout, metrics, notification, rebalance, rio, rpc, set disk, storage, tier. Replication pool/stat handles are projected into RustFS-local wrapper types here. |
+| `rustfs/src/storage_api.rs`, `rustfs/src/admin/storage_api.rs`, `rustfs/src/app/storage_api.rs` | Root, admin, and app owner boundaries: explicit aliases only, no `metadata`, `metadata_sys`, `quota`, `com`, or bare `init` module passthroughs; object and error aliases anchor on storage-api associated types and a local `StorageError`. |
+| `crates/scanner/src/storage_api.rs` | Bucket lifecycle, replication, metadata, capacity, config, data usage, disk, error, runtime, set disk, storage, tier. Replication queue config, admission, and heal object DTOs are projected into scanner-local types. |
+| `crates/obs/src/metrics/storage_api.rs` | Bucket bandwidth, lifecycle, replication, quota, capacity, data usage, error, runtime, storage; data usage is consumed as a local DTO projection. |
+| `crates/iam/src/storage_api.rs` | Config, error, notification, runtime, storage. |
+| `crates/heal/src/heal/storage_api.rs` | Data usage, disk, error, runtime, storage. |
+| `crates/notify/src/storage_api.rs` | Config, runtime, storage; no broad `config` or `global` module imports. |
+| `crates/protocols/src/swift/storage_api.rs` | Bucket metadata, bucket metadata system, error, runtime, storage. |
+| `crates/s3select-api/src/storage_api.rs` | Error, runtime, set disk, storage. |
+| `crates/e2e_test/src/storage_api.rs` | E2E harness bridge for bucket targets, disk walking, and RPC helpers; no grouped RPC passthroughs. |
+| `crates/ecstore/tests/storage_api.rs`, `crates/heal/tests/storage_api.rs`, `crates/scanner/tests/storage_api/mod.rs`, `fuzz/fuzz_targets/*_storage_api.rs` | Test and fuzz bridges: direct aliases or local wrappers; fuzz harnesses wrap bucket utility entrypoints instead of grouped passthroughs. |
+| `crates/test-utils/src/ecstore_test_compat.rs`, `crates/iam/tests/ecstore_test_compat/mod.rs`, `crates/protocols/tests/ecstore_test_compat/mod.rs` | Test-only compatibility harnesses that import the facade directly for fixture setup. |
 
-New production imports outside these boundary files are migration drift. Add a
-local boundary or storage-api contract first, then route consumers through it.
+`crates/replication/src/storage_api.rs` shares the file name but is not an ECStore boundary: it owns the delete work DTOs of `rustfs-replication`, which imports neither `rustfs_ecstore` nor `rustfs-storage-api`.
+
+Regenerate the boundary list with:
+
+```bash
+rg -l 'rustfs_ecstore::api' crates rustfs/src fuzz -g '*.rs' -g '!crates/ecstore/src/**'
+```
+
+New production imports outside these files are migration drift. Do not add direct `rustfs_ecstore::api` imports outside the boundary files; add a local boundary or a storage-api contract first, then route consumers through it.
 
 ## Split Dependency Inventory
 
-| Candidate | ECStore dependencies that block a crate split | Required owner contracts before movement |
-|---|---|---|
-| Lifecycle | Object API, `ECStore`, `SetDisks`, runtime sources/globals, bucket metadata/versioning/object lock/replication, disk, config, notification, audit, and tier services. | `LifecycleObjectStore`, `LifecycleMetadataStore`, `LifecycleRuntime`, `LifecycleReplicationSink`, and `LifecycleAuditSink`. |
-| Replication | Bucket target and metadata systems, bucket target client config, disk, object API, runtime sources, notification, and SetDisks lock timing. | `ReplicationStorage`, `ReplicationMetadataStore`, `ReplicationRuntime`, `ReplicationEventSink`, and `ReplicationLifecycleBridge`. |
-| SetDisks | Shared disks, endpoints, format state, namespace locks, cache, and implementations for object IO, namespace locking, bucket, object, list, multipart, and heal operations. | Pure shard source, disk error, bitrot IO, namespace lock, metrics label, and file metadata contracts before any operation family moves. |
-
-`crates/ecstore/tests/ecstore_contract_compat_test.rs` keeps compile-time
-coverage for `ECStore` and `SetDisks` storage-api trait compatibility before
-any facade shrink or operation-family movement.
+Lifecycle, replication, and `SetDisks` split blockers, extracted contracts, and guard rule names are tracked in [ecstore-module-split-plan.md](ecstore-module-split-plan.md) and the module inventories `crates/ecstore/src/bucket/lifecycle/README.md` and `crates/ecstore/src/bucket/replication/README.md`. `crates/ecstore/tests/ecstore_contract_compat_test.rs` keeps compile-time coverage for `ECStore` and `SetDisks` storage-api trait compatibility before any facade shrink or operation-family movement.
 
 ## Shrink Rules
 
-1. Do not remove a facade item until its downstream boundary has compile-time
-   coverage or a documented replacement.
-2. Do not add direct `rustfs_ecstore::api` imports outside the boundary files
-   listed above.
-3. Do not split lifecycle or replication into crates while they depend on
-   ECStore runtime state, queues, notification, audit, scanner, or SetDisks
-   internals.
-4. Do not replace `SetDisks` with multiple runtime structs in one PR. Move one
-   operation family only after contracts and focused tests exist.
-5. Remove or narrow one facade group per PR so rollback preserves object IO,
-   quorum, lifecycle/replication queues, scanner repair, notification/audit
-   events, and metadata compatibility.
-6. Keep `api::bucket`, `api::client`, `api::config`, `api::disk`, and
-   `api::tier` on explicit submodules and symbol lists; do not restore
-   `pub use crate::<owner>::{...}` whole-module passthroughs for those groups.
-
-## First PR Checklist
-
-- inventory the facade group and all external boundary consumers;
-- add explicit aliases or wrappers before deleting any broad passthrough;
-- run `./scripts/check_architecture_migration_rules.sh`;
-- run focused compile or tests for the touched owner boundary;
-- keep runtime behavior unchanged unless the PR is explicitly a code-bearing
-  follow-up with its own rollback plan.
+1. Do not remove a facade item until its downstream boundary has compile-time coverage or a documented replacement.
+2. Do not add direct `rustfs_ecstore::api` imports outside the boundary files listed above.
+3. Do not split lifecycle or replication into crates while they depend on ECStore runtime state, queues, notification, audit, scanner, or `SetDisks` internals.
+4. Do not replace `SetDisks` with multiple runtime structs in one change; move one operation family only after contracts and focused tests exist.
+5. Remove or narrow one facade group per change so rollback preserves object IO, quorum, lifecycle/replication queues, scanner repair, notification/audit events, and metadata compatibility.
+6. Keep `api::bucket`, `api::config`, `api::disk`, and `api::tier` on explicit submodules and symbol lists; do not restore `pub use crate::<owner>::{...}` whole-module passthroughs for those groups.

docs/architecture/ecstore-config-consumer-inventory.md

@@ -1,199 +0,0 @@
-# ECStore Config Consumer Inventory
-
-This inventory is the Phase 0 baseline for moving
-`rustfs_ecstore::config::{Config, KV, KVS}` safely. It records the current
-definitions, persistence helpers, global accessors, and direct consumers before
-any contract extraction, global-state migration, or crate split.
-
-Related issue: [`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660)
-
-## Scope
-
-In scope:
-
-- `rustfs_ecstore::config::KV`
-- `rustfs_ecstore::config::KVS`
-- `rustfs_ecstore::config::Config`
-- `rustfs_ecstore::config::DEFAULT_KVS`
-- `rustfs_ecstore::config::{get_global_server_config, set_global_server_config}`
-- `rustfs_ecstore::config::com::{read_config_without_migrate, save_server_config}`
-- Consumers that persist, clone, inspect, mutate, or pass these types across
-  runtime boundaries.
-- Selected adjacent users of `rustfs_ecstore::config::com::{read_config,
-  save_config, delete_config}` and related helper variants are listed separately
-  when they appear outside the core `Config`, `KV`, and `KVS` consumer map.
-  This is not a complete `com.rs` move inventory; any future `com.rs` move must
-  first inventory ECStore-internal persistence helper users too.
-
-Out of scope:
-
-- Unrelated `Config` types from `rustfs::config`, SDKs, TLS, SSH, KMS, OIDC
-  client libraries, or local module-specific config structs.
-- Storage-class-only imports are not treated as `Config`, `KV`, or `KVS`
-  consumers unless they also use the server-config model.
-- Pure route/action snapshot work already covered by
-  [`admin-route-action-snapshot.md`](admin-route-action-snapshot.md).
-
-## Current Shape
-
-Arrows show current source dependency or call direction: the left node imports
-or calls the right node.
-
-```mermaid
-flowchart TB
-    EC["crates/ecstore/src/config"]
-    Store["crates/ecstore/src/store/mod.rs"]
-    AppCtx["rustfs/src/app/context.rs"]
-    Server["rustfs/src/server/{event,audit}.rs"]
-    Admin["rustfs/src/admin"]
-    Notify["crates/notify"]
-    Audit["crates/audit"]
-    Targets["crates/targets"]
-    IAM["crates/iam/src/oidc.rs"]
-    Scanner["crates/scanner/src/{runtime_config,scanner}.rs"]
-
-    Store --> EC
-    AppCtx --> EC
-    Server --> AppCtx
-    Admin --> EC
-    Notify --> EC
-    Audit --> EC
-    Targets --> EC
-    Notify --> Targets
-    Audit --> Targets
-    IAM --> EC
-    Scanner --> EC
-```
-
-The config model is currently both a persisted server-config representation and
-the runtime carrier for notify, audit, target-plugin, scanner, and OIDC
-settings. Any move must preserve that dual role until consumers are migrated
-behind narrower contracts.
-
-## Core Model And Global State
-
-| Item | Current owner | Current role | Migration note |
-|---|---|---|---|
-| `KV` | `crates/ecstore/src/config/mod.rs` | Key/value entry with `hidden_if_empty` metadata and serde compatibility. | Preserve field names, aliases, defaults, and redaction behavior before any model move. |
-| `KVS(Vec<KV>)` | `crates/ecstore/src/config/mod.rs` | Ordered key/value set used by server config, target factories, admin rendering, tests, and examples. | Preserve tuple shape and methods: `new`, `get`, `lookup`, `is_empty`, `keys`, `insert`, `extend`. |
-| `Config(HashMap<String, HashMap<String, KVS>>)` | `crates/ecstore/src/config/mod.rs` | Server config map by subsystem and target. | Direct `.0` access is widespread; add wrappers only after preserving the current public shape. |
-| `DEFAULT_KVS` | `crates/ecstore/src/config/mod.rs` | Registry for defaults across storage class, scanner, notify, audit, and OIDC. | Move defaults only after an explicit registration contract exists. |
-| `GLOBAL_SERVER_CONFIG` | `crates/ecstore/src/config/mod.rs` | Process-wide mutable server config snapshot. | Migrate readers behind `AppContext` or a server-config provider before changing storage. |
-| `ConfigSys::init` | `crates/ecstore/src/config/mod.rs` | Reads persisted config, looks up derived config, and stores the global snapshot. | Startup order must remain unchanged until the lifecycle contract owns this dependency. |
-| `read_config_without_migrate` | `crates/ecstore/src/config/com.rs` | Loads persisted server config through ECStore-owned object I/O and storage-admin contracts. | Persistence stays in `ecstore` until pure model and persistence are separated. |
-| `save_server_config` | `crates/ecstore/src/config/com.rs` | Persists the canonical server config object. | Preserve external object shape and config-history behavior. |
-| `get_global_server_config` / `set_global_server_config` | `crates/ecstore/src/config/mod.rs` | Clone/read and replace the global server-config snapshot. | Do not remove until all runtime readers have an injected provider path. |
-
-## Consumer Map
-
-### ECStore Ownership, Persistence, And Defaults
-
-| Files | Current usage |
-|---|---|
-| `crates/ecstore/src/config/mod.rs` | Defines `KV`, `KVS`, `Config`, defaults, global snapshot, initialization, and tests. |
-| `crates/ecstore/src/config/com.rs` | Encodes, decodes, reads, writes, creates, and normalizes server config objects through ECStore-local persistence helpers. |
-| `crates/ecstore/src/config/{notify,audit,oidc,scanner,storageclass}.rs` | Register default `KVS` values and subsystem-specific parsing helpers. |
-| `crates/ecstore/src/store/mod.rs` | Exposes store-level server-config accessors that delegate to the global config snapshot. |
-
-### App Context And Server Startup Consumers
-
-| Files | Current usage |
-|---|---|
-| `rustfs/src/app/context.rs` | Defines `ServerConfigInterface`, keeps an `AppContext` server-config handle, and still falls back to `get_global_server_config`. |
-| `rustfs/src/server/event.rs` | Resolves server config through app context/global fallback before starting the notification runtime. |
-| `rustfs/src/server/audit.rs` | Resolves server config through app context/global fallback before starting the audit runtime. |
-
-### Admin Control-Plane Readers And Writers
-
-| Files | Current usage |
-|---|---|
-| `rustfs/src/admin/handlers/config_admin.rs` | Reads active/persisted server config, validates against `DEFAULT_KVS`, mutates `KVS`, saves config history, saves server config, and updates the global snapshot. |
-| `rustfs/src/admin/handlers/oidc.rs` | Reads and writes OIDC provider `KVS`, saves server config, and compares persisted config against the global snapshot for restart signaling. |
-| `rustfs/src/admin/handlers/audit_runtime_config.rs` | Reads persisted config, applies audit runtime target changes, saves server config, and reloads audit runtime state. |
-| `rustfs/src/admin/handlers/notify_runtime_access.rs` | Reads notification runtime config snapshots and passes `KVS` target changes into the notification system. |
-| `rustfs/src/admin/handlers/{event,audit}.rs` | Lists and validates notification/audit targets from `Config`; tests build `KV` and `KVS` fixtures. |
-| `rustfs/src/admin/handlers/plugins_instances.rs` | Maps target plugin `KVS` to response payloads and applies runtime target edits. |
-| `rustfs/src/admin/handlers/target_descriptor.rs` | Converts descriptor payloads into `KVS` for target plugin instances. |
-| `rustfs/src/admin/handlers/site_replication.rs` | Reads global server config for LDAP settings and parses LDAP `KVS` fixtures. |
-| `rustfs/src/admin/service/config.rs` | Reads persisted server config, validates storage-class `KVS`, derives target state, and updates global config/storage-class state. |
-| `rustfs/src/admin/router.rs` | Reads persisted/global server config for admin route behavior; route tests construct `Config`, `KV`, and `KVS`. |
-
-### Adjacent ECStore Config-Object Helper Users
-
-| Files | Current usage |
-|---|---|
-| `rustfs/src/admin/handlers/kms_dynamic.rs` | Uses generic `read_config` and `save_config` for dynamic KMS config objects. |
-| `rustfs/src/site_replication/state.rs` | Uses generic `read_config`, `save_config`, and `delete_config` (via the root storage facade) for site-replication state objects. |
-| `rustfs/src/admin/service/site_replication.rs` | Uses generic `read_config` and `save_config` for site-replication state normalization. |
-| `rustfs/src/server/module_switch.rs` | Uses generic `read_config` and `save_config` for module-switch config objects. |
-| `crates/iam/src/store/object.rs` | Uses generic `read_config_no_lock`, `read_config_with_metadata`, `save_config`, `save_config_with_opts`, and `delete_config` helper variants for IAM object-store persistence paths. |
-| `crates/scanner/src/{scanner,data_usage_define}.rs` | Uses generic `read_config` and `save_config` for scanner metadata and cache persistence paths. |
-
-### Runtime Target, Notify, And Audit Crates
-
-| Files | Current usage |
-|---|---|
-| `crates/notify/src/{global,integration,services,registry}.rs` | Carries `Config` into notification runtime startup/reload and target creation. |
-| `crates/notify/src/config_manager.rs` | Mutates `Config`, reads persisted server config with `read_config_without_migrate`, persists changes with `save_server_config`, and applies per-target `KVS` updates. |
-| `crates/notify/src/factory.rs` | Builds notification target arguments from `KVS`. |
-| `crates/notify/examples/{full_demo,full_demo_one}.rs` | Constructs `Config`, `KV`, and `KVS` directly for examples. |
-| `crates/audit/src/{global,system,registry}.rs` | Carries `Config` into audit runtime startup/reload and target creation. |
-| `crates/audit/src/factory.rs` | Builds audit target arguments from `KVS`. |
-| `crates/audit/tests/*.rs` | Constructs `Config` and `KVS` directly for runtime and parsing tests. |
-| `crates/audit/README.md` | Documents current direct `Config` usage. |
-| `crates/targets/src/plugin.rs` | Creates plugin targets from `Config` and merged `KVS`. |
-| `crates/targets/src/catalog/builtin.rs` | Declares builtin target descriptors and default `KVS` fields. |
-| `crates/targets/src/config/{common,target_args,loader,instance}.rs` | Collects, normalizes, redacts, and materializes target configs from `Config` and `KVS`, including environment overrides. |
-
-### Identity, Scanner, Tests, And Fixtures
-
-| Files | Current usage |
-|---|---|
-| `crates/iam/src/oidc.rs` | Reads global server config and parses OIDC provider `KVS`. |
-| `crates/scanner/src/{runtime_config,scanner}.rs` | Reads the global server-config snapshot and resolves scanner runtime config from `Config` and `KVS`. |
-| `rustfs/src/admin` handler/router tests, `crates/audit/tests/*.rs`, and selected in-crate tests in `crates/{targets,scanner}/src` | Build direct tuple-struct fixtures; use them as candidate regression guards during a pure model move. |
-
-## Dependency Risk Classification
-
-| Risk | Why it matters | Guardrail |
-|---|---|---|
-| `Config` is both persistence model and runtime input | A move can accidentally change persisted JSON/object shape or runtime target behavior. | Separate pure model contract from persistence helpers before moving `com.rs`. |
-| Direct `.0` map access is common | Replacing the tuple struct too early would create broad churn and likely behavior drift. | Preserve tuple shape in the first move, then add typed readers in later PRs. |
-| `KVS` is the effective target config carrier | Notify, audit, and target factories consume `KVS` after file/env merge. | Keep `KVS` API stable until target descriptor and runtime crates are behind a shared contract. |
-| `DEFAULT_KVS` registration is global | Defaults are initialized centrally and used by admin validation/rendering. | Add a registration contract before changing initialization order. |
-| Global snapshot readers still exist | Server, admin, IAM, scanner, and site-replication paths can still read global config. | Migrate readers through `AppContext`/provider paths in small steps after the model contract is stable. |
-| Persistence helpers depend on ECStore storage contracts | Moving them with the pure model would pull storage implementation dependencies upward. | Keep read/write helpers in `ecstore` until a storage-facing persistence contract is explicit. |
-
-## Recommended Migration Order
-
-1. Keep this inventory current while Phase 0 guardrails land.
-2. Add a focused contract surface for `KV`, `KVS`, and `Config` without changing
-   serialization, tuple-struct shape, or method names.
-3. Add compile-time or scripted checks for temporary compatibility markers and
-   config-model re-export coverage.
-4. Move only the pure model and defaults registration surface after targeted
-   regression checks cover unchanged persisted object shape, target `KVS` merge
-   behavior, and representative admin config rendering paths.
-5. Migrate global `Config` readers behind `ServerConfigInterface` or a narrower
-   provider in small PRs.
-6. Move persistence helpers only after object-I/O and storage-admin dependencies
-   can stay below the model contract.
-7. Evaluate crate split only after consumers no longer need old paths except
-   explicit `RUSTFS_COMPAT_TODO(<task-id>)` compatibility shims.
-
-## Do-Not-Change Contract
-
-The first migration steps must preserve:
-
-- `KV { key, value, hidden_if_empty }` serde behavior and redaction semantics.
-- `KVS(Vec<KV>)` tuple shape and public methods.
-- `Config(HashMap<String, HashMap<String, KVS>>)` tuple shape and public methods.
-- `Config::set_defaults`, `Config::unmarshal`, `Config::marshal`, and
-  `Config::merge` behavior.
-- `read_config_without_migrate` fallback/creation behavior for missing server
-  config objects.
-- `save_server_config` external object shape and config-history compatibility.
-- Existing notify, audit, scanner, OIDC, and target-plugin enable/disable
-  interpretation from `Config`/`KVS` inputs; business-rule changes stay out of
-  migration PRs.
-- AppContext/global fallback behavior until all readers are explicitly migrated.

docs/architecture/ecstore-layout-boundary.md

@@ -1,61 +1,39 @@
 # ECStore Layout Boundary
 
-This document records the `E-001` and `E-SET-001` foundation slice for the
-architecture migration.
+**Use this when:** you touch endpoint expansion, `FormatV3`, pool/set layout, or move files between ECStore's internal directories.
+**Source of truth:** `crates/ecstore/src/layout/` (static layout), `crates/ecstore/src/core/sets.rs` (`Sets`) and `crates/ecstore/src/set_disk/mod.rs` (`SetDisks`) for runtime orchestration, `crates/ecstore/src/api/mod.rs` (`pub mod layout`) for the public surface.
 
-## Directory Skeleton
+## Directory Ownership
 
-The ECStore migration uses these internal ownership buckets before any pure
-file moves:
+Ownership buckets under `crates/ecstore/src` (a subset; list the rest with `ls crates/ecstore/src`):
 
-- `api`: facade and compatibility re-export ownership.
-- `core`: store facade, object, bucket, list, multipart, and heal paths.
-- `layout`: static endpoint, disk, pool, and set layout descriptions.
-- `disk`: local disk, format, health, and disk error ownership.
-- `erasure`: erasure coding and bitrot ownership.
-- `metadata`: bucket metadata, config object-store, and data-usage ownership.
-- `cluster`: remote disk, peer, lock, membership, and health-control-plane
-  ownership.
-- `services`: lifecycle, replication, tier, notification, rebalance, and
-  metrics service ownership.
+| Directory | Owns |
+|---|---|
+| `api` | Facade and compatibility re-exports |
+| `core` | Store facade, pools, sets, and object/bucket/list/multipart/heal orchestration |
+| `layout` | Static endpoint, disk, pool, and set layout (`disks_layout`, `endpoint`, `endpoints`, `format`, `pool_space`, `set_heal`, `set_layout`) |
+| `disk` | Local disk, format compatibility, health, disk errors |
+| `erasure` | Erasure coding and bitrot |
+| `metadata` | Bucket metadata, config object store, data usage |
+| `cluster` | Remote disk, peer, lock, membership, health control plane |
+| `services` | Lifecycle, replication, tier, notification, rebalance, metrics services |
+| `set_disk`, `store`, `data_movement`, `data_usage`, `object_api`, `runtime` | Set-level operations, store init, pool data movement, usage accounting, object API helpers, runtime state owners |
 
 ## Static Set Layout
 
-Static layout is derived from persisted `FormatV3` data and input endpoint
-expansion. It may describe:
+Static layout is derived from persisted `FormatV3` data (`crates/ecstore/src/layout/format.rs`) and endpoint expansion (`crates/ecstore/src/layout/disks_layout.rs`). It may describe the deployment id, set count and drives per set, disk UUID positions inside `format.erasure.sets`, the distribution algorithm, and endpoint grouping produced before runtime disk initialization. It must not own disk handles, lock clients, reconnect loops, repair state, or shutdown signaling.
 
-- deployment id;
-- set count and drives per set;
-- disk UUID positions inside `format.erasure.sets`;
-- distribution algorithm;
-- endpoint grouping produced before runtime disk initialization.
+## Visibility
 
-Static layout must not own disk handles, lock clients, reconnect loops, repair
-state, or shutdown signaling.
-
-## Format And Disk Layout Ownership
-
-`layout::format` owns persisted format structures and disk UUID position lookup.
-`layout::disks_layout` owns command-line volume expansion into pool/set layout.
-
-Compatibility paths remain available through `disk::format` and `disks_layout`
-until downstream callers are moved or compatibility coverage allows removal.
+`layout::*` modules are `pub(crate)`; public access goes through `rustfs_ecstore::api::layout` (`DisksLayout`, `EndpointServerPools`, `Endpoints`, `PoolEndpoints`, `SetupType`). `disk::format` re-exports `layout::format` for crate-internal callers. Outer crates must not reach the root `endpoints` or `disks_layout` modules.
 
 ## Runtime Set Orchestration
 
-Runtime orchestration remains owned by `Sets` and `SetDisks` until a later pure
-move. It may describe:
-
-- flat disk index to `(set_index, disk_index)` mapping;
-- per-set local disk replacement after distributed setup detection;
-- per-set lock-client host deduplication;
-- endpoint reconnect monitoring and runtime shutdown signaling;
-- read/write/heal/list orchestration over initialized disks.
+`Sets` and `SetDisks` own the flat disk index to `(set_index, disk_index)` mapping, per-set local disk replacement after distributed setup detection, per-set lock-client host deduplication, endpoint reconnect monitoring and runtime shutdown signaling, and read/write/heal/list orchestration over initialized disks.
 
 ## Preservation Rules
 
 - Object-to-set hashing and distribution algorithm selection must not change.
 - Format `sets` ordering and disk UUID position lookup must not change.
 - Local disk replacement and lock-client mapping stay runtime-only.
-- Later file moves must keep old public paths or add explicit compatibility
-  coverage before deleting them.
+- File moves keep old public paths or add explicit compatibility coverage before deleting them.

docs/architecture/ecstore-module-split-plan.md

@@ -1,353 +1,85 @@
 # ECStore Module Split Plan
 
-This plan records the remaining ECStore split work after the final audit
-remediation pass. Runtime movement must still wait until each candidate
-boundary has explicit contracts, compatibility coverage, dependency evidence,
-and rollback steps.
+**Use this when:** you add lifecycle or replication logic and need to know which crate it belongs in, you plan to move an operation family out of `SetDisks`, or the guard fails on one of the split rules named below.
+**Source of truth:** `scripts/check_architecture_migration_rules.sh` (the rules), `crates/ecstore/src/bucket/lifecycle/README.md` and `crates/ecstore/src/bucket/replication/README.md` (module-level contract inventories, completion criteria, milestones), and [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md) (facade groups and boundary files).
 
 ## Current Shape
 
-| Area | Current owner | Size | Split status |
-|---|---|---:|---|
-| Bucket lifecycle | `crates/lifecycle/` + `crates/ecstore/src/bucket/lifecycle/` | core contracts + ECStore runtime | Core contract extracted |
-| Bucket replication | `crates/ecstore/src/bucket/replication/` | 15,619 lines | Contracts extracted; runtime move pending |
-| Set disks | `crates/ecstore/src/set_disk/` | state carrier plus operation modules | Keep in ECStore |
-| Public ECStore facade | `crates/ecstore/src/api/mod.rs` | broad compatibility surface | Shrink only through guarded PRs |
-| Embedded S3 client | `crates/s3-client/` (`rustfs-s3-client`) | ~8.4K lines | Extracted (rustfs/backlog#1842) |
+| Area | Owner | Split status |
+|---|---|---|
+| Bucket lifecycle | `crates/lifecycle/` (`rustfs-lifecycle`, pure contracts) + `crates/ecstore/src/bucket/lifecycle/` (runtime) | Core contracts extracted; runtime stays in ECStore |
+| Bucket replication | `crates/replication/` (`rustfs-replication`, contracts and wire formats) + `crates/ecstore/src/bucket/replication/` (worker runtime) | Contracts extracted; runtime move pending |
+| Set disks | `crates/ecstore/src/set_disk/` | Shared state carrier plus operation modules; stays in ECStore |
+| Public facade | `crates/ecstore/src/api/mod.rs` | Shrinks only through guarded changes |
+| S3 client | `crates/s3-client/` (`rustfs-s3-client`) | Extracted |
 
-Measured 2026-08-12: the whole crate is 265 files / ~288K lines (roughly half
-is inline `#[cfg(test)]` code). The largest single files are `disk/local.rs`
-(21,063 lines), `bucket/lifecycle/bucket_lifecycle_ops.rs` (11,961 lines), and
-`set_disk/mod.rs` (11,151 lines). Reproduce with:
+Measure size instead of trusting numbers in a document:
 
 ```bash
 find crates/ecstore/src -name '*.rs' | xargs wc -l | sort -rn | head
 find crates/ecstore/src/bucket/replication -name '*.rs' | xargs wc -l | tail -1
 ```
 
-No split step has landed since the contract-extraction PRs of 2026-07-04,
-while the `bucket/replication` runtime grew from 8,730 to 15,619 lines (+79%)
-through feature work (e.g. SSE-C ciphertext passthrough replication #5898,
-delete-marker purge retry/replay #5864). To keep the gap from widening: in
-domains that already have a contract crate, new replication runtime logic that
-does not need ECStore runtime state must land in `rustfs-replication`, not in
-`crates/ecstore/src/bucket/replication/`.
+Rule for new code: in a domain that already has a contract crate, new logic that does not need ECStore runtime state lands in that crate (`rustfs-lifecycle`, `rustfs-replication`), not under `crates/ecstore/src/bucket/`.
 
-The file split inside `set_disk/` is already operation-oriented: read, write,
-list, multipart, lock, heal, and replication code live in separate modules.
-The remaining large surface is the shared `SetDisks` state and cross-cutting
-contracts, not only file layout.
-
-## Completed: S3 Client Extraction (rustfs/backlog#1842)
-
-`crates/ecstore/src/client/` was a ~8.4K-line hand-written S3 HTTP client the engine uses to *consume* remote S3-compatible endpoints (ILM tier warm backends, transition targets). It was a legitimate engine capability misfiled inside the engine: it pulled `s3s`/`hyper` wire types into ecstore against ARCHITECTURE.md invariant 4, which distinguishes serving the S3 wire protocol (forbidden in ecstore) from consuming it (allowed, but in a dedicated crate).
-
-The extraction landed as: pure move of the 21 client modules to `crates/s3-client` (`rustfs-s3-client`) with a temporary re-export shim, then direct `rustfs_s3_client::` imports and shim deletion. The two server-side modules historically misfiled under `client/` stayed in ecstore and moved to their real homes: `object_api_utils.rs` under `object_api/`, `object_handlers_common.rs` under `bucket/lifecycle/` (behind the `replication_sink` boundary). The remaining serving-side `s3s` references in ecstore are ratcheted shrink-only by the `S3S_ECSTORE_FILES_BASELINE` counter in `scripts/check_s3s_footprint.sh`; per-module conversions to storage-level types (first: `bucket/object_lock/`) lower the baseline in the same change.
+The S3 client extraction is complete: the former `client/` directory moved to `crates/s3-client`, its two server-side modules moved to `crates/ecstore/src/object_api/object_api_utils.rs` and `crates/ecstore/src/bucket/lifecycle/object_handlers_common.rs`, and the remaining serving-side `s3s` references in ECStore are ratcheted shrink-only by `S3S_ECSTORE_FILES_BASELINE` in `scripts/check_s3s_footprint.sh`.
 
 ## Non-Negotiable Rules
 
-- Do not split crates in the same PR that moves runtime state or changes
-  startup behavior.
-- Do not change object placement, quorum, reader semantics, lifecycle queues,
-  replication queues, notification dispatch, audit events, or scanner repair
-  behavior during inventory and contract PRs.
-- Do not expose new direct ECStore internals to outer crates; use the existing
-  storage-api and owner-local facade boundaries.
-- Keep `rustfs_ecstore::api` compatibility visible until each consumer path has
-  compile coverage and an explicit replacement.
+- Do not split crates in the same change that moves runtime state or changes startup behavior.
+- Do not change object placement, quorum, reader semantics, lifecycle queues, replication queues, notification dispatch, audit events, or scanner repair behavior during inventory and contract work.
+- Do not expose new direct ECStore internals to outer crates; use storage-api and owner-local facade boundaries.
+- Keep `rustfs_ecstore::api` compatibility visible until each consumer path has compile coverage and an explicit replacement.
 
-## SetDisks Split Direction
+## Guarded Split Rules
 
-Do not replace `SetDisks` with several runtime structs in one change. The safe
-path is:
+Each rule is enforced by `scripts/check_architecture_migration_rules.sh`; the name is the vocabulary used in reviews and guard failures.
 
-1. Keep `SetDisks` as the shared state carrier while operation modules continue
-   to own read/write/list/multipart/lock/heal/replication behavior.
-2. Extract pure contracts first: shard source, disk error, bitrot IO, namespace
-   lock, metrics labels, and file metadata access.
-3. Move one operation family only after its contracts are covered by focused
-   tests and the facade compatibility path is explicit.
-4. Preserve the old `rustfs_ecstore::api::set_disk` surface until downstream
-   compatibility tests prove no caller depends on removed names.
+| Rule | What the guard checks |
+|---|---|
+| `LifecycleCrateCoreIndependence` | `crates/lifecycle` (rule validation, filtering, event evaluation, transition/expiration options, tag decoding, object-lock metadata checks, expiry-time rounding) imports no ECStore internals, `rustfs-filemeta`, or `rustfs-utils`; ECStore owns the `ObjectInfo` adapter in `crates/ecstore/src/bucket/lifecycle/core.rs`. |
+| `ReplicationCrateFileMetaIndependence` | Replication status, decision, MRF, resync, and target-reset wire contracts live in `crates/replication/src/filemeta.rs`; `rustfs-replication` neither imports nor depends on `rustfs-filemeta`. |
+| `ReplicationCrateStorageApiIndependence` | Delete work DTOs live in `crates/replication/src/storage_api.rs`; ECStore converts storage-api delete DTOs at its replication storage boundary; `rustfs-replication` does not depend on `rustfs-storage-api`. |
+| `ReplicationCrateUtilsIndependence` | HTTP metadata keys, S3 header labels, ETag trimming, and prefix matching used by replication wire contracts live in `crates/replication/src/http.rs`; `rustfs-replication` does not depend on `rustfs-utils`. |
+| `EcstoreReplicationBoundaryImports` | ECStore-side `rustfs_replication` imports are confined to the `*_boundary.rs` modules under `crates/ecstore/src/bucket/replication/`; grouped queue, stats, resync, and object-decision symbols each have one owning boundary file. |
+| `RuntimeReplicationFacadeConsumers` | Scanner, admin, storage-owner, and app code consume replication status/DTO/helper contracts through the `rustfs_ecstore` facade; the `rustfs` and `rustfs-scanner` crates do not depend on `rustfs-replication` directly. |
+| `StorageApiReplicationContracts` | Owner-facing storage-api delete DTO replication state/status helpers stay in `crates/storage-api/src/replication.rs`; replication worker DTOs stay in `rustfs-replication`. |
 
-The first executable SetDisks follow-up should be an inventory or guardrail PR,
-not a runtime split PR.
+## Lifecycle
 
-## Lifecycle Candidate
+`rustfs-lifecycle` owns the pure rule, event, evaluator, tag-filter, object-lock metadata check, and expiry-time contracts. ECStore keeps the object-store runtime, queues, tiering, audit/notification, metadata access (`crates/ecstore/src/bucket/lifecycle/metadata_boundary.rs`), and replication-delete scheduling adapters.
 
-`rustfs-lifecycle` now owns the pure lifecycle rule, event, evaluator, tag
-filtering, object-lock metadata check, and expiry-time contracts. ECStore keeps
-the object-store runtime, queues, tiering, audit/notification, metadata, and
-replication scheduling adapters.
+Coupling that still blocks a runtime move: lifecycle workers read ECStore runtime sources (object store, expiry and transition state, tier config, deployment id, local node name); stale multipart cleanup depends on `SetDisks` internals and bucket metadata; expiry schedules replication deletes through the replication lifecycle bridge; the lifecycle runtime coordinates scanner metrics and notification/audit side effects. The contract list and the next step live in `crates/ecstore/src/bucket/lifecycle/README.md`.
 
-Current coupling:
+## Replication
 
-- lifecycle workers and transition state read ECStore runtime sources for
-  object-store handles, expiry state, transition state, tier config, deployment
-  IDs, and local node names;
-- stale multipart cleanup depends on `SetDisks` internals and bucket metadata
-  through the lifecycle metadata boundary;
-- lifecycle expiry schedules bucket replication delete work through the
-  replication lifecycle bridge contract;
-- lifecycle evaluation uses S3 DTOs and replication status contracts from the
-  independent `rustfs-lifecycle`/`rustfs-replication` crates, while ECStore maps
-  `ObjectInfo` into lifecycle object options at the compatibility boundary;
-- lifecycle runtime still coordinates scanner metrics, notification/audit side
-  effects, metadata access, replication delete scheduling, and tier services.
+`rustfs-replication` owns resync status contracts, the persisted resync status wire format, filemeta-derived wire contracts, delete work DTOs, and HTTP helper contracts. ECStore keeps the worker runtime, error mapping, MRF persistence, and global pool/stat initialization.
 
-Current extracted contracts:
+Boundary layout inside `crates/ecstore/src/bucket/replication/`: `*_boundary.rs` modules concentrate imports from `rustfs-replication`, storage-api, filemeta, config, target, error, lock, msgp, versioning, tagging, bandwidth, queue, stats, resync, and object-decision surfaces; `replication_*_bridge.rs` modules (lifecycle, scanner, object, migration, target-config) expose replication scheduling to other owners without leaking DTO construction; `replication_config_store.rs` exposes config persistence and storage-class labels. Modules inside the directory use relative self-imports, and the facade in `mod.rs` uses explicit symbol lists, never wildcard re-exports.
 
-- `LifecycleCrateCoreIndependence`: lifecycle rule validation, filtering,
-  event evaluation, transition/expiration options, tag decoding, object-lock
-  metadata checks, and ILM expiry-time rounding live in `rustfs-lifecycle`.
-  `rustfs-lifecycle` must not import ECStore internals, file metadata, or
-  `rustfs-utils`; ECStore owns the `ObjectInfo` adapter in
-  `crates/ecstore/src/bucket/lifecycle/core.rs`.
+Consumers outside ECStore: RustFS runtime code receives pool/stat handles through storage-owner wrapper types in `rustfs/src/storage/storage_api.rs`; scanner code receives scanner-local config/admission/heal DTOs from `crates/scanner/src/storage_api.rs`; observability reads replication metrics through obs-local snapshot DTOs in `crates/obs/src/metrics/storage_api.rs`; app object and multipart writes call object-replication bridge helpers instead of constructing replication work DTOs.
 
-Required contracts before crate movement:
+Completion criteria, the milestone order, and the per-dependency contract inventory live in `crates/ecstore/src/bucket/replication/README.md` (sections "Completion Criteria" and "Milestones"). Remaining work starts from moving resyncer pure decision logic.
 
-- `LifecycleObjectStore`: object stat, delete, transition, restore, multipart
-  cleanup, and version-aware metadata operations needed by lifecycle workers.
-- `LifecycleMetadataStore`: lifecycle, object-lock, replication, bucket
-  versioning, and stale multipart metadata lookups without importing ECStore
-  implementation modules. Current lifecycle config reads are concentrated in
-  `crates/ecstore/src/bucket/lifecycle/metadata_boundary.rs`.
-- `LifecycleRuntime`: expiry state, transition state, tier config, deployment
-  ID, local node name, queue metrics, cancellation, and worker sizing.
-- `LifecycleReplicationSink`: schedule lifecycle-originated replication deletes
-  without depending on the replication implementation module.
-- `LifecycleAuditSink`: lifecycle audit and notification emission boundary.
+## SetDisks
 
-Next safe PR:
+Do not replace `SetDisks` with several runtime structs in one change:
 
-- move one runtime-facing dependency behind a trait or adapter owned by
-  `rustfs-lifecycle` without changing queue, transition, or delete behavior;
-- keep ECStore compatibility shims until scanner and RustFS app consumers stop
-  depending on `rustfs_ecstore::api::bucket::lifecycle` paths;
-- add focused tests for the moved contract and keep architecture guard coverage.
+1. Keep `SetDisks` as the shared state carrier while operation modules own read/write/list/multipart/lock/heal/replication behavior.
+2. Extract pure contracts first: shard source, disk error, bitrot IO, namespace lock, metrics labels, and file metadata access.
+3. Move one operation family only after its contracts are covered by focused tests and the facade compatibility path is explicit.
+4. Preserve the `rustfs_ecstore::api::set_disk` surface until downstream compatibility tests prove no caller depends on removed names.
 
-The module-level inventory lives in
-`crates/ecstore/src/bucket/lifecycle/README.md`.
+## Facade Shrink
 
-Focused verification for the first code-bearing lifecycle PR:
-
-- `cargo test -p rustfs-ecstore lifecycle --lib`
-- `cargo check -p rustfs-ecstore --tests`
-- `./scripts/check_architecture_migration_rules.sh`
-- `git diff --check`
-
-## Replication Candidate
-
-`rustfs-replication` now owns the resync status contracts and persisted resync
-status wire format. The remaining `bucket/replication` worker runtime is not
-ready for a full standalone crate yet.
-
-The completion criteria and milestone sequence for this candidate (when the
-split counts as done, the target end state, and the order of the remaining
-moves) live in the module inventory:
-`crates/ecstore/src/bucket/replication/README.md`, sections "Completion
-Criteria" and "Milestones". The originally proposed first code-bearing step
-(event sink / runtime contracts) has landed; remaining work starts from moving
-resyncer pure decision logic.
-
-Current coupling:
-
-- replication workers depend on `ReplicationStorage`, ECStore object APIs and
-  owner storage-api contracts through the replication storage boundary, bucket
-  target clients, bucket metadata, file metadata replication state through the
-  filemeta boundary, config-derived storage class labels through the config store, scanner repair
-  classification, runtime replication pool/stat handles, bucket monitor and
-  bandwidth reader access through local boundaries, local node names, and
-  notification events;
-- resync and delete replication paths call metadata paths through the metadata
-  boundary, while bucket target system access, target config types, and target
-  operation types are concentrated behind the replication target boundary;
-- lifecycle delete paths schedule replication work through
-  `ReplicationLifecycleBridge`, while scanner heal paths schedule replication
-  work through `ReplicationScannerBridge`, and app/SetDisks object write/delete
-  paths use `ReplicationObjectBridge`;
-- bucket metadata migration and bucket target removal checks use local
-  replication bridges instead of importing resyncer codec or config helper
-  internals;
-- resync options, bucket/target resync status DTOs, status display labels, and
-  the persisted resync status wire format live in `crates/replication`, with
-  ECStore retaining only error mapping and MRF persistence locally;
-- `ReplicationCrateFileMetaIndependence`: replication status, decision, MRF,
-  resync, and target-reset wire contracts are owned inside `rustfs-replication`
-  instead of importing `rustfs-filemeta`;
-- `ReplicationCrateStorageApiIndependence`: delete work DTOs are owned inside
-  `rustfs-replication`; ECStore converts storage-api delete DTOs at the
-  replication storage boundary instead of `rustfs-replication` importing
-  `rustfs-storage-api`;
-- `ReplicationCrateUtilsIndependence`: HTTP metadata keys, S3 header labels,
-  ETag trimming, and case-insensitive prefix matching used by replication wire
-  contracts are owned inside `rustfs-replication` instead of importing
-  `rustfs-utils`;
-- direct ECStore replication imports from `rustfs-replication` are limited to
-  `*_boundary.rs` modules;
-- storage-api delete replication status/state helpers use the local
-  `crates/storage-api/src/replication.rs` contract boundary; ECStore converts
-  those owner DTOs at the replication storage boundary before queueing work;
-- admin replication extension target filtering and resync request construction
-  stay behind the admin storage boundary instead of exposing replication work
-  DTO construction to handlers;
-- scanner, admin, storage-owner, and app storage replication status/DTO/helper
-  consumers import those contracts through the ECStore replication facade;
-- app object and multipart writes call object-replication boundary helpers
-  instead of constructing replication work DTOs or choosing object replication
-  operation types at the use-case layer;
-- RustFS runtime consumers receive replication pool/stat handles through
-  storage-owner wrapper types instead of carrying ECStore replication handles
-  through app, admin, startup, or workload-admission layers;
-- global replication pool/stat initialization still lives with ECStore runtime
-  compatibility state;
-- modules inside `bucket/replication` use local relative paths rather than the
-  ECStore owner path for replication self-imports;
-- replication runtime source access uses storage/bandwidth boundary aliases for
-  ECStore object store and bucket monitor implementation types;
-- the ECStore replication facade in `mod.rs` uses explicit compatibility
-  exports instead of wildcard re-exports from implementation modules.
-
-Required contracts before crate movement:
-
-- `ReplicationObjectIO`: object read/write primitives for config, MRF, resync
-  status, and multipart replication paths. ECStore object API reader/writer
-  types and storage-api object IO contracts are concentrated in
-  `crates/ecstore/src/bucket/replication/replication_storage_boundary.rs`.
-- `ReplicationStorage`: keep the existing trait as the starting point, then
-  split object read/write/delete, walk, and metadata update responsibilities
-  only when call sites prove a narrower shape. ECStore object API,
-  storage-api contracts, and read option types are concentrated in
-  `crates/ecstore/src/bucket/replication/replication_storage_boundary.rs`.
-- `ReplicationMetadataStore`: replication config, target reset headers,
-  MRF/resync state, and status persistence. Metadata sys access and replication
-  metadata path constants are exposed through the contract type in
-  `crates/ecstore/src/bucket/replication/replication_metadata_boundary.rs`.
-- `ReplicationConfigStore`: replication config persistence and config-derived
-  labels used by target options. Config read/save helpers and storage class
-  labels are exposed through the contract type in
-  `crates/ecstore/src/bucket/replication/replication_config_store.rs`.
-- `ReplicationFileMeta`: replication status, decisions, MRF entries, resync
-  decisions, and target reset helpers. ECStore concentrates filemeta-to-
-  replication compatibility conversions in
-  `crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs`,
-  while `FileInfo` remains in the storage boundary for storage trait bindings
-  and walk options.
-- `ReplicationCrateFileMetaIndependence`: filemeta wire contracts consumed by
-  replication workers are owned in `crates/replication/src/filemeta.rs`, and
-  `rustfs-replication` must not import or depend on `rustfs-filemeta`.
-- `ReplicationCrateStorageApiIndependence`: delete work DTOs consumed by
-  replication delete/queue/operation helpers are owned in
-  `crates/replication/src/storage_api.rs`, and `rustfs-replication` must not
-  import or depend on `rustfs-storage-api`.
-- `ReplicationCrateUtilsIndependence`: replication-specific HTTP metadata,
-  header, ETag, and prefix helper contracts are owned in
-  `crates/replication/src/http.rs`, and `rustfs-replication` must not import or
-  depend on `rustfs-utils`.
-- `EcstoreReplicationBoundaryImports`: ECStore-side imports from
-  `rustfs-replication` are concentrated in replication `*_boundary.rs` modules.
-- `RuntimeReplicationFacadeConsumers`: scanner, admin, storage-owner, and app
-  storage replication status/DTO/helper consumers import through
-  `rustfs-ecstore`; runtime code under `rustfs/src` does not import
-  `rustfs-replication` directly, and the RustFS runtime/scanner crates do not
-  depend on it.
-- `StorageApiReplicationContracts`: owner-facing storage-api delete DTO
-  replication state/status helpers remain concentrated in
-  `crates/storage-api/src/replication.rs`, while replication worker DTOs live in
-  `rustfs-replication`.
-- `ReplicationErrorBoundary`: ECStore error/result contracts and
-  replication-specific error classifiers. `crate::error` imports are
-  concentrated in
-  `crates/ecstore/src/bucket/replication/replication_error_boundary.rs`.
-- `ReplicationTargetStore`: bucket target listing, target client lookup,
-  target offline checks, target config types, and target operation option
-  types. Bucket target sys access, `BucketTargets`, and target operation types
-  are exposed through the contract type in
-  `crates/ecstore/src/bucket/replication/replication_target_boundary.rs`.
-- `ReplicationRuntime`: pool, stats, worker admission, bucket monitor, local
-  node identity, cancellation, and queue sizing. Concrete ECStore object store
-  and bucket monitor types stay behind local storage/bandwidth boundaries.
-- `ReplicationBandwidthLimiter`: target reader wrapping for replication
-  bandwidth accounting and throttling.
-- `ReplicationVersioningStore`, `ReplicationLockTiming`, `ReplicationMsgpCodec`,
-  and `ReplicationTagFilter`: smaller state/codec/filter contracts that keep
-  bucket versioning, SetDisks lock timing, MessagePack helpers, and bucket
-  tagging helper access behind local replication boundary types.
-- `ReplicationEventSink`: notification/audit events for skipped, failed, and
-  completed replication operations, including local event host selection.
-- `ReplicationLifecycleBridge`: lifecycle-originated delete and version-purge
-  scheduling is exposed through the contract type in
-  `crates/ecstore/src/bucket/replication/replication_lifecycle_bridge.rs`.
-- `ReplicationMigrationBridge`: persisted resync status decode/encode access
-  for bucket metadata migration is exposed through the contract type in
-  `crates/ecstore/src/bucket/replication/replication_migration_bridge.rs`.
-- `ReplicationResyncContracts`: resync options, target/bucket resync status,
-  status labels, and persisted status encoding live in `crates/replication`.
-- `ReplicationObjectBridge`: app and SetDisks object write/delete replication
-  decisions and scheduling are exposed through the contract type in
-  `crates/ecstore/src/bucket/replication/replication_object_bridge.rs`.
-- `ObsReplicationStatsSnapshot`: observability reads replication bucket/site
-  metrics through obs-local snapshot DTOs in
-  `crates/obs/src/metrics/storage_api.rs` instead of carrying the ECStore
-  replication stats handle through collectors.
-- `StorageReplicationPoolHandle` / `StorageReplicationStatsHandle`: RustFS app, admin,
-  startup, and workload-admission code use storage-owner wrapper types from
-  `rustfs/src/storage/storage_api.rs` for pool activity, resync, queue counts,
-  proxy stats, and site metrics snapshots.
-- `ReplicationScannerBridge`: scanner-originated replication heal scheduling is
-  exposed through the contract type in
-  `crates/ecstore/src/bucket/replication/replication_scanner_bridge.rs`.
-  Scanner consumers receive scanner-local replication config/admission/heal
-  object DTOs from `crates/scanner/src/storage_api.rs` instead of constructing
-  or inspecting replication queue DTOs directly.
-- `ReplicationTargetConfigBridge`: bucket target removal checks against
-  replication target rules are exposed through the contract type in
-  `crates/ecstore/src/bucket/replication/replication_target_config_bridge.rs`.
-- `ReplicationFacade`: the current `rustfs_ecstore::api::bucket::replication`
-  compatibility surface is an explicit symbol list guarded against wildcard
-  re-exports while downstream owners migrate to narrower contracts.
-
-First safe PR:
-
-- add a replication extraction inventory section or module-level README;
-- list current ECStore/runtime dependencies and the target contract owner for
-  each dependency;
-- keep global pool/stat initialization and queue behavior unchanged.
-
-The module-level inventory lives in
-`crates/ecstore/src/bucket/replication/README.md`.
-
-Focused verification for the first code-bearing replication PR:
-
-- `cargo test -p rustfs-ecstore replication --lib`
-- `cargo check -p rustfs-ecstore --tests`
-- `./scripts/check_architecture_migration_rules.sh`
-- `git diff --check`
-
-## Facade Shrink Plan
-
-The broad `rustfs_ecstore::api` facade remains a compatibility boundary, not a
-new architecture target. The current facade groups and external consumers are
-recorded in
-[`ecstore-api-facade-inventory.md`](ecstore-api-facade-inventory.md).
-Shrinking it must be monotonic:
-
-1. Inventory every public facade group and consumer.
-2. Add compile-time coverage before removing or narrowing a facade item.
-3. Move outer consumers to storage-api or owner-local compatibility boundaries.
-4. Remove one facade group per PR only after downstream compatibility tests pass.
-
-Do not delete facade groups only because the underlying module moved. Keep the
-facade stable until the replacement path is visible and tested.
+Facade groups, boundary files, and shrink rules are in [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md). Shrinking is monotonic: inventory, add compile-time coverage, move consumers to storage-api or owner-local boundaries, then remove one group per change. Do not delete facade groups only because the underlying module moved.
 
 ## Ready-To-Split Checklist
 
-A candidate split is ready for code movement only when all items below are true:
+A candidate is ready for code movement only when all of these hold:
 
-- dependency graph shows no cycle with ECStore, storage-api, runtime sources, or
-  owner-local compatibility modules;
+- the dependency graph shows no cycle with ECStore, storage-api, runtime sources, or owner-local compatibility modules;
 - contract traits compile without importing ECStore implementation modules;
 - old facade names have compatibility tests or explicit deprecation coverage;
 - focused tests cover the changed owner path before any full gate is attempted;
-- rollback preserves object IO, quorum, lifecycle/replication queues, scanner
-  repair, notification/audit events, and metadata compatibility.
+- rollback preserves object IO, quorum, lifecycle/replication queues, scanner repair, notification/audit events, and metadata compatibility.

docs/architecture/erasure-coding.md

@@ -1,10 +1,13 @@
 # Erasure Coding — Normative Algorithm & On-Disk Compatibility Contract
 
+**Use this when:** changing anything under `crates/ecstore/src/erasure/`, `crates/filemeta/`, `crates/ecstore/src/set_disk/`, storage-class or layout code, or any decode, quorum, or heal boundary; read §12 and §13 before editing.
+**Source of truth:** this document is normative for the algorithm and the on-disk / on-wire compatibility contract; the cited symbols are where the code enforces each rule.
+
 Status: normative. This document is the source of truth for how RustFS erasure-codes, stores, reads, reconstructs, and heals user data, and for the on-disk / on-wire compatibility contract that every future change must preserve. It governs the highest-risk code in the system: a regression here can silently corrupt or lose all user data, or make existing (and MinIO-migrated) objects permanently unreadable.
 
-Erasure coding, quorum/heal, and metadata/on-disk formats are **High-risk** per [AGENTS.md](../../AGENTS.md) ("Risk tiers"). Any behavior-affecting change to code this document governs requires the full seven-role adversarial validation and, for anything touching decode or the on-disk format, a regression test against real on-disk and MinIO-migrated samples before merge.
+Erasure coding, quorum/heal, and metadata/on-disk formats are **High-risk** per [AGENTS.md](../../AGENTS.md) ("Broad or High-Risk Changes"). Any behavior-affecting change to code this document governs requires adversarial review with the `adversarial-validation` skill and, for anything touching decode or the on-disk format, a regression test against real on-disk and MinIO-migrated samples before merge.
 
-This document describes the baseline (`main`) algorithm. Where the baseline has a known defect that a specific change corrects, that is called out inline; the *invariant* stated is always the correct rule the code must converge to, never the defect.
+This document describes the algorithm as implemented on `main`. The *invariant* stated is always the rule the code must satisfy; where the code enforces it, the enforcing symbol is cited.
 
 ## How to use this document
 
@@ -34,7 +37,6 @@ This document describes the baseline (`main`) algorithm. Where the baseline has
 11. Compatibility contract and decode tolerance
 12. Invariants checklist (the frozen contract)
 13. Change procedure and guardrails
-14. References
 
 ---
 
@@ -80,7 +82,7 @@ Two storage classes: `STANDARD` (SC) and `REDUCED_REDUNDANCY` (RRS) ([storagecla
 
 - **INVARIANT — parity bounds.** Parity must satisfy `parity ≤ N/2` for both classes, and `SC parity ≥ RRS parity` when both are non-zero ([storageclass.rs](../../crates/ecstore/src/config/storageclass.rs), `validate_parity` / `validate_parity_inner`). Enforcement nuance to be aware of: `validate_parity_inner` (the path a user-configured `EC:<parity>` storage class flows through) only applies the `parity ≤ N/2` check for `N > 2`, so degenerate small-set values (e.g. `EC:2` on `N = 2`, giving `data_blocks = 0`) are not caught there; the standalone `validate_parity` enforces the bound unconditionally but is applied only to the resolved default parity. A change that lets user-configured parity reach a write path must not assume the `≤ N/2` bound was enforced for `N ≤ 2`. Parity `0` is permitted (single-drive / capacity setups); there is no non-zero minimum.
 - **INVARIANT — per-pool validity.** Each pool's resolved parity must be valid for **that pool's own drive count**. A heterogeneous deployment (pools of different widths) must resolve parity per pool; applying one pool's parity to a narrower pool can drive `data_blocks = N − parity` to `0` and make encoding impossible.
-  - Baseline defect: `main` computes `common_parity_drives` from the **first** pool only and applies it to every pool ([store/init.rs](../../crates/ecstore/src/store/init.rs), `ec_drives_no_config` at [store/init_format.rs](../../crates/ecstore/src/store/init_format.rs)); this is issue #4801 (a smaller later pool panics with `TooFewDataShards`). The correct rule is per-pool resolution.
+  - Implemented by `resolve_write_layout` ([set_disk/mod.rs](../../crates/ecstore/src/set_disk/mod.rs)), which takes the pool index and resolves parity against that pool's own drive count; the default parity for a pool without explicit config comes from `ec_drives_no_config` ([store/init_format.rs](../../crates/ecstore/src/store/init_format.rs)).
 
 Per-write layout (the numbers that go into `xl.meta`), from the storage class or `default_parity_count`, with `opts.max_parity` forcing `N/2` for internal writes ([set_disk/ops/object.rs](../../crates/ecstore/src/set_disk/ops/object.rs)):
 
@@ -219,7 +221,7 @@ Fields: `version_id`, `mod_time`, `signature: [u8;4]`, `version_type`, `flags: u
 
 ### 6.6 Inline data
 
-Small objects store their payload inline after the container CRC ([filemeta_inline.rs](../../crates/filemeta/src/filemeta_inline.rs)): 1 version byte (`INLINE_DATA_VER = 1`) then a msgpack map of `version-key → bin`. **INVARIANT — the map key** is the version-id string, `"null"` (`NULL_VERSION_ID`) for the null/None version, else the lowercase hyphenated UUID. Presence is determined **on read** solely by the `meta_sys[inline-data]` body marker (`FileInfo::inline_data`); the read path gates inline extraction on that marker alone. The header `InlineData` flag is **written** (mirrored from the body on marshal) but is **not** consulted on read, and a disagreement is tolerated — MinIO may leave the header flag unset while inline data is present, so a reader must **not** require the flag and the marker to agree. The inline threshold is `should_inline` ([storageclass.rs](../../crates/ecstore/src/config/storageclass.rs)): inline if `shard_size ≤ inline_block/8` for versioned buckets, else `≤ inline_block`; `DEFAULT_INLINE_BLOCK = 128 KiB`.
+Small objects store their payload inline after the container CRC ([filemeta_inline.rs](../../crates/filemeta/src/filemeta_inline.rs)): 1 version byte (`INLINE_DATA_VER = 1`) then a msgpack map of `version-key → bin`. **INVARIANT — the map key** is the version-id string, `"null"` (`NULL_VERSION_ID`, [fileinfo.rs](../../crates/filemeta/src/fileinfo.rs)) for the null/None version, else the lowercase hyphenated UUID. Presence is determined **on read** solely by the `meta_sys[inline-data]` body marker (`FileInfo::inline_data`); the read path gates inline extraction on that marker alone. The header `InlineData` flag is **written** (mirrored from the body on marshal) but is **not** consulted on read, and a disagreement is tolerated — MinIO may leave the header flag unset while inline data is present, so a reader must **not** require the flag and the marker to agree. The inline threshold is `should_inline` ([storageclass.rs](../../crates/ecstore/src/config/storageclass.rs)): inline if `shard_size ≤ inline_block/8` for versioned buckets, else `≤ inline_block`; `DEFAULT_INLINE_BLOCK = 128 KiB`.
 
 ---
 
@@ -230,7 +232,7 @@ Small objects store their payload inline after the container CRC ([filemeta_inli
 - Encode-time gates: writable disks `< write_quorum` ⇒ `ErasureWriteQuorum`; committed shards `< write_quorum` after encode ⇒ error ([set_disk/ops/object.rs](../../crates/ecstore/src/set_disk/ops/object.rs)).
 - **INVARIANT — atomic commit with best-effort rollback.** Commit is `rename_data` (per-disk temp → final) fanned across all disks ([core/io_primitives.rs](../../crates/ecstore/src/set_disk/core/io_primitives.rs)). If write quorum is not met (`reduce_write_quorum_errs`), every successful disk is undone (`delete_version{undo_write:true}`) and the original quorum error is returned. **Baseline:** the rollback is **best-effort** — undo failures are counted and `warn!`-logged, never propagated or retried — so a write that both misses quorum *and* whose rollback partially fails can leave shards on some disks; that partial residue is reconciled later by heal/scanner, not by the commit path. The guarantee the commit path enforces is "never *reports* success below quorum", not "never leaves any bytes behind".
 - On success the newly committed dir is `fi.data_dir`. Separately, `reduce_common_data_dir` votes over each disk's **`old_data_dir`** (the *superseded* dir being dereferenced) and returns it when it reaches write_quorum, so the old dir can be reclaimed (`commit_rename_data_dir`) — it is a GC input, **not** the new `data_dir`. `classify_rename_convergence` classifies the commit (`PartialCommit` / `SignatureDivergent`), but **only the multipart-complete path consumes it** (`convergence.needs_heal()` → `send_heal_request`); the regular `put_object` path discards the convergence result and relies on the old-data-dir cleanup / `add_partial` heal enqueue instead.
-- The write layout (per-pool parity, storage class, `max_parity`) is computed **inline** in the write path ([set_disk/ops/object.rs](../../crates/ecstore/src/set_disk/ops/object.rs)); on `main` there is **no** `WriteLayout` type or `resolve_write_layout` function — do not cite either as if it exists (§13's symbol-citation rule). A future refactor may centralize this; add the symbol to the spec only once it lands in code.
+- The write layout (per-pool parity, storage class, `max_parity`) is resolved once by `resolve_write_layout` into a `WriteLayout` ([set_disk/mod.rs](../../crates/ecstore/src/set_disk/mod.rs)) and consumed by the object and multipart write paths ([set_disk/ops/object.rs](../../crates/ecstore/src/set_disk/ops/object.rs), [set_disk/ops/multipart.rs](../../crates/ecstore/src/set_disk/ops/multipart.rs)); resolution is per pool (§2.2).
 
 ---
 
@@ -238,7 +240,7 @@ Small objects store their payload inline after the container CRC ([filemeta_inli
 
 - **INVARIANT — read quorum = `data_blocks`.** `object_quorum_from_meta` returns `(read_quorum = data_blocks, write_quorum)` ([set_disk/metadata.rs](../../crates/ecstore/src/set_disk/metadata.rs)); `parity_blocks = common_parity(...)` is the parity value held by the most disks that still reaches its own read quorum. When `default_parity_count == 0`, read = write = all shards.
 - Authoritative FileInfo selection — `find_file_info_in_quorum` ([set_disk/metadata.rs](../../crates/ecstore/src/set_disk/metadata.rs)) groups valid metas by a content-identity SHA-256 (`file_info_quorum_hash`) that hashes size/flags/mod_time/transition/version_id/data_dir/parts and, for real objects, data/parity/distribution — **excluding replication-status keys** so replication noise never splits quorum. A meta counts only if its mod_time equals the common mod_time (or etag matches when mod_time is absent). The winning hash must reach quorum, else `ErasureReadQuorum`. Latest-version reads may escalate to write_quorum to avoid resurrecting a partially-overwritten version.
-- **INVARIANT — decode needs ≥ `data_blocks` shards.** The stripe reader requires `available_shards ≥ data_shards`; below that the read fails closed with a read-quorum error (never silent truncation) ([set_disk/read.rs](../../crates/ecstore/src/set_disk/read.rs), [set_disk/shard_source.rs](../../crates/ecstore/src/set_disk/shard_source.rs)). Before any `block_size` / `data_shards` division, `has_valid_dimensions()` must hold (`block_size > 0 && data_shards > 0`) or the read fails instead of dividing by zero ([erasure.rs](../../crates/ecstore/src/erasure/coding/erasure.rs)); note this guard runs *after* codec construction and fully covers only `block_size == 0` — a `data_blocks == 0` geometry panics earlier in the constructor (§13).
+- **INVARIANT — decode needs ≥ `data_blocks` shards.** The stripe reader requires `available_shards ≥ data_shards`; below that the read fails closed with a read-quorum error (never silent truncation) ([set_disk/read.rs](../../crates/ecstore/src/set_disk/read.rs), [set_disk/shard_source.rs](../../crates/ecstore/src/set_disk/shard_source.rs)). Codec geometry is validated at construction: `Erasure::try_new` / `try_new_with_options` ([erasure.rs](../../crates/ecstore/src/erasure/coding/erasure.rs)) return `ErasureConstructionError` for `data_shards == 0`, `block_size == 0`, shard-count overflow, or an unsupported shard configuration, so a read never reaches a `block_size` / `data_shards` division with invalid geometry (§13).
 - If `available ≥ data_blocks` but some shards are missing, the read is served **and** a background read-repair heal is enqueued.
 - **INVARIANT — cross-stripe read verification.** When a data shard is missing and `available > data_blocks`, reconstruction regenerates parity and compares it to the surviving parity; a mismatch is `InvalidData "inconsistent read source shards"` (backlog#832), catching corruption that passed per-shard bitrot but disagrees across the stripe ([erasure.rs](../../crates/ecstore/src/erasure/coding/erasure.rs)).
 
@@ -324,22 +326,12 @@ Decode tolerance
 
 ## 13. Change procedure and guardrails
 
-- **Risk tier.** All of the above is High-risk ([AGENTS.md](../../AGENTS.md)). Any behavior-affecting change requires the full seven-role adversarial validation.
+- **Risk tier.** All of the above is High-risk ([AGENTS.md](../../AGENTS.md) "Broad or High-Risk Changes"). Any behavior-affecting change requires adversarial review with the `adversarial-validation` skill.
 - **Keep this document in sync.** A change to any governed behavior, formula, format field, or invariant must update this spec in the same PR; renaming a cited symbol must update its reference here. The spec is normative and is the checklist the next change is reviewed against, so drift is a correctness defect. References are symbol-based (not line numbers) specifically so ordinary refactors do not invalidate them — but semantic changes still must.
 - **Adding an on-disk field** must be additive: new msgpack key or a `minor`/`meta_ver` bump with a read path for the old value; keep decoders skipping unknown keys; write both internal-key prefixes; never repurpose or reorder existing keys or header array positions.
 - **Never make a decode boundary stricter** than what §11 allows without (a) proving no legitimate older-RustFS or MinIO-migrated shape is rejected, and (b) a regression test against real on-disk and MinIO fixtures. New validation belongs at the trust boundary and must fail *open to a tolerant default*, not closed to `FileCorrupt`, for anything recoverable. (Concretely: rejecting a negative `actual_size`, or hard-failing a non-16-byte `transitioned-versionID`, breaks existing data — see §11.)
-- **Codec construction (baseline gap).** The baseline exposes panicking `Erasure::new` / `new_with_options`: the codec's shard-count validation surfaces as an `.expect` panic when `data_shards == 0 && parity_shards > 0` (`ReedSolomon::new` ⇒ `TooFewDataShards`). `has_valid_dimensions()` (`block_size > 0 && data_shards > 0`) is a **`&self`** method, so it can only run *after* construction — the read path builds the codec from on-disk geometry first and checks the guard second ([erasure.rs](../../crates/ecstore/src/erasure/coding/erasure.rs), [set_disk/read.rs](../../crates/ecstore/src/set_disk/read.rs)). It therefore reliably catches only the `block_size == 0` case (block size is never passed to `ReedSolomon::new`, so construction succeeds and the guard rejects it before any division); a `data_blocks == 0` xl.meta with `parity > 0` **panics in the constructor before the guard can run**. The correct fix is a **fallible constructor** (returning `Result`, not `.expect`) on any path reachable from untrusted metadata; until then `has_valid_dimensions()` is a partial preflight, not a complete guard.
+- **Codec construction is fallible.** Read, heal, multipart, and object write paths construct the codec through `Erasure::try_new` / `try_new_with_options` ([erasure.rs](../../crates/ecstore/src/erasure/coding/erasure.rs)) and surface `ErasureConstructionError` instead of panicking on geometry decoded from untrusted metadata (`data_shards == 0`, `block_size == 0`, shard-count overflow, unsupported shard counts). Do not reintroduce a panicking constructor on any path reachable from on-disk metadata; `has_valid_dimensions()` remains only a `&self` preflight for already-built codecs.
 - **Guardrail scripts** (part of `make pre-commit` / `make pre-pr`):
   - [check_architecture_migration_rules.sh](../../scripts/check_architecture_migration_rules.sh) keeps the erasure engine crate-private and under its owner module, and keeps erasure-cache / `GLOBAL_IS_ERASURE*` access behind ecstore helpers.
   - [check_doc_paths.sh](../../scripts/check_doc_paths.sh) validates that every repo path this document cites exists — keep citations to real paths.
 - **Tooling.** Inspect on-disk metadata with `dump_fileinfo` / `dump_versions` per [../operations/tier-ilm-debugging.md](../operations/tier-ilm-debugging.md) rather than guessing at bytes.
-
----
-
-## 14. References
-
-- Reed–Solomon codes; MDS property and GF(2⁸) byte-oriented coding — the standard basis for `rs-vandermonde` (Vandermonde generator matrix over GF(2⁸)).
-- `rustfs-erasure-codec` (RustFS fork of `reed-solomon-erasure`, GF(2⁸)) and `reed-solomon-simd` (GF(2¹⁶)) — declared in the workspace `Cargo.toml`.
-- HighwayHash-256 — the bitrot checksum family; π-derived default key.
-- MinIO `xl.meta` v1.3 format lineage — RustFS is byte-compatible for read + one-way migration; see [minio-file-format-compat.md](minio-file-format-compat.md) for the fixture-proven matrix and scope.
-- Related invariants: [placement-repair-invariants.md](placement-repair-invariants.md), [ecstore-layout-boundary.md](ecstore-layout-boundary.md), [decommission-compatibility.md](decommission-compatibility.md), [../operations/tier-ilm-debugging.md](../operations/tier-ilm-debugging.md), and [AGENTS.md](../../AGENTS.md) Cross-Cutting Domain Invariants.

docs/architecture/global-state-crate-split-plan.md

@@ -1,202 +1,67 @@
 # Global State And Crate Split Plan
 
-This document records the late global-state cleanup plan after the AppContext
-foundation, storage API contracts, ECStore layout, runtime lifecycle, and cluster
-control-plane boundaries are stable.
+**Use this when:** business logic needs runtime state (object store, endpoints, lock clients, lifecycle state, config) and you must pick the right boundary, or you are evaluating a new crate split out of ECStore.
+**Source of truth:** `crates/ecstore/src/runtime/global.rs` and `crates/ecstore/src/runtime/sources.rs` (ECStore-owned state and its adapter), `rustfs/src/app/context.rs` and the `runtime_sources.rs` owner modules under `rustfs/src` (RustFS resolvers), and the `rustfs_ecstore::api::global` boundary list in `scripts/check_architecture_migration_rules.sh`. The static inventory is [global-state-inventory.md](global-state-inventory.md).
 
-As of the Phase 7 closeout, runtime resolver fallbacks have been pushed out of
-the root facade and into explicit owner-local boundaries. Future work should
-therefore treat broad fallback removal as complete and use this document for the
-remaining ECStore-owned bootstrap state and crate-split decisions.
-
-The issue #730 global-state baseline and runtime migration target inventory are
-recorded in [`global-state-inventory.md`](global-state-inventory.md).
+Broad resolver-fallback removal is complete: runtime resolver fallbacks live in explicit owner-local boundaries, not in the root facade. What remains is ECStore-owned bootstrap state and crate-split decisions.
 
 ## Remaining Global Owners
 
-| Owner | Current role | Migration stance |
+| Owner | Role | Stance |
 |---|---|---|
 | `rustfs/src/app/context.rs` | AppContext-first resolver facade. | Resolver helpers stay context-first and do not construct concrete no-AppContext defaults. |
-| `rustfs/src/app/context/runtime_sources.rs` | Default adapters for KMS, IAM, object store, endpoints, config, metrics, and notification state used by AppContext construction. | This is an allowed adapter boundary, not a business logic owner. |
-| `rustfs/src/*/runtime_sources.rs` | Root, admin, app, server, startup, and storage owner-local runtime-source boundaries. | Business modules use these boundaries instead of calling global state directly; owner facades own any remaining no-AppContext compatibility defaults. |
-| `rustfs/src/*/storage_api.rs` | Root, admin, app, and storage owner-local storage contract/facade boundaries. | Storage helper and ECStore facade access remains visible at local owner boundaries. |
-| `crates/*/storage_api.rs` | External crate-local storage facade boundaries for IAM, scanner, heal, notify, observability, Swift, and S3 Select. | External runtime crates consume ECStore runtime state through `rustfs_ecstore::api::runtime` instead of the direct global facade. |
-| `crates/ecstore/src/runtime/global.rs` | ECStore bootstrap/runtime state owner. | Keep internal until ECStore has explicit owner handles for all remaining bootstrap state. |
-| `crates/ecstore/src/runtime/sources.rs` | ECStore runtime-source adapter over global state. | Preferred ECStore-internal access path while shrinking direct `runtime::global` reads. |
+| `rustfs/src/app/context/runtime_sources.rs` | Default adapters for KMS, IAM, object store, endpoints, config, metrics, and notification state used by AppContext construction. | Allowed adapter boundary, not a business-logic owner. |
+| `rustfs/src/runtime_sources.rs`, `rustfs/src/admin/runtime_sources.rs`, `rustfs/src/app/runtime_sources.rs`, `rustfs/src/server/runtime_sources.rs`, `rustfs/src/storage/runtime_sources.rs` | Owner-local runtime-source boundaries. | Business modules use these instead of global state; owner facades decide when to apply no-AppContext compatibility defaults. |
+| `rustfs/src/storage_api.rs`, `rustfs/src/admin/storage_api.rs`, `rustfs/src/app/storage_api.rs`, `rustfs/src/storage/storage_api.rs` | Owner-local storage contract/facade boundaries. | Storage helper and ECStore facade access stays visible at local owner boundaries. |
+| `crates/*/storage_api.rs` | External crate-local storage facade boundaries (IAM, scanner, heal, notify, observability, Swift, S3 Select). | External runtime crates read ECStore runtime state through `rustfs_ecstore::api::runtime`, never the global facade. |
+| `crates/ecstore/src/runtime/global.rs` | ECStore bootstrap/runtime state owner. | Internal until ECStore has explicit owner handles for all remaining bootstrap state. |
+| `crates/ecstore/src/runtime/sources.rs` | ECStore runtime-source adapter over global state. | Preferred ECStore-internal access path while direct `runtime::global` reads shrink. |
 
 ## Runtime Source Boundaries
 
-Runtime-source modules are the allowed compatibility layer between migrated
-consumers and process-global state. They must keep these properties:
+Runtime-source modules are the allowed compatibility layer between migrated consumers and process-global state. They keep these properties:
 
 - context-first lookup when an `AppContext` handle exists;
-- explicit fallback to the existing global only where compatibility still
-  requires it;
+- explicit fallback to the existing global only where compatibility still requires it, decided by the owner facade;
 - no hidden service construction in business logic;
-- no startup, readiness, IAM, KMS, lock, notification, or storage behavior
-  change in inventory or guardrail PRs.
+- the root `rustfs/src/runtime_sources.rs` is an entrypoint only: it composes no concrete fallback defaults (`unwrap_or`, `unwrap_or_else`, direct `init_global` or `new_global` calls);
+- production callers outside runtime-source and `storage_api.rs` boundary modules do not import ECStore global state directly.
 
-## Guarded Boundary List
+### Guarded Boundary List
 
-The architecture guard snapshots the files currently allowed to reference
-`rustfs_ecstore::api::global` directly:
+The guard pins the production files allowed to reference `rustfs_ecstore::api::global` directly:
 
 - `rustfs/src/storage/storage_api.rs`
 
-That boundary now keeps only bootstrap writes and lifecycle controls in the
-global facade. Read-only runtime getters must be exported through
-`rustfs_ecstore::api::runtime` and consumed through the local storage facade.
-New direct uses must either move behind an existing owner-local boundary or
-update this plan and the guard in the same reviewed migration PR.
+That boundary keeps only bootstrap writes and lifecycle controls (`set_global_endpoints`, `set_global_region`, `set_global_rustfs_port`, `set_object_store_resolver`, `shutdown_background_services`, `update_erasure_type`). Read-only runtime getters are exported through `rustfs_ecstore::api::runtime` and consumed through the local storage facade. A new direct use either moves behind an existing owner-local boundary or updates this plan and the guard in the same reviewed change.
 
 ## Fallback Removal Plan
 
-1. Keep AppContext-first lookup as the stable resolver contract.
-2. Keep concrete no-AppContext compatibility defaults only at owner-local
-   runtime-source facades that consume them.
-3. Do not let business logic call `AppContext` or ECStore globals directly when
-   an owner-local runtime-source boundary exists.
-4. Keep embedded startup and tests working before deleting any remaining owner
-   fallback.
-5. Do not remove ECStore bootstrap globals until ownership handles exist for
-   local disks, endpoint pools, lock clients, notification state, tier config,
-   lifecycle state, and object-store publication.
-
-## GLOB-007 Closeout Boundary
-
-`GLOB-007` is complete when these invariants hold:
-
-- root `rustfs/src/runtime_sources.rs` is an AppContext/root facade entrypoint
-  and no longer composes concrete fallback defaults with `unwrap_or`,
-  `unwrap_or_else`, direct `init_global`, or direct `new_global` calls;
-- private AppContext resolver helpers are context-first and do not hide fallback
-  closure parameters;
-- admin, app, storage, server, startup, and config owner facades decide when to
-  apply no-AppContext compatibility defaults;
-- production callers outside runtime-source and storage-api boundary modules do
-  not import ECStore global state directly;
-- the architecture guard keeps the direct `rustfs_ecstore::api::global`
-  boundary list explicit.
-
-Allowed remaining fallbacks are owner compatibility decisions, not resolver
-fallback families. They are kept so embedded startup, tests, and no-context
-callers preserve the previous behavior while higher layers continue migrating
-to explicit AppContext ownership.
+1. AppContext-first lookup is the stable resolver contract.
+2. Concrete no-AppContext compatibility defaults exist only at the owner-local runtime-source facades that consume them.
+3. Business logic does not call `AppContext` or ECStore globals directly when an owner-local runtime-source boundary exists.
+4. Embedded startup and tests keep working before any remaining owner fallback is deleted.
+5. ECStore bootstrap globals stay until ownership handles exist for local disks, endpoint pools, lock clients, notification state, tier config, lifecycle state, and object-store publication.
 
 ## Crate Split Evaluation
 
-`ecstore-erasure` and `storage-cluster` remain proposal-only until dependency
-cycles and hot-path risks are proven safe. The Phase 7 evaluation is complete
-for now: neither split is ready for code movement in this migration round.
-The follow-up ECStore module split plan is recorded in
-[`ecstore-module-split-plan.md`](ecstore-module-split-plan.md), including the
-remaining `SetDisks`, lifecycle, replication, and facade-shrink boundaries.
+`ecstore-erasure` and `storage-cluster` are proposal-only; neither is ready for code movement. Lifecycle and replication split status is tracked in [ecstore-module-split-plan.md](ecstore-module-split-plan.md).
 
-### CRATE-001: `ecstore-erasure`
+### `ecstore-erasure`
 
-Current coupling:
+Coupling: erasure decoding depends on disk errors, disk read timeouts, and set-disk shard sources; set-disk read/write/heal paths construct codecs in hot object I/O paths; bitrot readers/writers live in ECStore IO support and serve both erasure and set-disk code; `rustfs_ecstore::api::erasure` is still a public compatibility surface.
 
-- erasure decoding depends on disk errors, disk read timeouts, and set-disk
-  shard sources;
-- set-disk read/write/heal paths construct erasure codecs in hot object I/O
-  paths;
-- bitrot readers/writers live in ECStore IO support and are used by both
-  erasure and set-disk code;
-- public compatibility still exposes erasure symbols through
-  `rustfs_ecstore::api::erasure`.
+Decision: do not split. The boundary becomes a candidate only after shard-source, disk-error, bitrot, and metrics contracts are explicit enough to avoid a dependency cycle back into ECStore, backed by encode/decode/reconstruction benchmarks and a rollback plan that keeps read/write quorum and old-version decode unchanged.
 
-Decision: do not split in code yet. The erasure boundary is a candidate only
-after the shard-source, disk-error, bitrot, and metrics contracts are explicit
-enough to avoid a dependency cycle back into ECStore.
+### `storage-cluster`
 
-Required evidence before proposing the split:
+Coupling: cluster RPC remote-disk code depends on disk stores, disk health tracking, set-disk buffer sizing, local disk scan guards, internode metrics, and runtime credential/signature sources; peer S3 and peer REST clients share bucket metadata, disk quorum reduction, endpoint layout, local disk initialization, and store helpers; control-plane snapshots are separate from data-plane RPC, but remote disk and peer clients still own data-movement side effects inside ECStore.
 
-- `cargo tree -p rustfs-ecstore -e normal --depth 2` snapshot for dependency
-  impact;
-- focused benchmarks for encode/decode, read reconstruction, bitrot verification,
-  and large-object streaming;
-- contract sketch for shard sources, disk errors, bitrot IO, metrics, and file
-  metadata without importing ECStore implementation modules;
-- compatibility plan for `rustfs_ecstore::api::erasure` and test harnesses;
-- rollback plan that keeps object read/write quorum and old-version file decode
-  behavior unchanged.
-
-### CRATE-002: `storage-cluster`
-
-Current coupling:
-
-- cluster RPC remote disk code depends on disk stores, disk health tracking,
-  set-disk buffer sizing, local disk scan guards, internode metrics, and runtime
-  credential/signature sources;
-- peer S3 and peer REST clients share bucket metadata, disk quorum reduction,
-  endpoint layout, local disk initialization, and store helpers;
-- control-plane snapshots are separated from data-plane RPC, but remote disk and
-  peer clients still own data movement side effects inside ECStore.
-
-Decision: do not split in code yet. The storage-cluster boundary is a candidate
-only after remote disk, peer health, lock/quorum, runtime metrics, and endpoint
-layout contracts are explicit enough to stand below ECStore without circular
-dependencies.
-
-Required evidence before proposing the split:
-
-- dependency graph showing no cycle with ECStore, `rustfs-storage-api`, runtime
-  source owners, or cluster control-plane owners;
-- RPC contract sketch for remote disk, peer S3, peer REST, auth/signature,
-  internode metrics, and cancellation;
-- compatibility plan for `rustfs_ecstore::api::cluster`, `api::rpc`, and test
-  fixtures that build local disks or endpoint pools;
-- focused tests for remote disk error classification, peer health recovery,
-  per-pool quorum reduction, lock behavior, and data-stream request paths;
-- rollback plan that preserves quorum, remote disk IO, lock, peer health, and
-  data movement behavior.
-
-### CRATE-003: `bucket-lifecycle`
-
-Decision: do not split in code yet. Lifecycle remains coupled to ECStore object
-operations, bucket metadata, `SetDisks` stale multipart cleanup, tier config,
-runtime lifecycle state, scanner metrics, notification/audit side effects, and
-replication delete scheduling.
-
-Required evidence before proposing the split:
-
-- contract sketch for lifecycle object operations, metadata access, runtime
-  state, replication delete scheduling, and audit/notification sinks;
-- dependency graph showing the candidate crate can avoid importing ECStore
-  implementation modules;
-- focused tests for lifecycle evaluation, expiry, transition, stale multipart
-  cleanup, tier journal recovery, and lifecycle-originated replication deletes;
-- compatibility plan for `rustfs_ecstore::api::bucket::lifecycle` consumers;
-- rollback plan that preserves lifecycle queues, scanner repair accounting,
-  tier transitions, object deletion behavior, and notification/audit events.
-
-### CRATE-004: `bucket-replication`
-
-Decision: do not split in code yet. Replication remains coupled to ECStore
-object APIs, bucket target clients, metadata systems, file metadata replication
-state, ECStore-owned runtime replication pool/stat handles, bucket monitor
-state, scanner repair classification, lifecycle-originated deletes, and
-notification events. RustFS-facing runtime consumers should use storage-owner
-wrapper handles while that state remains in ECStore.
-
-Required evidence before proposing the split:
-
-- contract sketch for replication storage operations, metadata/target access,
-  runtime pool and stats, event sinks, and lifecycle/heal bridges;
-- dependency graph showing the candidate crate can avoid importing ECStore
-  implementation modules;
-- focused tests for object replication, delete replication, resync state, heal
-  repair queueing, target error handling, and queue admission;
-- compatibility plan for `rustfs_ecstore::api::bucket::replication` consumers;
-- rollback plan that preserves replication queues, MRF/resync state, target
-  client behavior, scanner repair, and event emission.
+Decision: do not split. The boundary becomes a candidate only after remote disk, peer health, lock/quorum, runtime metrics, and endpoint layout contracts can stand below ECStore without cycles, with compatibility plans for `rustfs_ecstore::api::cluster` and `api::rpc` and focused tests for remote disk error classification, peer health recovery, per-pool quorum reduction, lock behavior, and data-stream request paths.
 
 ## Preservation Rules
 
-- Do not reintroduce AppContext resolver fallback families in broad cleanup PRs.
-- Do not introduce direct global reads in admin, app, server, storage, scanner,
-  heal, IAM, notify, observability, Swift, or S3 Select business logic.
-- Do not split crates in the same PR that moves runtime state.
-- Do not change startup order, readiness, KMS fatal boundaries, IAM recovery,
-  lock quorum, object placement, reader behavior, or notification/audit
-  lifecycle while shrinking global state.
+- Do not reintroduce AppContext resolver fallback families in broad cleanups.
+- Do not introduce direct global reads in admin, app, server, storage, scanner, heal, IAM, notify, observability, Swift, or S3 Select business logic.
+- Do not split crates in the same change that moves runtime state.
+- Do not change startup order, readiness, KMS fatal boundaries, IAM recovery, lock quorum, object placement, reader behavior, or notification/audit lifecycle while shrinking global state.

docs/architecture/global-state-inventory.md

@@ -1,146 +1,64 @@
 # Global State Inventory
 
-This inventory records the issue #730 baseline for global runtime state after
-the AppContext foundation and owner-local runtime-source boundaries were added.
-It is intentionally documentation-only: it classifies migration targets without
-changing startup, readiness, object IO, lifecycle, replication, or notification
-behavior.
-
-## Counting Baseline
-
-The audit uses the current workspace Rust sources and keeps broad static
-caches separate from runtime migration targets.
-
-| Scope | Count | Command |
-|---|---:|---|
-| Rust source files | 1,252 | `rg --files -g '*.rs'` |
-| `OnceLock` references | 221 lines | `rg -n --glob '*.rs' 'OnceLock'` |
-| `GLOBAL_*` references | 273 lines | `rg -n --glob '*.rs' '\bGLOBAL_[A-Za-z0-9_]*\b'` |
-| `static NAME:` definitions | 621 lines | `rg -n --glob '*.rs' '^\s*(pub(\([^)]*\))?\s+)?static(\s+mut)?\s+[A-Za-z_][A-Za-z0-9_]*\s*:'` |
-| `lazy_static!` `static ref` definitions | 58 lines | `rg -n --glob '*.rs' '^\s*(pub\s+)?static\s+ref\s+[A-Za-z_][A-Za-z0-9_]*\s*:'` |
-| `static mut` definitions | 0 lines | `rg -n --glob '*.rs' '^\s*(pub(\([^)]*\))?\s+)?static\s+mut\s+'` |
+**Use this when:** you meet a `GLOBAL_*` static or an `OnceLock` and need to know whether it is a runtime ownership handle (reach it through a boundary), an owner-local static (leave it inside its module), or process-global by design.
+**Source of truth:** `crates/ecstore/src/api/mod.rs` (the `pub mod runtime` and `pub mod global` re-export lists), `crates/ecstore/src/runtime/global.rs`, `crates/ecstore/src/runtime/sources.rs`, and the statics themselves. Boundary rules are in [global-state-crate-split-plan.md](global-state-crate-split-plan.md).
 
 ## Global State Classification
 
 | Category | Rule | Representative owners |
 |---|---|---|
-| Process-global | Process identity, metrics registries, lock manager, audit guard, TLS material, or other state that is intentionally one per process. | `crates/credentials`, `crates/common`, `crates/io-metrics`, `crates/lock`, `crates/obs`, `crates/tls-runtime` |
-| Runtime migration target | Mutable runtime state that describes the active object store, endpoints, local disks, lifecycle, replication, notification, config, or background controllers. | `crates/ecstore/src/runtime/global.rs`, `crates/ecstore/src/runtime/sources.rs`, `rustfs/src/app/context/*` |
-| Owner-local compatibility | Existing compatibility adapters that are allowed to read globals while callers migrate to AppContext-first or owner-local runtime-source APIs. | `rustfs/src/*/runtime_sources.rs`, `rustfs/src/*/storage_api.rs`, `crates/*/storage_api.rs` |
-| Test or fixture state | Static setup used by tests to amortize expensive ECStore setup or isolate compatibility harness state. | `rustfs/src/app/*_test.rs`, `crates/scanner/tests/*`, `crates/ecstore/src/**/tests` |
-| Cache or constant | Regexes, metrics descriptors, defaults, KVS registrations, headers, path constants, and small process caches that are not runtime ownership handles. | `crates/config`, `crates/obs/src/metrics`, `crates/utils`, `rustfs/src/server/readiness.rs` |
-| Legacy naming or review-needed | Old MinIO-port naming, stale comments, or names that need owner confirmation before code movement. | `GLOBAL_OBJECT_API` |
+| Process-global | Process identity, metrics registries, lock manager, audit guard, TLS material, or other state intentionally one per process. | `GLOBAL_LOCK_MANAGER` (`crates/lock`), `GLOBAL_CONN_MAP` (`crates/common`), `GLOBAL_RUSTFS_RPC_SECRET` (`crates/credentials`), `AUDIT_SYSTEM` (`crates/audit`), `crates/io-metrics`, `crates/obs`, `crates/tls-runtime` |
+| Runtime migration target | Mutable runtime state describing the active object store, endpoints, local disks, lifecycle, replication, notification, config, or background controllers. | `crates/ecstore/src/runtime/global.rs`, `crates/ecstore/src/runtime/sources.rs`, `rustfs/src/app/context/` |
+| Owner-local compatibility | Adapters allowed to read globals while callers migrate to AppContext-first or owner-local runtime-source APIs. | `rustfs/src/*/runtime_sources.rs`, `rustfs/src/*/storage_api.rs`, `crates/*/storage_api.rs` |
+| Owner-local static | A static private to one module and reached only through that module's functions: caches, single-run guards, admission locks, module toggles. | The RustFS inventory below |
+| Test or fixture state | Static setup that amortizes expensive ECStore setup or isolates harness state. | `rustfs/src/app/*_test.rs`, `crates/scanner/tests/`, `crates/test-utils/src/ecstore_test_compat.rs` |
+| Cache or constant | Regexes, metrics descriptors, defaults, KVS registrations, headers, path constants. | `crates/config`, `crates/obs/src/metrics`, `crates/utils` |
 
 ## Runtime Migration Inventory
 
-These are the issue #730 targets that should remain visible until an owner
-migration PR removes or replaces each item.
+Runtime ownership handles that exist today. Reads go through `rustfs_ecstore::api::runtime`, bootstrap writes go through `rustfs_ecstore::api::global`, and RustFS code reaches both only from `rustfs/src/storage/storage_api.rs` and the AppContext resolvers.
 
-| State | Current boundary | Category | Migration stance |
-|---|---|---|---|
-| `APP_CONTEXT_SINGLETON` | `rustfs/src/app/context/global.rs` | Owner-local compatibility | Keep as the context-first facade while no-context startup and embedded callers still exist. |
-| `GLOBAL_OBJECT_API`, `GLOBAL_OBJECT_STORE_RESOLVER` | `crates/ecstore/src/runtime/global.rs`, `rustfs/src/app/context/global.rs`, and storage compatibility APIs | Runtime migration target | Do not migrate first; it is tied to storage startup, IAM-after-storage AppContext publication, and data-plane resolver compatibility. The object-store resolver is now published from the AppContext owner path, no longer re-exported from the RustFS storage root, and RustFS AppContext tests no longer use the old `new_object_layer_fn` fallback chain. RustFS storage root no longer re-exports ECStore runtime/global facade symbols; callers must use storage/app/admin facades. |
-| `GLOBAL_ENDPOINTS`, `GLOBAL_IS_ERASURE`, `GLOBAL_IS_DIST_ERASURE`, `GLOBAL_IS_ERASURE_SD`, `GLOBAL_ROOT_DISK_THRESHOLD` | `crates/ecstore/src/runtime/global.rs` and `crates/ecstore/src/runtime/sources.rs` | Runtime migration target | Endpoint and setup-type reads now flow through ECStore `api::runtime` helpers at the RustFS storage facade boundary; root-disk-threshold access stays behind ECStore runtime helpers. Move endpoint ownership only after readiness and quorum behavior have explicit coverage. |
-| `GLOBAL_LOCAL_DISK_MAP`, `GLOBAL_LOCAL_DISK_ID_MAP`, `GLOBAL_LOCAL_DISK_SET_DRIVES` | `crates/ecstore/src/runtime/global.rs` and `crates/ecstore/src/runtime/sources.rs` | Runtime migration target | Local disk map, disk-id cache, and set-drive access now stay behind ECStore runtime-source helpers instead of direct global access; preserve disk lookup, remote/local classification, and test reset hooks in later ownership changes. |
-| `GLOBAL_EXPIRY_STATE`, `GLOBAL_TRANSITION_STATE`, `GLOBAL_LIFECYCLE_SYS` | `crates/ecstore/src/bucket/lifecycle/*`, `crates/ecstore/src/runtime/global.rs`, and `crates/ecstore/src/runtime/sources.rs` | Runtime migration target | Lifecycle state globals now stay behind ECStore lifecycle owner helpers and ECStore runtime-source helpers; RustFS AppContext has expiry/transition state interfaces and resolver coverage, and daily tier stats derive from the transition-state handle instead of a separate context boundary; scanner expiry-state access still uses the ECStore runtime `expiry_state_handle` boundary until scanner gets an injected provider. |
-| `GLOBAL_REPLICATION_POOL`, `GLOBAL_REPLICATION_STATS`, `GLOBAL_BUCKET_MONITOR` | `crates/ecstore/src/bucket/replication/*`, `crates/ecstore/src/runtime/global.rs` | Runtime migration target | Replication pool/stat access now stays behind replication owner and ECStore runtime-source helpers; bucket-monitor reads now flow through ECStore `api::runtime` at the RustFS storage facade boundary while AppContext/runtime-source resolvers remain the caller boundary. |
-| `GLOBAL_TIER_CONFIG_MGR`, `GLOBAL_STORAGE_CLASS`, `GLOBAL_CONFIG_SYS`, `GLOBAL_SERVER_CONFIG` | `crates/ecstore/src/config`, `crates/config`, `rustfs/src/app/context/runtime_sources.rs` | Runtime migration target | Tier config manager reads and reloads now use the ECStore runtime-source helper; move remaining config state through config/runtime-source owners only, without combining storage-class behavior or persistence changes. |
-| `GLOBAL_EVENT_NOTIFIER`, `GLOBAL_NOTIFICATION_SYS` | `crates/ecstore/src/runtime/global.rs`, `crates/ecstore/src/runtime/sources.rs`, and `crates/ecstore/src/services/*` | Runtime migration target | `GLOBAL_EVENT_NOTIFIER` and `GLOBAL_NOTIFICATION_SYS` access now stay behind ECStore runtime-source and notification owner helpers; move remaining notification ownership only through notify/runtime-source boundaries. |
-| `EVENT_DISPATCH_HOOK` | `crates/ecstore/src/services/event_notification.rs`, RustFS server event bridge, and storage compatibility APIs | Runtime migration target / owner helper | Direct hook storage stays inside the ECStore event-notification owner; RustFS registers the bridge through the storage compatibility facade until event dispatch ownership moves behind an injected notification sink. |
-| `GLOBAL_BUCKET_METADATA_SYS` | `crates/ecstore/src/bucket/metadata_sys.rs`, `crates/ecstore/src/runtime/sources.rs`, and RustFS storage compatibility APIs | Runtime migration target | Bucket metadata system direct access now stays inside the ECStore metadata owner; callers use metadata owner helpers or storage/runtime-source compatibility functions until metadata ownership moves behind an injected runtime context. |
-| `GLOBAL_BOOT_TIME`, `GLOBAL_BACKGROUND_SERVICES_CANCEL_TOKEN`, `GLOBAL_DEPLOYMENT_ID`, `GLOBAL_REGION`, `GLOBAL_RUSTFS_PORT`, `GLOBAL_LOCAL_NODE_NAME_FALLBACK`, `GLOBAL_LOCAL_NODE_NAME_HEX_FALLBACK` | `crates/ecstore/src/runtime/global.rs`, `crates/ecstore/src/runtime/sources.rs` | Runtime migration target | Boot time, background service cancellation token reads, ECStore local-node-name fallback reads, and deployment ID/region/port reads now stay behind the ECStore runtime-source API; scalar writes remain behind bootstrap owner helpers until ownership handles replace them. |
-| `WORKLOAD_ADMISSION_SNAPSHOT_PROVIDER` | `crates/ecstore/src/runtime/sources.rs`, RustFS startup background setup, and storage compatibility APIs | Runtime migration target / owner helper | Startup publishes the workload provider through the storage compatibility facade, and ECStore data movement reads it only through the runtime-source helper until workload admission ownership moves into an explicit runtime context. |
-| `GLOBAL_LOCAL_LOCK_CLIENT`, `GLOBAL_LOCK_CLIENTS`, `GLOBAL_LOCK_MANAGER` | `crates/ecstore/src/runtime/global.rs`, `crates/lock` | Runtime migration target / process-global split | ECStore lock client reads now flow through ECStore `api::runtime` helpers at the RustFS storage facade boundary; preserve lock quorum and lock client selection while keeping the process-level lock manager separate from endpoint-specific clients. |
-| `GLOBAL_CONN_MAP`, `GLOBAL_LOCAL_NODE_NAME`, `GLOBAL_RUSTFS_HOST`, `GLOBAL_RUSTFS_ADDR`, `GLOBAL_ROOT_CERT`, `GLOBAL_MTLS_IDENTITY`, `GLOBAL_OUTBOUND_TLS_GENERATION` | `crates/common`, `crates/tls-runtime`, `crates/ecstore/src/runtime/sources.rs` | Runtime migration target / process-global split | Internode connection cache, common local node name, RustFS host/address reads, and outbound TLS material reads are now owned behind `rustfs_common` helpers; migrate the remaining transport and TLS state only after internode transport and outbound TLS ownership are explicit, without changing cached channel reuse or TLS reload semantics. |
-| `GLOBAL_RUSTFS_RPC_SECRET` | `crates/credentials`, `crates/ecstore/src/runtime/sources.rs` | Runtime migration target / process-global split | RPC auth token writes now stay behind the `rustfs_credentials` helper boundary; migrate only if runtime secret ownership changes, preserving lazy environment and credential-derived token semantics. |
-| `GLOBAL_HEAL_MANAGER`, `GLOBAL_HEAL_CHANNEL_PROCESSOR`, `GLOBAL_AHM_SERVICES_CANCEL_TOKEN` | `crates/heal/src/lib.rs` | Runtime migration target / process-global split | Direct access now stays inside the heal owner; callers use heal helper functions until heal runtime ownership moves behind explicit owner handles. |
-| `AUDIT_SYSTEM` | `crates/audit/src/global.rs` | Runtime migration target / process-global split | Direct global access now stays inside the audit owner; callers use audit helper functions until audit lifecycle ownership moves behind AppContext or a runtime-source boundary. |
-| `GLOBAL_PROCESSORS` | `crates/ecstore/src/services/batch_processor.rs`, `crates/ecstore/src/runtime/sources.rs` | Runtime migration target / owner helper | Direct static access now stays inside the ECStore batch processor owner; callers use `get_global_processors` or the ECStore runtime-source helper until processor ownership moves into an injected runtime context. |
-| `INTERNODE_DATA_TRANSPORT` | `crates/ecstore/src/cluster/rpc/internode_data_transport.rs` | Runtime migration target / owner helper | Direct static access now stays inside the ECStore internode transport owner; callers use `build_internode_data_transport_from_env` until backend selection moves into an injected runtime context. |
-| `GLOBAL_KMS_SERVICE_MANAGER` | `crates/kms/src/service_manager.rs`, RustFS KMS runtime sources | Runtime migration target / owner helper | Direct static access now stays inside the `rustfs_kms` service manager owner; RustFS callers use KMS helpers or AppContext/runtime-source handles until KMS ownership fully moves into runtime context. |
-| `GLOBAL_CAPACITY_MANAGER` | `crates/object-capacity/src/capacity_manager.rs`, RustFS capacity service | Runtime migration target / owner helper | Direct static access now stays inside the object-capacity owner; callers use `get_capacity_manager` or isolated manager factories until capacity ownership moves into an injected runtime context. |
-| `GLOBAL_BUCKET_TARGET_SYS` | `crates/ecstore/src/bucket/bucket_target_sys.rs`, admin/app/scanner/replication target paths | Runtime migration target / owner helper | Direct static access now stays inside the ECStore bucket target owner; callers still use `BucketTargetSys::get()` until bucket target ownership moves behind a runtime-source or replication target boundary. |
-| `USAGE_MEMORY_CACHE`, `USAGE_CACHE_UPDATING` | `crates/ecstore/src/data_usage/mod.rs` | Runtime migration target / owner-local cache | Data-usage memory overlay and singleflight state stay private to the ECStore data-usage owner; callers use data-usage functions until scanner/data-usage ownership moves behind an injected runtime context. |
+| Handle (`rustfs_ecstore::api::runtime`) | Backing state | Stance |
+|---|---|---|
+| `object_store_handle` | `GLOBAL_OBJECT_API`, `GLOBAL_OBJECT_STORE_RESOLVER` (`crates/ecstore/src/runtime/global.rs`); the resolver is published from the AppContext owner path | Do not migrate first: tied to storage startup, IAM-after-storage AppContext publication, and data-plane resolver compatibility. |
+| `endpoint_pools`, `setup_is_erasure`, `setup_is_dist_erasure`, `setup_is_erasure_sd`, `first_cluster_node_is_local` | `GLOBAL_ENDPOINTS` and setup-type state (`crates/ecstore/src/runtime/global.rs`) | Move endpoint ownership only after readiness and quorum behavior have explicit coverage. |
+| `local_disk_map_read` | Local disk map and set-drive state (`crates/ecstore/src/runtime/sources.rs`) | Preserve disk lookup, remote/local classification, and test reset hooks. |
+| `expiry_state_handle`, `transition_state_handle` | Lifecycle expiry and transition state, `GLOBAL_LIFECYCLE_SYS` (`crates/ecstore/src/runtime/global.rs`) | Lifecycle owner helpers and the AppContext `ExpiryStateInterface` (`rustfs/src/app/context/interfaces.rs`) are the caller boundary; the scanner still reads `expiry_state_handle` until it gets an injected provider. |
+| `global_tier_config_mgr` | Tier config manager | Reads and reloads stay behind this helper. |
+| `bucket_monitor` | Replication bandwidth monitor | Replication pool/stat handles are projected into RustFS wrapper types at the storage boundary. |
+| `global_lock_client`, `global_lock_clients` | `GLOBAL_LOCAL_LOCK_CLIENT`, `GLOBAL_LOCK_CLIENTS` (`crates/ecstore/src/runtime/global.rs`) | Preserve lock quorum and client selection; the process-level `GLOBAL_LOCK_MANAGER` stays separate. |
+| `boot_time`, `deployment_id`, `region`, `rustfs_port` | `GLOBAL_BOOT_TIME`, deployment id, region, and port state (`crates/ecstore/src/runtime/global.rs`) | Scalar writes remain behind the `api::global` setters (`set_global_endpoints`, `set_global_region`, `set_global_rustfs_port`, `set_object_store_resolver`, `shutdown_background_services`, `update_erasure_type`). |
 
-## Owner-Local Cache Inventory
+Owner-helper handles outside the runtime-source list stay inside their owner and are reached through owner functions: `GLOBAL_EVENT_NOTIFIER` (`crates/ecstore/src/runtime/global.rs`); `GLOBAL_NOTIFICATION_SYS`, `EVENT_DISPATCH_HOOK`, `GLOBAL_PROCESSORS`, `INTERNODE_DATA_TRANSPORT`, `GLOBAL_BUCKET_TARGET_SYS`, `GLOBAL_CONFIG_SYS`, `GLOBAL_STORAGE_CLASS`, `WORKLOAD_ADMISSION_SNAPSHOT_PROVIDER` (ECStore owner modules); `GLOBAL_SERVER_CONFIG` (`crates/config/src/server_config.rs`); `GLOBAL_HEAL_RUNTIME`, `GLOBAL_AHM_SERVICES_CANCEL_TOKEN` (`crates/heal/src/lib.rs`); `GLOBAL_KMS_SERVICE_MANAGER` (`crates/kms/src/service_manager.rs`); `GLOBAL_CAPACITY_MANAGER` (`crates/object-capacity/src/capacity_manager.rs`); `APP_CONTEXT_SINGLETON` (`rustfs/src/app/context/global.rs`).
 
-These owner-local caches and static guards are part of the broad issue #730
-`OnceLock` audit, but they are not runtime ownership handles. They stay private
-to the defining owner module; callers must use the existing owner APIs instead
-of reaching across module boundaries.
+Regenerate:
 
-| State | Owner boundary | Category | Migration stance |
-|---|---|---|---|
-| `READ_REPAIR_HEAL_CACHE` | `crates/ecstore/src/set_disk/read.rs` | Cache or constant / owner-local cache | Read-repair heal suppression stays local to set-disk read handling. |
-| `DISK_COMPRESSION_CONFIG` | `crates/ecstore/src/io_support/compress.rs` | Cache or constant / owner-local cache | Disk compression environment parsing stays local to IO support compression helpers. |
-| `CACHED_MAX_INFLIGHT_BYTES`, `CACHED_BATCH_BLOCKS`, `CACHED_BYTESMUT_INGEST` | `crates/ecstore/src/erasure/coding/encode.rs` | Cache or constant / owner-local cache | Erasure encode tuning caches stay local to the coding owner. |
-| `CACHED_PUT_LARGE_BATCH_MIN_SIZE_BYTES`, `CACHED_MULTIPART_PUT_LARGE_BATCH_MIN_SIZE_BYTES`, `OBJECT_LOCK_DIAG_ENABLED` | `crates/ecstore/src/set_disk/mod.rs` | Cache or constant / owner-local cache | Set-disk batching and diagnostics caches stay local to the set-disk owner. |
-| `DRIVE_TIMEOUT_PROFILE_CACHE`, `DRIVE_TIMEOUT_HEALTH_POLICY_CACHE` | `crates/ecstore/src/disk/disk_store.rs` | Cache or constant / owner-local cache | Drive timeout environment caches stay local to the disk-store owner. |
-| `TIER_FREE_VERSION_RECOVERY_STARTED`, `TIER_DELETE_JOURNAL_RECOVERY_STARTED` | `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` | Cache or constant / owner-local static guard | Lifecycle recovery single-run guards stay local to lifecycle operations. |
-| `REMOTE_DELETE_INFLIGHT`, `REMOTE_DELETE_LIMITER`, `REMOTE_DELETE_BREAKER`, `REMOTE_TIER_DELETE_TEST_HOOK` | `crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs` | Cache or constant / owner-local static guard | Remote tier delete concurrency, breaker, and test hook state stay local to the tier sweeper owner. |
-| `ACTIVE_REGISTRY`, `BackendCapacity` | `crates/kms/src/policy.rs` | Process-global owner-local admission capacity registry | KMS policy generations share only active semaphore capacity by backend identity; each generation owns fresh bounded queues and circuit breakers. Callers access this state only through `RetryPolicy`. |
+```bash
+rg -n -A4 'pub use crate::runtime::(sources|global)::' crates/ecstore/src/api/mod.rs
+rg -n --glob '*.rs' 'static (ref )?GLOBAL_[A-Z_]+' crates rustfs/src
+```
 
 ## RustFS Owner-Local Static Inventory
 
-These RustFS-side lazy, atomic, and `OnceLock` statics are also part of the
-issue #730 process-static audit. They are private implementation details for
-their owner modules, not shared runtime ownership handles. This section excludes
-allocator statics, public contract/error references, route handler constants,
-and `APP_CONTEXT_SINGLETON`, which is classified in the runtime migration
-inventory. Generic function-local names such as `CACHE`, `LOCK`, `INIT`, and
-`ENABLED` are documented by owner row instead of name-regex guarded.
+RustFS-side statics that matter architecturally because other modules are tempted to reach them. They stay private to their owner module; callers use the owner's functions.
 
-| State | Owner boundary | Category | Migration stance |
-|---|---|---|---|
-| `KEYSTONE_AUTH`, `KEYSTONE_MAPPER`, `KEYSTONE_CONFIG` | `rustfs/src/auth_keystone.rs` | Process-global owner-local state | Keystone authentication provider, identity mapper, and config stay private to the Keystone auth owner. |
-| `LICENSE_STATE`, `LICENSE_VERIFIER` | `rustfs/src/license.rs` | Process-global owner-local state | License state and verifier selection stay private to the license owner; callers use license helper functions. |
-| `CPU_CONT_GUARD`, `PROFILING_CANCEL_TOKEN` | `rustfs/src/profiling.rs` | Process-global owner-local guard | CPU profiling guard and cancellation state stay private to the profiling owner. |
-| `MEMORY_SYSTEM` | `rustfs/src/memory_observability.rs` | Process-global owner-local cache | Memory sampling keeps the `sysinfo::System` cache private to the memory observability owner. |
-| `DISPLAY_CONFIG_SNAPSHOT`, `GLOBAL_CONFIG_SNAPSHOT` | `rustfs/src/config/snapshot.rs` | Process-global owner-local state | Config snapshots stay private to the config snapshot owner. |
-| `BUFFER_CONFIG_SINGLETON`, `BUFFER_PROFILE_ENABLED` | `rustfs/src/config/workload_profiles.rs` | Process-global owner-local state | Workload buffer profile configuration stays private to workload profile helpers. |
-| `LEGACY_CREDENTIAL_WARNED_KEYS` | `rustfs/src/config/config_struct.rs` | Process-global owner-local cache | Legacy credential warning de-duplication stays private to config parsing. |
-| `CONSOLE_CONFIG` | `rustfs/src/admin/console.rs` | Process-global owner-local state | Console bootstrap config stays private to the admin console owner. |
-| `ACTIVE_HTTP_REQUESTS` | `rustfs/src/server/http.rs` | Process-global owner-local counter | HTTP request inflight accounting stays private to the HTTP server owner. |
-| Function-local `CACHE` and `LOCK` statics | `rustfs/src/server/readiness.rs` | Cache or constant / owner-local cache | Readiness and cluster-health caches stay function-local to readiness probes. |
-| `USE_STARSHARD_CACHE`, `BUCKET_CACHE_SMALL`, `BUCKET_CACHE_LARGE` | `rustfs/src/storage/ecfs_extend.rs` | Cache or constant / owner-local cache | Bucket validation cache backend selection and cache storage stay private to the ECFS extension owner. |
-| `GLOBAL_SSE_DEK_PROVIDER`, `SSE_TEST_LOCK` | `rustfs/src/storage/sse.rs` | Owner-local cache / test state | SSE DEK provider cache and test serialization lock stay private to the SSE owner. |
-| `AUTH_FS` | `rustfs/src/storage/access.rs` | Cache or constant / owner-local cache | Authorization tag-condition lookup keeps its filesystem helper private to the access owner. |
-| `DEADLOCK_DETECTOR` | `rustfs/src/storage/deadlock_detector.rs` | Process-global owner-local state | Deadlock detector lifecycle state stays private to the storage deadlock detector owner. |
-| `CONCURRENCY_MANAGER`, `ACTIVE_GET_REQUESTS`, `ACTIVE_PUT_REQUESTS` | `rustfs/src/storage/concurrency/*` | Process-global owner-local scheduler state | Storage concurrency manager and request counters remain inside the storage concurrency owner boundary. |
-| `GET_OBJECT_BUFFER_THRESHOLD_WARNED`, `GET_READER_STREAM_BUFFER_SIZE_OVERRIDE`, function-local `ENABLED`, `OBJECT_SEEK_SUPPORT_THRESHOLD`, `OBJECT_SEEK_SUPPORT_CONCURRENCY_THRESHOLDS` | `rustfs/src/app/object/get.rs` | Cache or constant / owner-local cache | Object GET/seek tuning caches and warning guards stay private to object usecase helpers. |
-| `SUPPORTED_HEADERS` | `rustfs/src/storage/options.rs` | Cache or constant / owner-local constant | Supported-header lookup state stays private to storage option parsing. |
-| `AUDIT_TARGET_SPECS`, `NOTIFICATION_TARGET_SPECS` | `rustfs/src/admin/handlers/audit.rs`, `rustfs/src/admin/handlers/event.rs`, `rustfs/src/admin/handlers/plugins_instances.rs` | Cache or constant / owner-local constant | Admin target descriptor tables stay private to their handler owners. |
-| `SITE_REPLICATION_PEER_CLIENT` | `rustfs/src/site_replication/transport.rs` | Process-global owner-local cache | Site-replication peer client cache stays private to the site-replication transport module. The state RMW transaction holds no process-local mutex — see `rustfs/src/site_replication/state_lock.rs`. |
-| `AUDIT_MODULE_ENABLED`, `NOTIFY_MODULE_ENABLED`, `PERSISTED_NOTIFY_MODULE_ENABLED`, `PERSISTED_AUDIT_MODULE_ENABLED`, `PERSISTED_MODULE_SWITCH_CONFIGURED` | `rustfs/src/server/audit.rs`, `rustfs/src/server/event.rs`, `rustfs/src/server/module_switch.rs` | Process-global owner-local toggles | Audit/notify module snapshots stay private to the server module switch owners. |
-| `DELETE_TAIL_TOTAL`, `DELETE_CLEANUP_TOTAL`, `DELETE_REPLICATION_TOTAL`, `DELETE_NOTIFY_TOTAL` | `rustfs/src/delete_tail_activity.rs` | Process-global owner-local counters | Delete-tail activity counters stay private behind delete-tail activity helpers. |
-| `EMBEDDED_SERVER_STARTED` | `rustfs/src/startup_lifecycle.rs` | Process-global owner-local guard | Embedded startup single-start protection stays private to startup lifecycle. |
-| `TEST_OUTBOUND_TLS_GENERATION` | `rustfs/src/site_replication/mod.rs` | Test or fixture state | Outbound TLS generation test hook state stays private to site-replication transport tests. |
-| `TEST_REMAINING_FAILURES` | `rustfs/src/startup_iam.rs` | Test or fixture state | IAM startup retry injection state stays private to debug/test startup code. |
-| `CAPACITY_DIRTY_SCOPE_ENV`, `CAPACITY_DIRTY_SCOPE_INIT`, `GLOBAL_ENV`, function-local `INIT` | `rustfs/src/app/*_test.rs` | Test or fixture state | App integration test fixture state stays private to the owning test modules. |
+| Static | Owner | Stance |
+|---|---|---|
+| `KEYSTONE_AUTH`, `KEYSTONE_MAPPER`, `KEYSTONE_CONFIG` | `rustfs/src/auth_keystone.rs` | Keystone provider, mapper, and config stay private to the Keystone owner. |
+| `DEADLOCK_DETECTOR` | `rustfs/src/storage/deadlock_detector.rs` | Detector lifecycle stays private to the storage deadlock detector. |
+| `CONCURRENCY_MANAGER` | `rustfs/src/storage/concurrency/manager.rs` | Storage concurrency scheduler state stays inside the concurrency owner. |
+| `GLOBAL_KMS_DEK_PROVIDER`, `GLOBAL_SSE_DEK_PROVIDER` | `rustfs/src/storage/sse.rs` | DEK provider caches stay private to the SSE owner. |
+| `ECSTORE_EVENT_DISPATCH_HOOK` | `rustfs/src/server/event.rs` | Event bridge registration goes through the storage facade. |
+| `AUDIT_MODULE_ENABLED`, `NOTIFY_MODULE_ENABLED` | `rustfs/src/module_switches.rs` | Module toggles are read through module-switch helpers; `MODULE_SWITCH_RMW_LOCK` (`rustfs/src/server/module_switch.rs`) serializes persisted updates. |
+| `RUNTIME_CONFIG_RELOAD_MUTEX` | `rustfs/src/admin/service/config.rs` | Serializes dynamic config reload fanout. |
+| `EMBEDDED_RUNTIME_OWNERS` | `rustfs/src/startup_shutdown.rs` | Embedded runtime owner handles used for shutdown ordering. |
+| `SERVICE_FROZEN` | `rustfs/src/admin/handlers/system.rs` | Service freeze flag stays behind the system admin handler. |
+| `RECONCILER` | `rustfs/src/site_replication_reconcile.rs` | Site-replication reconciler singleton. |
+| `CONSOLE_CONFIG` | `rustfs/src/admin/console.rs` | Console bootstrap config. |
+| `LICENSE_STATE`, `LICENSE_VERIFIER` | `rustfs/src/license.rs` | License state and verifier stay behind license helpers. |
 
-## First Code-Bearing Candidate
+Regenerate the full list (long, mostly caches and test hooks):
 
-`GLOBAL_EXPIRY_STATE` is the safest first runtime migration candidate:
-
-- AppContext already exposes `ExpiryStateInterface` and resolver coverage in
-  `rustfs/src/app/context.rs`.
-- ECStore access is already concentrated in
-  `crates/ecstore/src/runtime/sources.rs`.
-- The main external readers can be moved through storage/observability facades
-  before changing lifecycle queue ownership.
-
-Do not migrate `GLOBAL_OBJECT_API` first. It is coupled to storage startup,
-object-store resolver publication, IAM-after-storage AppContext initialization,
-and broad data-plane compatibility.
-
-## Verification
-
-Inventory and guardrail PRs should run:
-
-- `bash -n scripts/check_architecture_migration_rules.sh`
-- `./scripts/check_architecture_migration_rules.sh`
-- `cargo fmt --all --check`
-- `git diff --check`
-
-Code-bearing migration PRs must add focused tests for the owner being moved
-before running broader gates.
+```bash
+rg -n '^\s*(pub(\(crate\))? )?static [A-Z_]+' rustfs/src
+```

docs/architecture/heal-concurrency-model.md

@@ -0,0 +1,77 @@
+# Heal concurrency model
+
+**Use this when:** changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that touches the same `(bucket, object)` commit surface; or evaluating whether RustFS needs a persistent per-object healing marker like MinIO's `x-minio-healing`.
+**Source of truth:** `crates/ecstore/src/set_disk/ops/heal.rs` (`heal_object_with_explicit_version_regen`, `HealObjectLockKind`, `HEAL_RENAME_INCOMPLETE`), `crates/ecstore/src/set_disk/ops/object.rs` (PUT/DELETE lock sections, `reconcile_old_data_cleanup_receipts`), `crates/ecstore/src/set_disk/core/io_primitives.rs` (`commit_rename_data_dir`, `report_old_data_dir_cleanup`, `reclaim_orphan_data_dirs`), `crates/filemeta/src/fileinfo.rs` (`FileInfo::set_healing`), `crates/heal/src/heal/manager/queue.rs` (dedup keys).
+
+## Model
+
+Heal and every foreground or background write path serialize on the same object-level namespace write lock (a quorum lock RPC in distributed mode, the in-process lock manager on a single node; granularity is the object, the version component is always `None`), and heal holds its guard across the whole rename commit. MinIO's `x-minio-healing` marker is an out-of-lock defence against version-cleanup logic inside `RenameData` interleaving with a heal commit; RustFS's commit model has no such interleaving, so no persistent marker exists (`x-minio-healing` does not occur in `crates/` or `rustfs/`) and none is needed. Three layers replace it:
+
+| Layer | Mechanism | Owner |
+| --- | --- | --- |
+| In-lock mutual exclusion | Heal and all write-path commit points take the `(bucket, object)` namespace write lock. | `acquire_heal_object_lock` in `crates/ecstore/src/set_disk/ops/heal.rs`; lock sections in `ops/object.rs` and `ops/multipart.rs` |
+| Commit-model isolation | `rename_data` contains no version cleanup that could interleave with heal. Physical deletion of a replaced old `data_dir` runs after the object lock is released (the commit tail) and only for unshared directories already superseded by the new commit. | `commit_rename_data_dir` in `crates/ecstore/src/set_disk/core/io_primitives.rs` |
+| Transient healing flag | `FileInfo::set_healing` sets the internal `SUFFIX_HEALING` key on the in-memory `FileInfo` of a heal commit; `rename_data` reads it through `is_healing` to clear a stale non-empty target `data_dir` before the rename (in-place repair reuses the `data_dir`, and `rename(2)` cannot replace a non-empty directory). The key is never persisted (`is_skip_meta_key` in `crates/filemeta/src/filemeta.rs`). A non-heal commit that meets a non-empty target fails explicitly; tests lock both directions. | `crates/filemeta/src/fileinfo.rs`, `crates/ecstore/src/disk/local.rs` |
+
+## Heal lock scope
+
+`heal_object` delegates to `heal_object_with_explicit_version_regen`, which takes the namespace write lock at entry unless `opts.no_lock` is set and binds the guard to the function scope. The guard covers the quorum metadata read, EC reconstruction, per-disk rename commit, tmp cleanup, the `HEAL_RENAME_INCOMPLETE` partial-commit return, and orphan `data_dir` reclamation (`reclaim_orphan_data_dirs`).
+
+Read-repair heals (`opts.read_repair`) hold a shared lock (`HealObjectLockKind::Read`) during reconstruction so readers keep flowing, then `acquire_revalidated_read_repair_commit_lock` takes the write lock and re-reads a commit fingerprint; a changed fingerprint aborts the commit (`read_repair_commit_stale`).
+
+## Lock-intersection matrix
+
+| # | Concurrent path | Lock held by that path | Outcome | Where |
+| --- | --- | --- | --- | --- |
+| 1 | PUT commit | object write lock; `rename_data` inside it | serialized | `ops/object.rs` put commit |
+| 2 | PUT old `data_dir` tail cleanup | none (runs after the lock is dropped) | unlocked, semantically safe ([commit tail](#commit-tail-cleanup)) | `commit_rename_data_dir` in `core/io_primitives.rs` |
+| 3 | DELETE object or version | object write lock; `delete_version` inside it | serialized | `ops/object.rs` `delete_object` |
+| 4 | Batch DELETE | per-object write locks (batch lock RPC in distributed mode) | serialized | `ops/object.rs` `delete_objects` |
+| 5 | CompleteMultipartUpload | object write lock plus upload-path lock; rename inside | serialized | `ops/multipart.rs` |
+| 6 | CompleteMultipart tail cleanup | none (after lock drop) | unlocked, semantically safe ([commit tail](#commit-tail-cleanup)) | `ops/multipart.rs` |
+| 7 | AbortMultipartUpload | upload-path lock in the multipart bucket only | disjoint resources: abort never touches the object `data_dir` or `xl.meta` | `ops/multipart.rs` |
+| 8 | ILM expiry including DeleteAllVersions | `delete_prefix_object=true` keeps the object lock; `FreeVersionTask` locks explicitly; noncurrent batches use batch locks | serialized | `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` |
+| 9 | Pure prefix delete | `delete_prefix` without `delete_prefix_object` takes no child-object lock | unlocked; no production caller ([prefix delete](#pure-prefix-delete)) | `ops/object.rs` lock condition in `delete_object` |
+| 10 | Orphan `data_dir` reclamation | none inside the function; its only production caller runs inside the heal lock | serialized within heal | `reclaim_orphan_data_dirs` in `core/io_primitives.rs` |
+| 11 | Old-cleanup receipt reconciliation | none inside the function; caller runs inside the heal lock and an epoch fence rejects stale receipts | serialized | `reconcile_old_data_cleanup_receipts` in `ops/object.rs` |
+| 12 | Replication | data plane writes to the remote over HTTP; local metadata write-back takes the object lock | serialized or disjoint | `crates/ecstore/src/bucket/replication/replication_resyncer.rs` |
+| 13 | Data movement, rebalance, decommission source cleanup | explicit object lock plus version-unchanged recheck; `no_lock` only reuses an already-held guard | serialized | `crates/ecstore/src/data_movement/mod.rs` |
+| 14 | CopyObject | destination object lock through the PUT chain | serialized | `ops/object.rs` `copy_object` |
+| 15 | Another heal task (different `HealType`, or `force_start`) | dedup keys are per `HealType` and `force_start` skips dedup, so tasks may coexist | serialized on the namespace write lock | `make_dedup_key_for_type` in `crates/heal/src/heal/manager/queue.rs` |
+| 16 | Admin heal with `nolock=true` | caller bypasses the lock | unlocked by operator choice ([no_lock](#no_lock-and-force_start)) | `rustfs/src/admin/handlers/heal.rs` |
+| 17 | Stale multipart cleanup | upload-path lock in the multipart bucket | disjoint resources | `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` |
+
+## Residual windows
+
+### Commit tail cleanup
+
+Rows 2 and 6. After a write path commits and releases the object lock, it best-effort deletes the replaced old `data_dir`; the code deliberately does not block the next operation on this. The deletion can race a concurrent heal reading or rebuilding that same old `data_dir`, but the race is semantically safe:
+
+- The target is an unshared `data_dir` already replaced by the new commit. Heal's canonical metadata comes from quorum arbitration (ETag, mod time), and quorum already points at the new version, so heal cannot resurrect the replaced version as canonical.
+- The worst outcome is one transient failure or no-op for the heal round on the old version; the next round converges. Cleanup residue is reported and re-queued for heal via `report_old_data_dir_cleanup`.
+- Long heals such as drive replacement request explicit versions and read quorum metadata inside the lock, so the tail does not affect them.
+
+### Pure prefix delete
+
+Row 9. `delete_prefix && !delete_prefix_object` takes no child-object locks (an object namespace lock cannot protect a recursive prefix delete), so a heal running during the prefix delete could theoretically rebuild a version from stale quorum metadata. Every production `delete_prefix: true` call site also sets `delete_prefix_object: true` (and therefore takes the object lock); the remaining `delete_prefix`-only call sites are in test modules. A future caller that needs a pure prefix delete must prove isolation from heal and scanner at the call site (for example a bucket-level scan fence).
+
+### `no_lock` and `force_start`
+
+Row 16. Admin heal requests pass the client's `nolock` parameter through (`rustfs/src/admin/handlers/heal.rs`), matching the MinIO madmin option. Setting it is an explicit operator choice that accepts races with concurrent writes; it is documented, not restricted.
+
+Heal-side invariants that hold regardless of the caller:
+
+- Dedup keys are disjoint across `HealType` (object, metadata, MRF, EC decode, prefix), and admin `force_start` skips dedup. Several heal tasks for one object can therefore exist at once, but every production entry calls `heal_object` with `no_lock=false`, so their execution bodies serialize on the namespace write lock.
+- Read-repair's local TTL reservation dedups only its own source and does not block heals from other sources; the namespace lock is the backstop.
+- The healing flag is never persisted, so there is no reverse risk of a leftover marker making a later commit yield incorrectly.
+
+## Regression tests
+
+Both live in the test module of `crates/ecstore/src/set_disk/ops/heal.rs`:
+
+| Test | Invariant |
+| --- | --- |
+| `heal_racing_version_delete_never_resurrects_the_deleted_version` | With a doomed version's shards corrupted, a versioned DELETE and a deep heal contend on the same lock; the deleted version is not resurrected and the surviving version is intact. |
+| `heal_racing_unversioned_overwrites_preserves_the_last_commit` | Unversioned overwrite commits (exercising the commit-tail old `data_dir` deletion) race a deep-heal loop; the final current version is exactly the last commit (ETag-level equality). |
+
+Related: the atomic-commit and best-effort-rollback invariants for the write path are in [erasure-coding.md](erasure-coding.md).

docs/architecture/kms-bulk-rekey-contract.md

@@ -1,184 +1,143 @@
 # KMS Bulk Rekey Job Contract
 
-This document defines the contract for the object-side bulk rekey job: a long-running administrative job that re-wraps stored data-key envelopes under the current key-encryption key (KEK) without rewriting object bodies. A first execution engine has shipped: the sweep in `rustfs/src/kms_rekey.rs`, driven by the admin endpoints in `rustfs/src/admin/handlers/kms_rekey.rs`. The contract remains the acceptance bar; where the shipped v1 sweep deliberately narrows it, the [Implementation Status](#implementation-status-v1-sweep) section records the deviation so the document and the tree cannot drift apart silently.
+**Use this when:** changing the bulk envelope re-wrap sweep (`rustfs/src/kms_rekey.rs`), its admin endpoints (`rustfs/src/admin/handlers/kms_rekey.rs`), the re-wrap primitive, or anything that decides which objects a rekey may touch.
+**Source of truth:** `rustfs/src/kms_rekey.rs`, `rustfs/src/admin/handlers/kms_rekey.rs`, `rewrap_object_encryption_metadata` in `rustfs/src/storage/sse.rs`, `KmsManager::rewrap_data_key` / `KmsManager::describe_data_key_wrapping` in `crates/kms/src/manager.rs`, `put_object_metadata` in `crates/ecstore/src/set_disk/ops/object.rs`.
 
-It tracks [`rustfs/backlog#1642`](https://github.com/rustfs/backlog/issues/1642), which lands the `bulk migrate/rekey` line of [`rustfs/backlog#1562`](https://github.com/rustfs/backlog/issues/1562).
+The bulk rekey job re-wraps stored data-key envelopes under the current key-encryption key (KEK) without rewriting object bodies. This document is the acceptance bar; where the shipped v1 sweep deliberately narrows it, [Implementation Status](#implementation-status-v1-sweep) records the deviation.
 
 ## Scope
 
-- Applies to: the job lifecycle, ownership, idempotency, failure semantics, exclusion rules, and completion evidence for bulk envelope re-wrap.
-- Out of scope, and deliberately so: the cryptographic definition of a single-object re-wrap (owned by the re-wrap primitive), master key material migration between KMS backends, a pause state, multi-node parallel execution, and destruction of superseded key versions.
-
-### Why master key material migration is not this job
-
-Vault Transit, AWS KMS, and HSM backends are designed so that key material cannot be exported. There is no path that moves a Local master key into Transit, and the reverse direction would export production key material from an HSM onto local disk, which is a security regression. The one case that is both possible and useful, Local to Local, is already served by the KMS backup and restore bundle in `crates/kms/src/backup/local_export.rs` and `crates/kms/src/backup/local_restore.rs`. Nothing in this contract creates a second, weaker copy of that capability.
+- Applies to: job lifecycle, ownership, idempotency, failure semantics, exclusion rules, and completion evidence for bulk envelope re-wrap.
+- Out of scope: the cryptographic definition of a single-object re-wrap (owned by the primitive), master key material migration between backends, a pause state, multi-node parallel execution, destruction of superseded key versions.
+- Master key material migration is not this job: Vault Transit, AWS KMS, and HSM backends do not export key material, and the one useful case (Local to Local) is already served by `crates/kms/src/backup/local_export.rs` and `crates/kms/src/backup/local_restore.rs`.
 
 ## Implementation Status (v1 Sweep)
 
-The shipped sweep (`rustfs/src/kms_rekey.rs`, admin surface `POST /rustfs/admin/v3/kms/keys/rekey` plus `/status` and `/cancel`, all gated on the cluster-scoped `kms:Rekey` action) implements the contract with these deliberate narrowings:
+The shipped sweep (`POST /rustfs/admin/v3/kms/keys/rekey` plus `/status` and `/cancel`, gated on the cluster-scoped `kms:Rekey` action) narrows the contract as follows:
 
-- **One sweep per process, not scope-scoped admission.** A single in-memory slot serializes sweeps cluster-wide on the node that received the request; a second start request is refused with the running job id. This is narrower than the scope-scoped ownership below — two disjoint-scope jobs cannot run concurrently — which is the safe direction: concurrent sweeps would double every KMS round-trip for zero extra coverage. The persisted CAS job record, lease, and crash-recovered ownership described under [Skeleton, Ownership, And Admission](#skeleton-ownership-and-admission) are not implemented; job state and counters are process-local and reset on restart. Correctness does not depend on them: the envelope itself is the resume state.
-- **Cursor-free convergence.** No checkpoint exists at all. The contract already declared the cursor a performance optimization; v1 takes that to its limit — recovery from a crash, cancel, or partial failure is re-running the sweep, and every already-current envelope costs one describe-shaped KMS call and no write.
-- **Backend gate at start.** The start endpoint refuses with `501` when the configured backend does not advertise `BackendCapabilities::rewrap`. Vault KV2 and Vault Transit pass; Local, Static, and AWS are refused. This is the "refused at admission" behavior the contract requires for AWS, and it is also what disarms the Local blocker below: a sweep can only run where superseded key versions demonstrably remain decryptable.
-- **Collapsed exclusion counting.** Plaintext objects, SSE-C objects, and MinIO-sealed envelopes are counted together as `not_applicable` rather than per-class; delete markers and directory entries are skipped without counting. Per-class exclusion counts remain future work.
-- **No dry run.** The dry-run report model below is not implemented; the closest present capability is reading `/status` counters from a completed sweep.
-- **Admission posture.** The sweep processes exactly one object at a time — each iteration awaits a KMS round-trip and, on rewrap, one metadata write — so its foreground contention is bounded by strict serialization, the KMS policy layer's shared concurrency cap, and the storage layer's own namespace locks and quorum rules. It does not integrate with a workload-admission mechanism, because [workload-admission-contracts.md](workload-admission-contracts.md) currently defines an observation-only snapshot surface repo-wide, with no runtime admission API for any background job to join. When such a mechanism exists, this job joins it alongside the scanner, heal, and decommission; until then, the requirement is bounded contention, which serialization provides.
+| Contract item | v1 behavior |
+|---|---|
+| Ownership / admission | One in-memory slot per process serializes sweeps; a second start request is refused with the running job id. No persisted CAS job record, lease, or crash-recovered ownership; counters are process-local and reset on restart. |
+| Resume cursor | None. Recovery from crash, cancel, or partial failure is re-running the sweep; every already-current envelope costs one describe-shaped KMS call and no write. |
+| Backend gate | Start refuses with `501` when the backend does not advertise `BackendCapabilities::rewrap` (`crates/kms/src/backends/mod.rs`). Vault KV2 and Vault Transit pass; Local, Static, and AWS are refused. |
+| Exclusion counting | Plaintext, SSE-C, and MinIO-sealed envelopes are counted together as `not_applicable`; delete markers and directory entries are skipped without counting. |
+| Dry run | Not implemented; the closest capability is `/status` counters from a completed sweep. |
+| Admission posture | Exactly one object at a time (one KMS round-trip, then at most one metadata write). No workload-admission integration: [workload-admission-contracts.md](workload-admission-contracts.md) defines an observation-only snapshot surface with no runtime admission API for a background job to join. |
 
-What v1 keeps exactly as contracted: work units are `(bucket, object, versionId)` with `latest_only: false`; `mod_time` is never set on the rewrap write; object-lock retention is inherited from `put_object_metadata`; the rewrap replaces every stored envelope copy by value match across the RustFS-internal and MinIO-compatible slots, and treats "no replaceable copy found" as an error rather than a silent success — the stale-branch hazard rule from [Metadata Write Contract](#metadata-write-contract); failures are counted and logged per object and never abort the sweep; cancellation is cooperative and terminal.
+Kept exactly as contracted: work units are `(bucket, object, versionId)` with `latest_only: false`; `mod_time` is never set on the rewrap write; object-lock retention is inherited from `put_object_metadata`; every stored envelope copy is replaced by value match across the RustFS-internal and MinIO-compatible slots, and "no replaceable copy found" is an error, not a silent success; failures are counted and logged per object and never abort the sweep; cancellation is cooperative and terminal.
 
 ## Terms
 
 | Term | Meaning |
 |---|---|
-| Envelope | The sealed data key (DEK) stored on an object version's metadata, together with the identifiers needed to unseal it. |
-| Re-wrap primitive | A single-object operation that unseals one envelope and re-seals it under the target KEK, changing metadata only. Implemented as `rewrap_object_encryption_metadata` in `rustfs/src/storage/sse.rs`, over `KmsManager::rewrap_data_key`. |
-| Rekey job | The scan-and-drive layer defined by this document, which applies the re-wrap primitive across a scope. |
+| Envelope | The sealed data key (DEK) stored on an object version's metadata, with the identifiers needed to unseal it. |
+| Re-wrap primitive | Single-object operation that unseals one envelope and re-seals it under the target KEK, changing metadata only: `rewrap_object_encryption_metadata` over `KmsManager::rewrap_data_key`. |
+| Rekey job | The scan-and-drive layer defined here, applying the primitive across a scope. |
 | Work unit | One `(bucket, object, versionId)` triple. Never `(bucket, object)`: each version carries its own envelope. |
-| Scope | The bucket and prefix selector that bounds one job, and the unit of admission exclusion. |
-| Target state | The envelope state the job is driving toward: sealed under the intended key id at the current KEK version. |
+| Scope | The bucket and prefix selector that bounds one job; the unit of admission exclusion. |
+| Target state | Envelope sealed under the intended key id at the current KEK version. |
 
-## What the Job Does And Does Not Do
+## What The Job Does And Does Not Do
 
-The job re-wraps envelopes. It never rewrites object bodies. Erasure-coded shards, part layout, ETag, and storage usage must be unchanged across a rekey; only encryption metadata keys may differ. Metadata-only rewrite is supported by the storage layer: `put_object_metadata` is declared on `ObjectStore` in `crates/ecstore/src/store/mod.rs`, dispatched in `crates/ecstore/src/core/sets.rs`, and implemented in `crates/ecstore/src/set_disk/ops/object.rs`, where it takes a namespace write lock, selects the version named by `opts.version_id`, and merges `opts.eval_metadata` into the existing `FileInfo` metadata under read and write quorum.
-
-**The job never destroys a superseded key version.** This is the hardest constraint in this contract, and every other guarantee rests on it. A job that fails halfway leaves some objects wrapped under the new KEK version and some under the old one. That state is fully serviceable — reads and writes both succeed — precisely and only because the old version can still decrypt. Destroying old versions from inside the job would convert a resumable operational action into irreversible data loss on partial failure. Destruction stays a separate, human-initiated operation gated on usage evidence.
-
-A job must therefore refuse to start when the target key's retention policy would allow the superseded version to leave the retention window while the job runs.
+- Re-wraps envelopes only. Erasure-coded shards, part layout, ETag, and storage usage are unchanged; only encryption metadata keys may differ. The metadata-only write is `put_object_metadata` (declared on `ObjectStore` in `crates/ecstore/src/store/mod.rs`, dispatched in `crates/ecstore/src/core/sets.rs`, implemented in `crates/ecstore/src/set_disk/ops/object.rs`).
+- **Never destroys a superseded key version.** A half-finished job leaves some envelopes under the new KEK version and some under the old; that state is serviceable only because the old version still decrypts. Destruction stays a separate, human-initiated operation gated on usage evidence.
+- Must refuse to start when the target key's retention policy would let the superseded version leave the retention window while the job runs.
 
 ## Idempotency Model
 
-Re-running the job must be safe and must converge. The intended source of idempotency is the object metadata itself: the envelope's own state is the target state, so a re-run reads what is already correct and skips it. No separate idempotency table is required, and the job identity is only a `job_id: Uuid` for reporting and ownership, following the ILM manual transition job record in `crates/ecstore/src/bucket/lifecycle/manual_transition_job.rs`.
+Idempotency comes from object metadata itself: the envelope's state is the target state, so a re-run reads what is already correct and skips it. No idempotency table; the job identity is a `job_id: Uuid` for reporting and ownership, following `ManualTransitionJobRecord` in `crates/ecstore/src/bucket/lifecycle/manual_transition_job.rs`.
 
-Two consequences follow, and both are contract requirements:
-
-- **The resume cursor is a performance optimization, not a correctness dependency.** Losing a checkpoint may cause a rescan and a higher skip count, never a wrong result. This is what makes crash recovery cheap: checkpoints may be throttled rather than written per object, following the `PersistThrottle` policy in `crates/heal/src/heal/resume.rs`, which flushes after a bounded number of buffered mutations or a bounded interval, whichever comes first. That module states the same reasoning for heal: because the operation is idempotent, a crash re-does at most one throttle window.
-- **The job is at-least-once with target-state idempotency, never exactly-once.** No design may introduce exactly-once machinery for work units.
+- **The resume cursor is a performance optimization, not a correctness dependency.** Losing a checkpoint may cause a rescan and a higher skip count, never a wrong result. Checkpoints may therefore be throttled (`PersistThrottle` in `crates/heal/src/heal/resume.rs`).
+- **At-least-once with target-state idempotency, never exactly-once.** No design may introduce exactly-once machinery for work units.
 
 ### Reading the wrapping KEK version
 
-The self-evidencing property above holds only when the wrapping KEK version is observable. It is, for every backend that actually rotates, but not from a dedicated metadata field and not by the same mechanism on each backend.
-
-There is no key-version metadata key: object metadata carries the key **id** (`x-rustfs-encryption-key-id` in `rustfs/src/storage/sse.rs`, defaulting to `default`) and the sealed blob under `x-rustfs-encryption-key`, and nothing else names a version. `DecryptResponse` in `crates/kms/src/types.rs` does not report one either, though `EncryptResponse` does.
-
-The version is nonetheless recoverable, because the sealed blob is structured. `x-rustfs-encryption-key` stores the base64 of the backend ciphertext, and for every backend that builds one that ciphertext is the JSON of `DataKeyEnvelope` (`crates/kms/src/encryption/dek.rs`). Reading it needs no new metadata: base64-decode the value, then parse the JSON. The read path in `rustfs/src/storage/sse.rs` already does exactly this discrimination, calling `is_data_key_envelope` on the decoded blob to pick a provider, so this is an established in-tree pattern rather than a new capability.
-
-Where the version sits inside that structure is backend-specific:
+There is no key-version metadata key. Object metadata carries the key id (`x-rustfs-encryption-key-id`) and the sealed blob under `x-rustfs-encryption-key`; `DecryptResponse` in `crates/kms/src/types.rs` does not report a version either. The version is recoverable because the sealed blob is structured: for every backend that builds one, the ciphertext is the JSON of `DataKeyEnvelope` (`crates/kms/src/encryption/dek.rs`), and the read path already discriminates on it via `is_data_key_envelope` in `rustfs/src/storage/sse.rs`.
 
 | Backend | Rotates | Where the wrapping version lives | Recoverable by a scan |
 |---|---|---|---|
-| Vault KV2 (`crates/kms/src/backends/vault.rs`) | Yes | `DataKeyEnvelope::master_key_version`, populated from the key record's version | Yes, from the envelope JSON |
-| Vault Transit (`crates/kms/src/backends/vault_transit.rs`) | Yes | The `vault:vN:` prefix of the ciphertext held in the envelope's `encrypted_key`; the envelope's own version field is deliberately `None` because Transit ciphertext self-describes | Yes, by parsing that prefix |
-| Local (`crates/kms/src/backends/local.rs`) | No — rotation is rejected | Nowhere; the version field is hardcoded `None` because a key has exactly one material | Moot while rotation is rejected |
-| Static (`crates/kms/src/backends/static_kms.rs`) | No — single fixed key | Nowhere; hardcoded `None` | Moot |
-| AWS (`crates/kms/src/backends/aws.rs`) | AWS-managed | Inside the opaque `CiphertextBlob`; no `DataKeyEnvelope` is built at all | **No** |
+| Vault KV2 (`crates/kms/src/backends/vault.rs`) | Yes | `DataKeyEnvelope::master_key_version` | Yes, from the envelope JSON |
+| Vault Transit (`crates/kms/src/backends/vault_transit.rs`) | Yes | `vault:vN:` prefix of the ciphertext in `encrypted_key`; the envelope's version field is deliberately `None` | Yes, by parsing that prefix |
+| Local (`crates/kms/src/backends/local.rs`) | No, rotation is rejected | Nowhere; hardcoded `None` | Moot while rotation is rejected |
+| Static (`crates/kms/src/backends/static_kms.rs`) | No | Nowhere; hardcoded `None` | Moot |
+| AWS (`crates/kms/src/backends/aws.rs`) | AWS-managed | Inside the opaque `CiphertextBlob`; no `DataKeyEnvelope` | **No** |
 
-Two traps follow, and both are contract rules.
+Contract rules that follow:
 
-**`None` does not mean one thing.** On Vault KV2 it means a pre-versioning envelope, and `resolve_envelope_master_key_version` resolves it to the key's recorded baseline version, or to the current version for a key that was never rotated — never implicitly to whatever is current now. On Transit it is permanent and expected, and the version must be read from the ciphertext prefix instead. On Local and Static it is unconditional. A scan that reads `None` as a single condition will misclassify three different situations, so version extraction must be dispatched by backend, never inferred from the field alone.
-
-**Local's `None` is coupled to the blocker below.** The Local backend omits the version specifically because rotation is rejected there. When [`rustfs/backlog#1565`](https://github.com/rustfs/backlog/issues/1565) gives Local a rotation history, that construction must begin recording the wrapping version in the same change, or Local silently becomes a second unreadable backend and loses idempotent skip along with it. This coupling is not obvious from either issue and must not be discovered later.
-
-The requirement this places on the re-wrap primitive is therefore narrower than "record a version", most of which the tree already satisfies:
-
-- The primitive must expose the wrapping version through **one backend-dispatched accessor** — satisfied by `KmsManager::describe_data_key_wrapping`, which dispatches per backend so callers never reimplement envelope-field or ciphertext-prefix parsing, which would also put KMS format knowledge on the wrong side of the crate boundary.
-- The primitive must report **"already at target state" as an outcome distinct from "re-wrapped"**, so the job counts a skip instead of inferring one.
-- For AWS, neither is achievable by inspection, and the contract must say so rather than pretend otherwise (see below).
-
-### The cost of recognizing the target state
-
-Skipping already-current objects is achievable, and it is not free. Every scanned work unit costs a base64 decode plus a JSON parse of its envelope, and on Transit an additional prefix parse. That is CPU and allocation per object version, not extra I/O: the metadata is already being read by the scan, and no KMS round trip is involved. Envelopes are small, so the cost is bounded per object, but at bulk scale it is the dominant cost of a dry run and of the skip check in a re-run, and it belongs in the rate and admission budget rather than being treated as free.
-
-This cost buys three things, all of which the contract requires and none of which are available without it: a re-run that skips completed work and performs zero metadata writes, a dry run that reports which KEK versions are actually in scope, and the per-object half of completion evidence.
-
-**AWS is the exception, and it is a scoping exception rather than a cost.** Its ciphertext is opaque to RustFS, so no inspection can tell a current envelope from a stale one. A rekey scope on an AWS-backed key therefore cannot skip, cannot report version composition in a dry run, and cannot self-evidence completion; a re-run would re-wrap every object again. AWS also rotates backing key material transparently on decrypt, so the operational need that motivates this job is weaker there to begin with. Until there is a reason to do otherwise, AWS-backed keys are out of scope for bulk rekey, and a job must refuse such a scope at admission rather than start one whose re-runs silently rewrite everything.
+- **`None` does not mean one thing.** KV2: pre-versioning envelope, resolved by `resolve_envelope_master_key_version` to the key's recorded baseline, never implicitly to "current". Transit: permanent and expected; read the ciphertext prefix. Local/Static: unconditional. Version extraction must be dispatched by backend, never inferred from the field alone.
+- **Local's `None` is coupled to the Local blocker.** If Local gains rotation history (`rustfs/backlog#1565`), envelope version recording must land in the same change, or Local becomes a second unreadable backend.
+- The primitive exposes the wrapping version through **one backend-dispatched accessor** (`KmsManager::describe_data_key_wrapping`) and reports **"already at target state" as an outcome distinct from "re-wrapped"**.
+- **AWS is a scoping exception.** Its ciphertext is opaque, so no scan can skip, report version composition, or self-evidence completion; a re-run would rewrap everything. AWS-backed keys are out of scope and must be refused at admission.
+- Skip detection costs a base64 decode plus JSON parse (plus a prefix parse on Transit) per work unit: CPU, not I/O, and part of the rate budget rather than free.
 
 ## Failure Semantics
 
-A partially complete rekey is a valid, serviceable state, not a damaged one. It requires no emergency handling, no fail-closed startup guard, and no rollback. This is the sharpest difference from KMS backup restore, whose intermediate state genuinely is unserviceable and which therefore fails closed on startup when its commit marker is present.
-
-The precondition is that superseded key versions remain decryptable. Where that precondition does not hold, the whole model collapses (see Blockers).
-
-Cancellation is cooperative and terminal. A canceled job reaches a terminal state with already-processed objects left in the target state; restarting on the same scope skips them.
-
-## Pause Is Not Provided
-
-The originating requirement asked for pause, resume, and idempotent retry. This contract provides cancel, cursor restart, and rate control instead, and does not provide a pause state.
-
-Seven long-running job frameworks exist in the tree — ILM manual transition, heal resume (`crates/heal/src/heal/resume.rs`), tier mutation intent (`crates/ecstore/src/services/tier/tier_mutation_intent.rs`), decommission and rebalance (`crates/ecstore/src/core/pools.rs`), the scanner (`crates/scanner/src/scanner.rs`), and KMS backup restore. None of them has a pause state; each has cancel or stop only. That consistency is a design position, not an oversight. A paused job has to answer what it still holds: whether its lease is renewed, whether it keeps its scope admission slot, and how long it may stay paused before it is abandoned. Each answer adds state and a failure mode.
-
-The two things pause is actually asked for are that the job must not overwhelm the data path, and that stopping it must not throw away progress. Rate and admission control delivers the first; cancel plus cursor restart delivers the second. Both are existing patterns.
+- A partially complete rekey is a valid, serviceable state: no emergency handling, no fail-closed startup guard, no rollback. This is the sharpest difference from KMS backup restore, whose intermediate state is unserviceable and fails closed on startup.
+- Precondition: superseded key versions remain decryptable (see Blockers).
+- Cancellation is cooperative and terminal; restarting on the same scope skips already-processed objects.
+- Pause is not provided. None of the tree's long-running job frameworks (ILM manual transition, heal resume, tier mutation intent, decommission/rebalance, scanner, KMS restore) has a pause state; rate control plus cancel-and-restart deliver what pause is asked for without lease/slot/abandonment state.
 
 ## Objects That Cannot Be Rekeyed
 
-These must be enumerated during the scan and excluded with a counted reason. Encountering one is never a job failure, and the execution phase must not touch them.
+Enumerated during the scan and excluded with a counted reason; never a job failure; the execution phase must not touch them.
 
 | Class | Disposition | Reason |
 |---|---|---|
-| SSE-C objects | Exclude and count | The server never holds the customer key, so it can neither unseal nor re-seal the envelope. |
-| Objects transitioned to a remote tier | Exclude and count | The body lives remotely; the relationship between local metadata and the remote object's encryption needs its own analysis first. See [tier-ilm-debugging.md](../operations/tier-ilm-debugging.md). |
-| In-progress multipart uploads | Exclude and count | Each part carries its own envelope and an incomplete upload is not a stable work unit. `crates/kms/src/key_impact.rs` already models this as a distinct reference scope. |
-| Unencrypted objects | Exclude and count | No envelope to re-wrap. |
-| Objects under object-lock retention | Governed by the storage layer, see below | |
+| SSE-C objects | Exclude and count | The server never holds the customer key. |
+| Objects transitioned to a remote tier | Exclude and count | Body lives remotely; see [tier-ilm-debugging.md](../operations/tier-ilm-debugging.md). |
+| In-progress multipart uploads | Exclude and count | Each part carries its own envelope; `crates/kms/src/key_impact.rs` models this as a distinct reference scope. |
+| Unencrypted objects | Exclude and count | No envelope. |
+| Objects under object-lock retention | Governed by the storage layer (see Metadata Write Contract) | |
 
-Replication destinations are unresolved: whether an envelope metadata rewrite must propagate to a replica depends on [`rustfs/backlog#1619`](https://github.com/rustfs/backlog/issues/1619). Until that closes, this contract does not authorize propagation.
+Replication destinations are unresolved: propagation depends on `rustfs/backlog#1619`. Until it closes, a rewrap never propagates to a replica and each site runs its own sweep.
 
 ## Metadata Write Contract
 
-Three properties of `put_object_metadata` constrain the re-wrap write, all confirmed in `crates/ecstore/src/set_disk/ops/object.rs`.
+Three properties of `put_object_metadata` (`crates/ecstore/src/set_disk/ops/object.rs`) constrain the re-wrap write:
 
-**The merge is additive; it cannot remove keys.** `opts.eval_metadata` entries are inserted into the existing metadata map. There is no removal path. Overwriting a key that keeps its name is therefore safe, but a re-wrap that changes *which* metadata keys describe the envelope leaves the old keys behind permanently.
-
-That is a structural hazard, not a theoretical one. `rustfs/src/storage/sse.rs` selects its decrypt branch on the mere presence of the MinIO-compatible seal-algorithm header: `parse_minio_managed_sealed_key` returns a sealed key whenever that header is present with the expected value, and the caller then takes the MinIO branch in preference to the RustFS-native one. A re-wrap that writes a RustFS-native envelope onto an object carrying MinIO-compatible headers, without clearing them, steers subsequent reads down the stale branch. Any re-wrap that changes envelope shape must neutralize the superseded keys in the same write, and cannot rely on deletion to do it.
-
-**Object-lock retention is enforced before the merge.** `check_object_lock_retention_update`, defined in `crates/ecstore/src/set_disk/mod.rs`, runs before `eval_metadata` is applied. Rekey inherits that decision rather than restating it: whatever that check permits for a metadata update, rekey permits; whatever it refuses, rekey counts as an exclusion. Rekey must not acquire a bypass.
-
-**`mod_time` is preserved unless the caller sets it.** The implementation assigns `fi.mod_time` only when `opts.mod_time` is `Some`. The re-wrap path must leave it unset, so that a rekey does not perturb lifecycle rule evaluation — an age-based expiry or transition rule reading a refreshed `mod_time` across a whole bucket would be a cross-feature regression.
+- **The merge is additive; it cannot remove keys.** Overwriting a key that keeps its name is safe; a re-wrap that changes *which* keys describe the envelope leaves the old keys behind. This is a live hazard: `parse_minio_managed_sealed_key` in `rustfs/src/storage/sse.rs` selects the MinIO decrypt branch on the mere presence of the MinIO seal-algorithm header, so a RustFS-native envelope written onto MinIO-compatible headers without neutralizing them steers reads down the stale branch. Any envelope-shape change must neutralize superseded keys in the same write.
+- **Object-lock retention is enforced before the merge.** `check_object_lock_retention_update` (`crates/ecstore/src/set_disk/mod.rs`) runs first; rekey inherits its decision and must not acquire a bypass.
+- **`mod_time` is preserved unless the caller sets it.** The re-wrap path leaves it unset so age-based lifecycle rules are not perturbed.
 
 ## Skeleton, Ownership, And Admission
 
-The ILM manual transition job is the structural template. `ManualTransitionJobRecord` in `crates/ecstore/src/bucket/lifecycle/manual_transition_job.rs` already carries `job_id`, `scope_key`, `owner_id`, `lease_id` with an expiry, a state machine including an explicit `Unknown` state for a corrupt journal, `cancel_requested`, a report, and a queue snapshot. Records are persisted under dedicated metadata-bucket prefixes with a schema string and checksum, and mutated with S3 conditional writes (`if_match` for updates, `if_none_match` for creates) so that ownership transitions are compare-and-swap rather than last-write-wins. Crash recovery, cooperative cancel via `request_manual_transition_job_cancel`, and capability advertisement through `ManualTransitionJobCapabilities` in `rustfs/src/admin/handlers/system.rs` all follow from that shape.
-
-Ownership is scope-scoped, not cluster-scoped. Two jobs on disjoint scopes may run concurrently; two jobs on the same scope must be refused by admission. The scanner's leader lock with epoch fencing in `crates/scanner/src/scanner.rs` is the wrong granularity here because it enforces exactly one worker per cluster; it stays a reference for fencing technique only.
-
-The first implementation is single-node: one owner plus a lease plus recovery is sufficient for correctness. Multi-node parallel execution is a throughput optimization and is out of scope until correctness and its acceptance evidence are both in place.
-
-Because the job runs online, it must not contend its way into the foreground data path. The v1 posture — strict serialization plus the KMS policy layer's shared cap and the storage layer's own locks — and the reason no workload-admission mechanism is joined yet are recorded under [Implementation Status](#implementation-status-v1-sweep); when a runtime admission mechanism exists per [workload-admission-contracts.md](workload-admission-contracts.md), this job joins it alongside the scanner, heal, and decommission.
-
-## What Is Taken From KMS Backup, And What Is Not
-
-Four things transfer:
-
-- The durable file commit protocol in `crates/kms/src/backends/local.rs` — write, fsync the file, publish by rename or hard link, fsync the parent directory — together with its injectable `CommitStep` failpoints.
-- The write-receipt ownership proof in `crates/kms/src/backup/vault_restore.rs`. Its distinction is the reusable idea: the list of intended targets proves nothing about ownership, and only a receipt recording the version a write actually landed at may authorize touching that record later; everything else is reported as never-written or not-at-written-version. Bulk rekey faces the identical problem when a concurrent writer modifies an object between the job's read and its write-back. Such an object must be counted as a conflict and skipped, never overwritten.
-- The sequence guard `VaultRestoreSequence` in the same module: a small, domain-free state machine that makes phase order structural. Rekey's phases are scan, plan, apply, verify.
-- The three-part dry-run report model in `crates/kms/src/backup/dry_run.rs` — blockers, conflicts, and external mismatches, with a permission predicate that requires all three to be empty — and its zero-write contract: the report is pure data with no handles and no drop-time side effects.
-
-The lifecycle model does not transfer, and must not be adapted. Backup restore is synchronous, one-shot, single-node, requires an empty target, has no progress surface, and requires the KMS service to be out of `Running` state; its admin layer says as much in `rustfs/src/admin/handlers/kms_backup.rs`. Its commit marker enumerates every file up front, which does not scale to object counts. Its publish primitive is no-clobber, whereas rekey rewrites existing state by definition. Forcing rekey into that four-phase protocol produces an all-or-nothing transaction over the whole scope, which is not operable at this scale.
-
-The job also does not belong in the KMS crate. `crates/kms/Cargo.toml` does not depend on `rustfs-ecstore` and must not: the job body is object scanning and metadata rewriting, which is ecstore and admin territory. The KMS crate supplies the re-wrap primitive only.
+- Structural template: `ManualTransitionJobRecord` (`job_id`, `scope_key`, `owner_id`, `lease_id` with expiry, state machine with explicit `Unknown`, `cancel_requested`, report, queue snapshot), persisted with S3 conditional writes so ownership transitions are compare-and-swap; capability advertisement via `ManualTransitionJobCapabilities` in `rustfs/src/admin/handlers/system.rs`.
+- Ownership is scope-scoped: disjoint scopes may run concurrently; same-scope jobs are refused by admission. The scanner leader lock in `crates/scanner/src/scanner.rs` is the wrong granularity (one worker per cluster) and is a fencing reference only.
+- First implementation is single-node; multi-node parallelism is a throughput optimization deferred until correctness evidence exists.
+- Taken from KMS backup: the durable file commit protocol in `crates/kms/src/backends/local.rs` (`CommitStep` failpoints); the write-receipt ownership proof and `VaultRestoreSequence` phase guard in `crates/kms/src/backup/vault_restore.rs` (a concurrent writer between read and write-back is a conflict-and-skip, never an overwrite); the three-part zero-write dry-run report model in `crates/kms/src/backup/dry_run.rs`. Not taken: the synchronous, empty-target, all-or-nothing restore lifecycle. The job does not belong in `crates/kms` (which must not depend on `rustfs-ecstore`); KMS supplies the primitive only.
 
 ## API Surface
 
-This section originally required reusing the MinIO-compatible batch-job endpoints in `rustfs/src/admin/handlers/batch_job.rs` and forbade a second REST surface. The shipped v1 superseded that rule: the sweep landed on RustFS-specific endpoints (`/v3/kms/keys/rekey`, `/status`, `/cancel`), reviewed and merged with the engine. The batch-job surface parses MinIO's full job-definition format, whose semantics (per-job flags, retries, notifications) the v1 sweep does not implement — and accepting a job definition whose semantics cannot be executed is exactly what this section forbids.
-
-The rule that survives is about live semantics, not endpoint shape: **one operation must never have two live semantics.** Today there is one live surface (the RustFS endpoints) and one refusing stub — `KNOWN_JOB_TYPES` in `batch_job.rs` still lists `keyrotate`, and `start-job` still returns a deliberate `NotImplemented`, unknown types get `InvalidRequest`, `list-jobs` returns an empty list, and status, describe, and cancel return a no-such-job error. That `NotImplemented` remains an external promise: the batch-job `keyrotate` type must keep refusing until it either proxies to this same engine with full batch-job semantics or is removed. It must never report success while it executes nothing, and it must never grow a second, divergent rekey implementation.
+- Live surface: the RustFS endpoints above. The MinIO-compatible batch-job surface (`rustfs/src/admin/handlers/batch_job.rs`) still lists `keyrotate` in `KNOWN_JOB_TYPES` and returns a deliberate `NotImplemented` from `start-job`.
+- Rule: **one operation must never have two live semantics.** The batch-job `keyrotate` type must keep refusing until it proxies to this engine with full batch-job semantics or is removed; it must never report success while executing nothing.
 
 ## Completion Evidence
 
-A job that reports success has not proven anything until no object in the scope still references the superseded key version. That evidence surface is the key usage inventory, whose typed foundation already exists in `crates/kms/src/key_impact.rs`. That module is deliberately built so a report can never claim a key is unused: it has no `in_use`, no `unreferenced`, and no `safe_to_delete` field, and instead reports which sources were consulted and how completely they could be read. It lists object envelopes and in-progress multipart uploads among its reference scopes and currently marks both as not scanned.
-
-Rekey must inherit that discipline. An empty result means nothing was found in the sources that were scanned, never that nothing references the key. A report that cannot state its own coverage is not completion evidence, and must not be used to authorize destroying anything.
+Completion is proven only when no object in scope still references the superseded key version. The evidence surface is the key usage inventory in `crates/kms/src/key_impact.rs`, which deliberately has no `in_use` / `unreferenced` / `safe_to_delete` field and instead reports which sources were consulted and how completely. Rekey inherits that discipline: an empty result means nothing was found in the sources scanned, never that nothing references the key.
 
 ## Blockers
 
-**Resolved by capability gating — Local rotation history.** [`rustfs/backlog#1565`](https://github.com/rustfs/backlog/issues/1565) (no rotation history in the Local backend) was a hard blocker while a sweep could run against Local: without retained superseded versions, a rekey interrupted halfway would leave every unprocessed object permanently unreadable after rotation, falsifying the partial-completion guarantee this contract is built on. The shipped resolution is not rotation history but scope: the Local backend is positioned as non-production, rotation stays rejected there, and the sweep's start endpoint refuses any backend that does not advertise `BackendCapabilities::rewrap` — so a sweep can only run where the retained-versions invariant holds by construction (Vault KV2 and Vault Transit). If Local ever gains rotation, the coupling recorded under [Reading the wrapping KEK version](#reading-the-wrapping-kek-version) still applies: rotation history and envelope version recording must land in the same change before Local may advertise `rewrap`.
-
-**Resolved — the execution chain is complete.** The envelope-level primitive (`KmsManager::rewrap_data_key`, `KmsManager::describe_data_key_wrapping` in `crates/kms/src/manager.rs`), the object-level adapter (`rewrap_object_encryption_metadata` in `rustfs/src/storage/sse.rs`, which reads a version's envelope, reconstructs its encryption context, re-wraps, and returns the metadata overrides), and the sweep that drives the adapter and persists through `put_object_metadata` (`rustfs/src/kms_rekey.rs`) all exist.
-
-**Affects acceptance, not start — still open.** Key usage inventory coverage over object envelopes: `crates/kms/src/key_impact.rs` still reports `ObjectEnvelopes` and `InProgressMultipartUploads` as not scanned, so a completed sweep's counters are evidence from that run only, not inventory-grade completion proof. KMS key list pagination, which a job enumerating keys would hit. And [`rustfs/backlog#1619`](https://github.com/rustfs/backlog/issues/1619), which decides replica propagation — until it closes, a rewrap never propagates to a replica site and each site runs its own sweep.
+| Item | Status |
+|---|---|
+| Local rotation history (`rustfs/backlog#1565`) | Resolved by capability gating: Local stays non-production, rotation stays rejected, and the start endpoint refuses any backend without `BackendCapabilities::rewrap`. If Local ever gains rotation, envelope version recording must land in the same change. |
+| Execution chain | Resolved: primitive (`rewrap_data_key`, `describe_data_key_wrapping`), object adapter (`rewrap_object_encryption_metadata`), sweep (`rustfs/src/kms_rekey.rs`). |
+| Key usage inventory coverage | Open: `key_impact.rs` still reports `ObjectEnvelopes` and `InProgressMultipartUploads` as not scanned, so sweep counters are evidence from that run only. |
+| KMS key list pagination | Open; a job enumerating keys would hit it. |
+| Replica propagation (`rustfs/backlog#1619`) | Open; no propagation until it closes. |
 
 ## Verification Expectations
 
-This list is the acceptance bar for the full contract, not a claim about what the v1 sweep has already demonstrated: the dry-run and checkpoint items await the features themselves (a cursor-free sweep satisfies the checkpoint-deletion clause vacuously), and per-class exclusion counting is narrowed as recorded under [Implementation Status](#implementation-status-v1-sweep).
+Acceptance bar for the full contract (dry-run and checkpoint items await those features; a cursor-free sweep satisfies the checkpoint clause vacuously):
 
-Implementation work under this contract must be able to demonstrate, at minimum: that dry run performs zero storage writes; that non-rekeyable objects are excluded and counted rather than failing the job; that an immediate second run skips every object and writes no metadata, on both a KV2-backed and a Transit-backed scope, since the two recover the wrapping version by different mechanisms; that a scope on an AWS-backed key is refused at admission rather than accepted as a job whose re-runs rewrite everything; that an envelope with no recorded version is classified by backend rather than by the bare `None`; that deleting the checkpoint changes only the skip count, not the outcome; that a killed and recovered job reaches a terminal state while every object remains readable throughout; that a concurrent writer causes a conflict-and-skip rather than an overwrite; that ETag, part layout, and storage usage are unchanged at the `xl.meta` level; that each version of a multi-version object is processed independently with its `versionId` intact; that superseded key versions still exist and still decrypt afterward; and that success, skip, exclusion, conflict, and failure counts sum to the number of work units scanned.
+1. Dry run performs zero storage writes.
+2. Non-rekeyable objects are excluded and counted rather than failing the job.
+3. An immediate second run skips every object and writes no metadata, on both a KV2-backed and a Transit-backed scope.
+4. A scope on an AWS-backed key is refused at admission.
+5. An envelope with no recorded version is classified by backend, not by the bare `None`.
+6. Deleting the checkpoint changes only the skip count, not the outcome.
+7. A killed and recovered job reaches a terminal state while every object stays readable throughout.
+8. A concurrent writer causes conflict-and-skip, not an overwrite.
+9. ETag, part layout, and storage usage are unchanged at the `xl.meta` level.
+10. Each version of a multi-version object is processed independently with its `versionId` intact.
+11. Superseded key versions still exist and still decrypt afterward.
+12. Success, skip, exclusion, conflict, and failure counts sum to the number of work units scanned.

docs/architecture/minio-file-format-compat.md

@@ -1,416 +1,148 @@
-# MinIO File-Format Interoperability — Gap Analysis & Phased Plan
+# MinIO On-Disk Format Interoperability
 
-Assesses how closely the RustFS on-disk format matches MinIO's, so that a
-MinIO drive set can be read (and eventually served) by RustFS and vice versa.
-This is a **plan and analysis document**. It changes no storage code. Every
-claim below cites the code that backs it.
+**Use this when:** deciding whether a MinIO drive set, bucket-metadata blob, or SSE object can be read or imported by a given RustFS build, or before touching any constant or codec listed under Version Anchors.
+**Source of truth:** `crates/filemeta/src/filemeta.rs`, `crates/filemeta/src/filemeta/codec.rs`, `crates/ecstore/src/bucket/metadata.rs`, `crates/ecstore/src/bucket/migration.rs`, `rustfs/src/storage/sse.rs`, `rustfs/Cargo.toml` `[features]`, `.github/workflows/ci.yml`, `.github/workflows/minio-interop.yml`.
 
-Scope: the two on-disk artifacts that matter for interop are the per-object
-`xl.meta` (object metadata + inline data) and the per-bucket `.metadata.bin`
-(bucket configuration blob). IAM/config layout is noted where it affects
-bucket-metadata migration.
+This is an interop contract, not a plan. Migration is one-way (MinIO to RustFS). Erasure-coding internals are owned by [erasure-coding.md](erasure-coding.md); this document owns the interop claim, the fixture evidence, and the out-of-scope list.
 
-Refs rustfs/backlog#580.
+## Scope Matrix By Build Variant
 
-## Executive Summary
+Build variants are the `rustfs` crate features in `rustfs/Cargo.toml`: `default`, `full`, and `rio-v2` (which enables `rustfs-ecstore/rio-v2` and pulls in `crates/rio-v2`). `rio-v2` is absent from both `default` and `full`.
 
-- **`xl.meta`**: RustFS writes `XL_META_VERSION = 3` and reads meta_ver ≤ 3,
-  including legacy meta_ver 2 objects with legacy checksums. Magic `XL2 `,
-  erasure algorithm `rs-vandermonde` (Reed-Solomon), and HighwayHash256 bitrot
-  all match MinIO. `xl.meta` interop is the **strong** part of the story.
-- **`.metadata.bin`**: RustFS uses the same filename, the same 4-byte
-  `format|version` header, the same MessagePack blob layout, and the same
-  per-config field encodings (XML/JSON) as MinIO's `bucketMetadata`. The
-  divergence is a small set of RustFS-only fields (table-bucket support,
-  bucket-targets meta) — not a format mismatch.
-- **Migration**: RustFS already ships a one-way importer that reads a legacy
-  meta bucket and rewrites bucket-metadata + IAM config into the RustFS meta
-  bucket (`crates/ecstore/src/bucket/migration.rs`).
-- **Server-side encryption**: not covered by the above. Objects MinIO wrote with SSE-S3, SSE-KMS, or SSE-C are **not readable by RustFS** in any shipped build. See [Part C](#part-c--server-side-encryption-sse) before planning a migration that includes encrypted objects.
+| MinIO artifact | `default` / `full` build | `rio-v2` build | Notes |
+|---|:--:|:--:|---|
+| Unencrypted `xl.meta` (meta_ver 1-3, inline, multipart, versioned, delete marker) | Read | Read | Part A. Normalized to meta_ver 3 on rewrite. |
+| Transitioned (tiered) `xl.meta` | Not fixture-proven | Not fixture-proven | Out of scope; see erasure-coding.md for the tolerant `transitioned-versionID` read rule. |
+| `.metadata.bin` bucket config | Read and imported | Read and imported | Part B. Importer reads a `.minio.sys` layout end to end. |
+| IAM config under `config/iam/` | Imported | Imported | `try_migrate_iam_config`; legacy field aliases normalized. |
+| SSE-S3 / SSE-KMS objects, MinIO builtin static KMS | Fail closed, diagnosed | Read | Part C. Requires the shared master key. |
+| SSE-C objects | Fail closed, diagnosed | Read | Part C. Customer key supplied per request. |
+| Any SSE object, MinIO backed by KES / KMS plugin / MinKMS | Fail closed | Fail closed | Not planned; the DEK is sealed by the KES service. |
+| RustFS-written drive set read by a live MinIO binary | Unsupported | Unsupported | Set-level divergence: MinIO looks for `.minio.sys`, RustFS writes `.rustfs.sys`. |
+| RustFS-written SSE objects read by MinIO | Unsupported | Unsupported | Part C, reverse direction. |
 
-For unencrypted objects the remaining work is verification breadth and closing
-per-config parsing gaps, not a format rewrite. Encrypted objects are a separate,
-unsolved axis (rustfs/backlog#1638).
+## Version Anchors
 
----
+These constants are compatibility anchors. Bumping any of them requires a read-compat path for the prior value and a migration story, exactly as the meta_ver 2 to 3 read path provides. Values live in code; do not copy them elsewhere.
+
+| Anchor | Symbol | File | Rule |
+|---|---|---|---|
+| `xl.meta` magic | `XL_FILE_HEADER` | `crates/filemeta/src/filemeta.rs` | Must equal MinIO's XL2 magic. |
+| Container major / minor | `XL_FILE_VERSION_MAJOR`, `XL_FILE_VERSION_MINOR` | `crates/filemeta/src/filemeta.rs` | `check_xl2_v1` (`crates/filemeta/src/filemeta/codec.rs`) rejects `major > XL_FILE_VERSION_MAJOR`. |
+| Header version | `XL_HEADER_VERSION` | `crates/filemeta/src/filemeta.rs` | `decode_xl_headers` rejects `header_ver > XL_HEADER_VERSION`. |
+| Metadata version | `XL_META_VERSION` | `crates/filemeta/src/filemeta.rs` | Written by `FileMeta::new`; `decode_xl_headers` rejects `meta_ver > XL_META_VERSION` (accept-older, reject-newer). |
+| Bucket metadata header | `BUCKET_METADATA_FORMAT`, `BUCKET_METADATA_VERSION` | `crates/ecstore/src/bucket/metadata.rs` | Checked by `check_header`; both match MinIO's `bucketMetadataFormat` / `bucketMetadataVersion`. |
+| Erasure algorithm string | `ERASURE_ALGORITHM` | `crates/ecstore/src/object_api/mod.rs` | `rs-vandermonde`; enum `ErasureAlgo` in `crates/filemeta/src/fileinfo.rs`. |
+| Meta bucket names | `RUSTFS_META_BUCKET`, `MIGRATING_META_BUCKET`, `BUCKET_META_PREFIX` | `crates/ecstore/src/disk/mod.rs` | `.rustfs.sys` is the live meta bucket; `.minio.sys` is the importer source. |
 
 ## Part A — `xl.meta` Object Format
 
-### Version support
-
-| Aspect | Value | Evidence |
+| Aspect | Contract | Where |
 |---|---|---|
-| Write version (`meta_ver`) | 3 | `crates/filemeta/src/filemeta.rs:54` (`XL_META_VERSION = 3`), written in `FileMeta::new` at `crates/filemeta/src/filemeta.rs:121` |
-| Read versions accepted | ≤ 3 (1, 2, 3) | Decode rejects only `meta_ver > XL_META_VERSION` — see `crates/filemeta/src/filemeta/codec.rs` (`decode_xl_headers`); `load_or_convert` doc at `crates/filemeta/src/filemeta.rs:864` |
-| Legacy meta_ver 2 read | Supported (with legacy checksum) | Regression fixtures `test_issue_2265_legacy_meta_v2_object_compatibility` / `test_issue_2288_legacy_xlmeta_compatibility` at `crates/filemeta/src/filemeta.rs:1130`, `:1152`; `uses_legacy_checksum` asserted at `:1174` |
+| Version probe | `read_format_versions` returns `(major, minor, header_ver, meta_ver)` without a full parse | `crates/filemeta/src/filemeta/codec.rs` |
+| Read compatibility | Accepts meta_ver 1-3 including legacy meta_ver 2 with legacy checksums (`uses_legacy_checksum`); `load_or_convert` normalizes on rewrite | `crates/filemeta/src/filemeta.rs`, `crates/ecstore/src/set_disk/read.rs` |
+| Container layout | 8-byte header, bin-length-prefixed msgpack header block, CRC trailer, optional inline data (MinIO XL2 v1 shape) | `crates/filemeta/src/filemeta/codec.rs` |
+| Erasure coding | Reed-Solomon Vandermonde, `rs-vandermonde` identifier, codec crate `rustfs-erasure-codec` | `Cargo.toml`, [erasure-coding.md](erasure-coding.md) |
+| Bitrot | `HighwayHash256S` default; `HighwayHash256SLegacy` (fixed key) for older shards | `crates/ecstore/src/io_support/bitrot.rs`, `crates/ecstore/tests/legacy_bitrot_read_test.rs` |
+| Inline data | Inline block after the CRC trailer; `null` / version-id keying via `data_key_for_version`; `physical_data_dir` accounting | `crates/filemeta/src/filemeta.rs`, `crates/filemeta/src/filemeta/inline_data.rs` |
 
-RustFS is a **read-forward-compatible** consumer of MinIO's `xl.meta`: it can
-parse older MinIO objects and normalizes them to meta_ver 3 on rewrite. It does
-not write MinIO's older versions.
-
-### Container header
-
-| Field | RustFS value | Evidence |
-|---|---|---|
-| Magic | `XL2 ` (`[b'X', b'L', b'2', b' ']`) | `crates/filemeta/src/filemeta.rs:46` |
-| File version major / minor | 1 / 3 | `crates/filemeta/src/filemeta.rs:51-52` |
-| Header version | 3 | `crates/filemeta/src/filemeta.rs:53` |
-| Magic + version check (decode entry) | `check_xl2_v1` validates magic and rejects `major > 1` | `crates/filemeta/src/filemeta/codec.rs:45-61` |
-| Version-only probe (no full parse) | `read_format_versions` returns `(major, minor, header_ver, meta_ver)` | `crates/filemeta/src/filemeta/codec.rs:30-43` |
-
-The layout after the 8-byte header is `bin-length-prefixed msgpack header block`
-followed by a CRC trailer and optional inline data — matching MinIO's XL2 v1
-container.
-
-### Erasure coding
-
-| Aspect | Value | Evidence |
-|---|---|---|
-| Algorithm enum | `ErasureAlgo::ReedSolomon = 1` | `crates/filemeta/src/fileinfo.rs:83-106` |
-| Algorithm string | `rs-vandermonde` | `crates/filemeta/src/fileinfo.rs:31` (`ERASURE_ALGORITHM`); also `crates/ecstore/src/object_api/mod.rs:52` |
-| Codec crate | `rustfs-erasure-codec` (Reed-Solomon, SIMD) | `Cargo.toml:277` |
-
-Same Reed-Solomon Vandermonde scheme and identifier string as MinIO.
-
-### Bitrot / shard integrity
-
-| Aspect | Value | Evidence |
-|---|---|---|
-| Default hash | `HashAlgorithm::HighwayHash256S` | Bitrot read/write paths in `crates/ecstore/src/io_support/bitrot.rs` (e.g. `:564`, `:767`) |
-| Legacy variant | `HighwayHash256SLegacy` (fixed key) for old objects | referenced from `rustfs_utils::HashAlgorithm` (imported at `crates/ecstore/src/io_support/bitrot.rs:26`) |
-| HighwayHash crate | `highway` 1.3.0 | `Cargo.toml:252` |
-| Legacy bitrot read coverage | dedicated test | `crates/ecstore/tests/legacy_bitrot_read_test.rs` |
-
-MinIO uses HighwayHash256 for bitrot; RustFS's default `HighwayHash256S` is
-compatible, with a legacy-key variant retained for older shards.
-
-### Inline data
-
-Small objects are inlined into the `xl.meta` container after the CRC trailer
-rather than written as a separate `part.1`. Handling lives in
-`crates/filemeta/src/filemeta/inline_data.rs` (e.g. `physical_data_dir` and the
-shared-data-dir accounting), and the inline block is appended/consumed by the
-codec in `crates/filemeta/src/filemeta/codec.rs`. This mirrors MinIO's inline
-data feature and the `null`/version-id keying used for the inline map
-(`data_key_for_version` at `crates/filemeta/src/filemeta.rs:69`, legacy key at
-`:77`).
-
-### `xl.meta` interop verdict
-
-| Item | Done | Partial | Todo |
-|---|:--:|:--:|:--:|
-| Read MinIO meta_ver ≤ 3 | ✅ | | |
-| Legacy meta_ver 2 + legacy checksum read | ✅ | | |
-| XL2 container magic/version parity | ✅ | | |
-| Reed-Solomon `rs-vandermonde` parity | ✅ | | |
-| HighwayHash256 bitrot parity | ✅ | | |
-| Inline data parity | ✅ | | |
-| Broad fixture corpus from real MinIO writers | | ⚠️ | |
-| Write-back parity for round-trip (RustFS→MinIO read) | | ⚠️ | |
-
-The two ⚠️ items are verification breadth, not known incompatibilities: the
-current fixtures are targeted regressions (issues #2265, #2288), and there is no
-CI job proving a MinIO binary can re-read a RustFS-written `xl.meta`.
-
----
+MinIO stores an inlined object body as `[HighwayHash256 (32 B)][body]`. Feeding the raw inline shard through RustFS's `BitrotReader` with `HighwayHash256S` verifies the checksum and yields the exact payload; the bitrot prefix is not a format incompatibility.
 
 ## Part B — Bucket Metadata (`.metadata.bin`)
 
-### On-disk layout
-
-| Aspect | RustFS value | Evidence |
+| Aspect | Contract | Where |
 |---|---|---|
-| Meta bucket | `.rustfs.sys` | `crates/ecstore/src/disk/mod.rs:29` (`RUSTFS_META_BUCKET`) |
-| Bucket-config prefix | `buckets` | `crates/ecstore/src/disk/mod.rs:34` (`BUCKET_META_PREFIX`) |
-| Blob file | `.metadata.bin` | `crates/ecstore/src/bucket/metadata.rs:227` (`BUCKET_METADATA_FILE`) |
-| Full path | `buckets/{bucket}/.metadata.bin` | `crates/ecstore/src/bucket/metadata.rs:415-416` (`save_file_path`) |
-| Header | `format: u16 LE` + `version: u16 LE`, both `= 1` | `crates/ecstore/src/bucket/metadata.rs:228-229`, checked in `check_header` at `:595-614` |
-| Body | MessagePack-encoded `BucketMetadata` | `marshal_msg`/`unmarshal` at `crates/ecstore/src/bucket/metadata.rs:582-593`; read strips the 4-byte header (`unmarshal(&data[4..])` at `:1079`) |
+| Path | `buckets/{bucket}/.metadata.bin` under the meta bucket (`BUCKET_METADATA_FILE`, `save_file_path`) | `crates/ecstore/src/bucket/metadata.rs` |
+| Header | 4 bytes: `format: u16 LE` + `version: u16 LE`, stripped before `unmarshal` | `check_header` in `crates/ecstore/src/bucket/metadata.rs` |
+| Body | MessagePack-encoded `BucketMetadata`; field names map one-to-one onto MinIO's `bucketMetadata` (PascalCase on the wire) | `BucketMetadata` in `crates/ecstore/src/bucket/metadata.rs` |
+| Per-config encoding | XML for S3-XML configs, JSON for policy / quota / targets / ACL; the per-config filename constants (`policy.json`, `lifecycle.xml`, ...) are `update_config` field-selector keys, not separate files | `update_config`, `parse_all_configs` in `crates/ecstore/src/bucket/metadata.rs` |
+| RustFS-only fields | `bucket_targets_config_meta_json`, `table_bucket_config_json`; a MinIO reader ignores unknown msgpack fields | `crates/ecstore/src/bucket/metadata.rs` |
+| Partial interop | `bucket_targets` meta side-channel is RustFS-specific; `bucket_acl` round-trips as a blob but only canned ACLs are enforced (see [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md)) | |
 
-This is the same design as MinIO's bucket metadata: a single
-`.minio.sys/buckets/<bucket>/.metadata.bin` blob with a 4-byte
-`bucketMetadataFormat|bucketMetadataVersion` header and a msgpack body. The
-filename, header shape, and format/version values (`1`/`1`) all match. The
-`BucketMetadata` field names correspond one-to-one to MinIO's `bucketMetadata`
-struct (`policyConfigJSON`, `lifecycleConfigXML`, `objectLockConfigXML`, …).
+### Importer
 
-> Correction to a common misconception: modern MinIO does **not** store each
-> bucket config as a separate loose `versioning.json` / `lifecycle.json` file —
-> it embeds them in the same `.metadata.bin` blob, with XML for the S3-XML
-> configs and JSON for policy/quota/targets. The per-config filename constants
-> in RustFS (`policy.json`, `lifecycle.xml`, …) are the **keys used by
-> `update_config`** to select a field, not separate on-disk files.
+`crates/ecstore/src/bucket/migration.rs` is a one-way, idempotent importer from a `MIGRATING_META_BUCKET` (`.minio.sys`) layout into `.rustfs.sys`, run at startup from `rustfs/src/startup_bucket_metadata.rs`:
 
-### Interop matrix (backlog#580 items)
-
-Field/constant references are in `crates/ecstore/src/bucket/metadata.rs`.
-"Encoding" is the payload RustFS stores in that field and must match MinIO's for
-byte-level interop. Getter functions live in
-`crates/ecstore/src/bucket/metadata_sys.rs`.
-
-| Config item | RustFS field / constant | Encoding | MinIO field | Status |
-|---|---|---|---|---|
-| versioning | `versioning_config_xml` / `BUCKET_VERSIONING_CONFIG` = `versioning.xml` | XML | versioningConfigXML | Done |
-| quota | `quota_config_json` / `BUCKET_QUOTA_CONFIG_FILE` = `quota.json` | JSON | quotaConfigJSON | Done |
-| object_lock | `object_lock_config_xml` / `OBJECT_LOCK_CONFIG` = `object-lock.xml` | XML | objectLockConfigXML | Done |
-| replication | `replication_config_xml` / `BUCKET_REPLICATION_CONFIG` = `replication.xml` | XML | replicationConfigXML | Done |
-| policy | `policy_config_json` / `BUCKET_POLICY_CONFIG` = `policy.json` | JSON | policyConfigJSON | Done |
-| lifecycle | `lifecycle_config_xml` / `BUCKET_LIFECYCLE_CONFIG` = `lifecycle.xml` | XML | lifecycleConfigXML | Done |
-| tagging | `tagging_config_xml` / `BUCKET_TAGGING_CONFIG` = `tagging.xml` | XML | taggingConfigXML | Done |
-| bucket_targets | `bucket_targets_config_json` + `bucket_targets_config_meta_json` / `BUCKET_TARGETS_FILE` = `bucket-targets.json` | JSON | bucketTargetsConfigJSON (+ meta variant) | Partial |
-| notification | `notification_config_xml` / `BUCKET_NOTIFICATION_CONFIG` = `notification.xml` | XML | notificationConfigXML | Done |
-| encryption | `encryption_config_xml` / `BUCKET_SSECONFIG` = `bucket-encryption.xml` | XML | encryptionConfigXML | Done |
-| cors | `cors_config_xml` / `BUCKET_CORS_CONFIG` = `cors.xml` | XML | corsConfigXML | Done |
-| public_access | `public_access_block_config_xml` / `BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG` = `public-access-block.xml` | XML | publicAccessBlockConfigXML | Done |
-| bucket_acl | `bucket_acl_config_json` / `BUCKET_ACL_CONFIG` = `bucket-acl.json` | JSON | bucketACLConfigJSON | Partial |
-
-Field definitions: `crates/ecstore/src/bucket/metadata.rs:274-336`. Constants:
-`:227-247`. `update_config` field routing: `:678-761`. `parse_all_configs` is
-invoked on load (`load_bucket_metadata_parse` at `:1043`).
-
-Notes on the two "Partial" rows:
-
-- **bucket_targets** — RustFS carries an extra `bucket_targets_config_meta_json`
-  field (`:288`) beyond MinIO's single targets blob. The primary
-  `bucket-targets.json` payload is interoperable; the meta side-channel is
-  RustFS-specific and a MinIO reader would ignore it. ACL enforcement itself is
-  bounded (S3 `PutBucketAcl`/`PutObjectAcl` accept canned ACLs only — see
-  [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md)).
-- **bucket_acl** — stored and round-tripped in the blob, but ACL grant
-  semantics are intentionally limited at the S3 layer.
-
-RustFS also defines fields with no interop requirement from backlog#580 but
-worth noting so a migration tool does not choke on them: `logging_config_xml`,
-`website_config_xml`, `accelerate_config_xml`, `request_payment_config_xml`
-(`:242-245`), and the RustFS-only `table_bucket_config_json`
-(`BUCKET_TABLE_CONFIG` = `table-bucket.json`, `:248`). A MinIO reader that does
-not know `table_bucket_config_json` will ignore the unknown msgpack field.
-
-### Old-RustFS → new-RustFS migration
-
-RustFS ships a one-way importer that reads a legacy meta bucket
-(`MIGRATING_META_BUCKET`) and rewrites both bucket metadata and IAM config into
-the current RustFS meta bucket, skipping entries that already exist
-(idempotent). See `crates/ecstore/src/bucket/migration.rs`:
-
-- `try_migrate_bucket_metadata` copies `buckets/{bucket}/.metadata.bin` and the
-  replication resync blob for each bucket (`crates/ecstore/src/bucket/migration.rs:193`).
-- `try_migrate_iam_config` walks `config/iam/` and normalizes legacy IAM
-  records — legacy timestamp fields (`update_at` → `updatedAt`) and legacy
-  policy-mapping field aliases (`policies` → `policy`) are rewritten
-  (`normalize_iam_config_blob` at `:97`; regression test at `:428`).
-- Bucket resync metadata is re-encoded through `ReplicationMigrationBridge`
-  (`normalize_bucket_meta_blob` at `:178`).
-
-This importer is the practical basis for a MinIO → RustFS bucket-metadata
-migration: because the blob layout and field encodings already match, the
-missing piece is a source adapter that points the importer at a MinIO
-`.minio.sys` layout rather than the RustFS legacy layout.
-
-### Bucket-metadata interop verdict
-
-| Item | Done | Partial | Todo |
-|---|:--:|:--:|:--:|
-| `.metadata.bin` filename + header + msgpack layout parity | ✅ | | |
-| Per-config field encodings (XML/JSON) match MinIO | ✅ | | |
-| versioning/quota/object_lock/replication/policy/lifecycle/tagging/notification/encryption/cors/public_access round-trip | ✅ | | |
-| bucket_targets primary blob | ✅ | | |
-| bucket_targets meta side-channel + ACL grant semantics | | ⚠️ | |
-| Old-RustFS → new-RustFS importer | ✅ | | |
-| MinIO `.minio.sys` source adapter for the importer | | | ❌ |
-| CI proof a MinIO-written `.metadata.bin` loads unchanged | | | ❌ |
-
----
+| Function | Imports |
+|---|---|
+| `try_migrate_bucket_metadata` | `buckets/{bucket}/.metadata.bin` plus the replication resync blob (`normalize_bucket_meta_blob` via `ReplicationMigrationBridge`) |
+| `try_migrate_iam_config` | `config/iam/` records; `normalize_iam_config_blob` rewrites legacy timestamp and policy-mapping aliases |
 
 ## Part C — Server-Side Encryption (SSE)
 
-Reading MinIO-written SSE objects is implemented, with a deliberate build boundary. The read path lives behind the `rio-v2` feature and is a **special-purpose migration capability**: it is not compiled into released binaries or container images, and there is no short-term plan to promote it into default builds. A default build fails such reads closed with a diagnosed error (see "How default builds fail" below); a `rio-v2` build reads them, within the scenario matrix below. The read-path work was tracked in rustfs/backlog#1638 (landed across rustfs/rustfs#6191, #6784, #6785).
+The `xl.meta` around a MinIO SSE object parses in every build, so such objects list, HEAD, and report plausible sizes; only payload readability depends on the build. KMS wire protocols (AWS `awsJson1_1` client in `crates/kms/src/backends/aws.rs`, MinIO KES) are non-targets.
 
-### Scope boundary: KMS wire protocols and the production gate
-
-This document covers MinIO on-disk metadata and object-encryption seams only. The **AWS KMS wire protocol** and the **MinIO KES wire protocol** are explicit non-targets: RustFS's AWS backend uses the AWS SDK's `awsJson1_1` client path (`crates/kms/src/backends/aws.rs:830`), while KES compatibility is outside this interop work. Those ecosystem evaluations remain separate work in the [#1562 Production Ready exit gate](https://github.com/rustfs/backlog/issues/1562), whose compatibility criterion covers MinIO/RustFS SSE data and rolling upgrades. Closing #1638 does not by itself close that gate.
-
-Note the asymmetry with Parts A and B: the `xl.meta` around a MinIO SSE object parses fine, so such objects list, HEAD, and report plausible sizes. Only payload readability depends on the build and the scenario.
-
-### What can and cannot be migrated
-
-| Object class | Default build | `rio-v2` build | Notes |
+| Object class | `default` / `full` | `rio-v2` | Requirement |
 |---|:--:|:--:|---|
-| Unencrypted objects | ✅ | ✅ | Parts A and B apply. |
-| Bucket metadata, IAM config | ✅ | ✅ | Via the importer, once a `.minio.sys` source adapter exists (see Part B). |
-| Bucket-level default-encryption *configuration* | ✅ | ✅ | The `encryption` config blob round-trips as a blob; it does not make existing ciphertext readable. |
-| SSE-S3 / SSE-KMS, MinIO builtin static KMS (`MINIO_KMS_SECRET_KEY`), single- and multipart | ❌ diagnosed | ✅ | Requires `RUSTFS_SSE_S3_MASTER_KEY` set to the same 32-byte key material as MinIO's static secret. Proven against real MinIO fixtures (rustfs/rustfs#6191). |
-| SSE-C, MinIO-written | ❌ diagnosed | ✅ | Detection via MinIO's sealed-key slot; the customer key is proven by the AEAD unseal, since MinIO stores no key MD5 (rustfs/rustfs#6785). |
-| Any SSE, MinIO backed by KES / KMS plugin / MinKMS | ❌ | ❌ **not planned** | The wrapped DEK is sealed by the KES service itself; it is not a Vault/Transit ciphertext RustFS could be pointed at. Re-encrypt on the MinIO side before migrating. |
-| Objects sealed with legacy `DARE-SHA256` (`InsecureSealAlgorithm`) | ❌ | ❌ out of scope | Pre-DAREv2-HMAC MinIO; `parse_minio_managed_sealed_key` rejects the algorithm and the read fails closed. |
-| RustFS-written SSE objects read back by MinIO | ❌ | ❌ | See "Reverse direction". |
+| SSE-S3 / SSE-KMS, MinIO builtin static KMS, single- and multipart | Fail closed, diagnosed | Read | `RUSTFS_SSE_S3_MASTER_KEY` (base64, 32 bytes) equal to the source MinIO's static secret. |
+| SSE-C, MinIO-written | Fail closed, diagnosed | Read | Client supplies the customer key per request; MinIO stores no key MD5, so the AEAD unseal is the key proof. |
+| Any SSE, MinIO backed by KES / KMS plugin / MinKMS | Fail closed | Fail closed | Not planned. Re-encrypt or decrypt on the MinIO side first. |
+| Bucket default-encryption *configuration* | Round-trips | Round-trips | A config blob; it does not make existing ciphertext readable. |
 
-### The seams, and where they closed
+### Seams
 
-The cryptographic primitives were never the gap — RustFS implements the same DARE V2 stream format, object-key derivation, and sealing. Three seams above the cryptography rejected MinIO-written objects; all three are closed in `rio-v2` builds.
+The cryptography (DARE v2 stream format, object-key derivation, sealing) was never the gap. Three metadata seams above it rejected MinIO objects; all are closed in `rio-v2` builds. Symbols are in `rustfs/src/storage/sse.rs` unless noted.
 
-| # | Seam | Resolution |
-|---|---|---|
-| 1 | Managed-SSE detection required the *persisted* public `x-amz-server-side-encryption` key, which MinIO synthesizes at response time and never stores. | Closed by rustfs/rustfs#6191: `infer_minio_managed_sse_type` infers the scheme from which MinIO sealed-key slot is present (the slot also selects the sealing-key domain, so a wrong inference cannot silently derive a wrong key). Inference from the KMS key id would misclassify — MinIO writes `-S3-Kms-Key-Id` on SSE-S3 objects too. |
-| 2 | MinIO's wrapped-DEK ciphertext was not accepted by any envelope parser. | Closed by rustfs/rustfs#6191: `decrypt_minio_kms_data_key` implements MinIO's builtin-KMS sealing (`sealingKey = HMAC-SHA256(master, iv)`), accepting both the raw `sealed‖iv‖nonce` layout and the legacy `{"aead": ...}` JSON. Routing is by the data key's own byte shape — RustFS's strict JSON envelopes are recognized positively, everything else goes to the MinIO decoder — because slot names cannot distinguish the writer. `LocalSseDekEnvelope` keeps `deny_unknown_fields`. |
-| 3 | SSE-C detection keyed on the stored customer-algorithm header, which MinIO also never persists, and the early key check demanded a stored key MD5 MinIO does not write. | Closed by rustfs/rustfs#6785: `stored_ssec_metadata` also accepts MinIO's SSE-C sealed-key slot (rio-v2 builds only), and `verify_ssec_key_match` tolerates a missing stored MD5 for exactly that shape — the AEAD unseal remains the key proof, and a wrong key still fails there. |
-
-Two further single-part defects were fixed on the way (both rustfs/rustfs#6191 follow-ups): multipart classification now trusts MinIO's own `X-Minio-Internal-Encrypted-Multipart` marker instead of an ETag-length heuristic (MinIO stores *encrypted* ETags, so every single-part SSE object mis-classified as multipart), and single-part plaintext sizes are recovered by DARE reverse-size arithmetic (`dare_v2_decrypted_size`) since MinIO records an explicit size only for multipart uploads.
+| Seam | Resolution |
+|---|---|
+| Managed-SSE detection required the persisted public `x-amz-server-side-encryption` key, which MinIO synthesizes at response time | `infer_minio_managed_sse_type` infers the scheme from which MinIO sealed-key slot is present; the slot also selects the sealing-key domain, so a wrong inference cannot derive a wrong key. |
+| MinIO's wrapped-DEK ciphertext was accepted by no envelope parser | `decrypt_minio_kms_data_key` implements MinIO's builtin-KMS sealing for both the raw `sealed‖iv‖nonce` layout and the legacy `{"aead": ...}` JSON. Routing is by byte shape: `LocalSseDekEnvelope` (`deny_unknown_fields`) is recognized positively, everything else goes to the MinIO decoder. |
+| SSE-C detection keyed on the stored customer-algorithm header, which MinIO also never persists, and demanded a stored key MD5 | `stored_ssec_metadata` accepts MinIO's SSE-C sealed-key slot (`rio-v2` only); `verify_ssec_key_match` tolerates a missing stored MD5 for exactly that shape. |
+| Multipart classification used an ETag-length heuristic (MinIO stores encrypted ETags) | Trusts MinIO's own `X-Minio-Internal-Encrypted-Multipart` marker (`crates/utils/src/http/header_compat.rs`). |
 
 ### How default builds fail
 
-The read fails closed: ciphertext is never served as plaintext. `is_object_encryption_marker` matches the whole `x-minio-internal-server-side-encryption-` prefix, so `ObjectInfo::is_encrypted()` is true for these objects, and the read plan refuses to construct a reader without decryption material. Since rustfs/rustfs#6784 the refusal is diagnosed: the resolver raises a typed error naming the condition — in default builds it points at the MinIO-compatible sealed format and the `rio-v2` read path it would require — and it surfaces as S3 `InvalidObjectState` (non-retryable) instead of the former undiagnosed 500 `InternalError`. List and HEAD still succeed, because `xl.meta` parses normally.
-
-### What a `rio-v2` migration build needs
-
-- A binary built with `--features rio-v2`. The feature is deliberately absent from `default` and `full` in `rustfs/Cargo.toml`; released binaries and images never include it.
-- For SSE-S3/SSE-KMS objects: `RUSTFS_SSE_S3_MASTER_KEY` (base64, 32 bytes) set to the same key material as the source MinIO's `MINIO_KMS_SECRET_KEY`. For SSE-C objects: nothing server-side — the client supplies the customer key per request, as on MinIO.
-- The interop harness is the evidence chain: `rustfs/src/storage/minio_generated_read_test.rs` (`#[ignore]` reader tests over real MinIO-generated fixtures, run with `--features rio-v2`), the fixture lab under `crates/rio-v2/tests/minio_fixture_lab/`, and the `minio-interop` workflow. The SSE-C lane of that harness (customer-key handout from a fixture capture to the reader test) is not wired yet; SSE-C coverage currently lives in the unit suite, which builds the MinIO shape with the same sealing primitives the fixture suite proved byte-compatible.
-
-Known unverified edge: MinIO seals ETags on SSE objects (`SealETag`); RustFS does not unseal them, so ETag display and `If-Match` semantics on migrated SSE objects are not guaranteed to match MinIO's.
+`is_object_encryption_marker` (`crates/utils/src/http/header_compat.rs`) matches the whole `x-minio-internal-server-side-encryption-` prefix, so `ObjectInfo::is_encrypted()` is true and the read plan refuses to construct a reader without decryption material. The refusal is a typed error that names the MinIO-compatible sealed format and the `rio-v2` read path it requires, surfaced as S3 `InvalidObjectState` (non-retryable). Ciphertext is never served as plaintext.
 
 ### Reverse direction
 
-Migrating back is also unsupported. Under `rio-v2` RustFS writes its own DEK envelope into MinIO's sealed-key metadata slots and labels it with MinIO's seal algorithm (`rustfs/src/storage/sse.rs:1830-1852`), so the metadata is MinIO-shaped while the key bytes are not MinIO-openable. Default builds do not populate those slots at all (`rustfs/src/storage/sse.rs:1796-1798`). Treat RustFS-written SSE objects as readable only by RustFS.
+Under `rio-v2` RustFS writes its own DEK envelope into MinIO's sealed-key metadata slots labelled with MinIO's seal algorithm, so the metadata is MinIO-shaped while the key bytes are not MinIO-openable. Default builds do not populate those slots. Treat RustFS-written SSE objects as readable only by RustFS. Known unverified edge: MinIO seals ETags on SSE objects; RustFS does not unseal them, so ETag display and `If-Match` on migrated SSE objects are not guaranteed to match MinIO.
 
-### Migration options
+### Migration options for encrypted objects
 
-- For static-KMS MinIO sources: run the migration through a `rio-v2` build with the shared master key (see above), either serving reads in place or copying objects out into a default-build cluster (the copy re-encrypts under RustFS's own KMS).
-- For KES/MinKMS-backed sources, or when a special-purpose build is not wanted: decrypt on the MinIO side first — rewrite the affected objects as plaintext, or copy them out through MinIO's S3 endpoint, which decrypts on read — and let RustFS apply its own encryption on ingest.
-- Leave encrypted objects on MinIO and migrate only unencrypted data.
+1. Static-KMS source: run the migration through a `rio-v2` build with the shared master key, serving in place or copying into a default-build cluster (the copy re-encrypts under RustFS's own KMS).
+2. KES / MinKMS source, or no special-purpose build wanted: decrypt on the MinIO side (rewrite as plaintext, or copy out through MinIO's S3 endpoint) and let RustFS encrypt on ingest.
+3. Leave encrypted objects on MinIO and migrate only unencrypted data.
 
-Inventory the source first — bucket default-encryption settings mean objects can be encrypted without the uploader having asked for it, so "we never set SSE headers" is not sufficient evidence that a bucket has no encrypted objects.
+Inventory the source first: bucket default encryption means objects can be encrypted without the uploader asking, so "we never set SSE headers" is not evidence that a bucket has no encrypted objects.
 
-### SSE interop verdict
+## rio-v2 variant lifecycle
 
-| Item | Done | Partial | Todo |
-|---|:--:|:--:|:--:|
-| DARE V2 stream format parity | ✅ | | |
-| Object-key derivation / sealing parity | ✅ | | |
-| Managed-SSE detection accepts MinIO-written metadata (`rio-v2`) | ✅ | | |
-| MinIO builtin-KMS wrapped-DEK parser (raw + legacy JSON) | ✅ | | |
-| SSE-C detection accepts MinIO-written metadata (`rio-v2`) | ✅ | | |
-| Read MinIO-written SSE-S3 / SSE-KMS end to end, single- and multipart | ✅ | | |
-| Read MinIO-written SSE-C end to end | | ⚠️ unit-proven; fixture-lab lane unwired | |
-| Migrated-object sealed-ETag semantics | | | ❌ unverified |
-| KES / MinKMS / legacy `DARE-SHA256` sources | | | ❌ not planned |
-| RustFS-written SSE objects readable by MinIO | | | ❌ |
-| CI proof of SSE read parity | | ⚠️ `minio-interop` workflow; nightly once re-enabled | |
+`rio-v2` is a dormant, special-purpose migration variant tracked under `rustfs/backlog#1835`.
 
----
+| Fact | Value |
+|---|---|
+| Shipping status | Ships in no default build: absent from `default` and `full` in `rustfs/Cargo.toml`; released binaries and container images never include it. Enable with `--features rio-v2`. |
+| Pull-request coverage | `test-and-lint-rio-v2` in `.github/workflows/ci.yml`: clippy plus `cargo nextest` for `rustfs` and `rustfs-ecstore` with `--features rio-v2`. This is the cfg-seam guard; it keeps the feature compiling and its unit suite green on every pull request. |
+| Full-suite lane | `build-rustfs-debug-binary-rio-v2` and `e2e-tests-rio-v2` in `ci.yml` run only on the weekly schedule and manual dispatch (`cache-warm.yml` keeps the `ci-feat-rio` cache warm so the scheduled build fits its timeout). |
+| Interop evidence | `.github/workflows/minio-interop.yml` (nightly plus manual) regenerates real MinIO backend trees via `crates/rio-v2/tests/minio_fixture_lab/` and runs the `#[ignore]` reader tests in `rustfs/src/storage/minio_generated_read_test.rs` with `--features rio-v2`. Its freshness is tracked in `.github/scheduled-validations.json`. |
+| Promote-or-delete condition | The variant stays dormant until one of two things happens. **Promote**: a release commits to shipping MinIO SSE migration as a supported capability; then `rio-v2` joins `default`/`full`, the scheduled lanes run on every pull request, and this section is rewritten. **Delete**: no release commits to it and the scheduled lanes are not kept green; then the feature flag, `crates/rio-v2`, the cfg seams in `rustfs/src/storage/sse.rs`, the three `ci.yml` jobs, the `cache-warm.yml` warm step, `minio-interop.yml`, and its `scheduled-validations.json` entry are removed in one change. Either outcome must update `ARCHITECTURE.md` and the `ci.yml` job comments that cite this section. |
 
-## Phased Plan
+## Fixture Evidence
 
-The format is already close; the plan is verification, a source adapter, and
-closing the two partial encodings — not a rewrite.
+Fixtures were captured from a real MinIO single-drive instance and live under `crates/filemeta/tests/fixtures/minio/` and `crates/ecstore/tests/fixtures/minio/`. These tests run in the normal `cargo test` / nextest lanes.
 
-### Phase 1 — Read parity, proven (verification)
+| Test | File | Proves |
+|---|---|---|
+| `parses_real_minio_object_xlmeta` | `crates/filemeta/src/filemeta.rs` | Inline, two-version plus delete-marker, and multipart `xl.meta` parse to the expected `FileInfo`. |
+| `parses_real_minio_bucket_metadata_blob_without_loss` | `crates/ecstore/src/bucket/metadata.rs` | The msgpack blob decodes via MinIO's field names and `parse_all_configs` loads every config in the corpus, including MinIO's lifecycle `<ExpiryUpdatedAt>` and replication `DeleteMarkerReplication` / `ExistingObjectReplication` extensions. |
+| `reads_minio_inline_bucket_metadata_via_bitrot` | `crates/ecstore/src/bucket/metadata.rs` | The inline shard's HighwayHash prefix verifies under `HighwayHash256S` and yields the exact `.metadata.bin` blob. |
+| `migrates_real_minio_bucket_metadata_end_to_end` | `crates/ecstore/src/bucket/migration.rs` | A real `.metadata.bin` seeded under a `.minio.sys` layout is imported by `try_migrate_bucket_metadata` into `.rustfs.sys` byte-identical, through the object layer on a 4-drive `ECStore`. |
+| `test_issue_2265_legacy_meta_v2_object_compatibility`, `test_issue_2288_legacy_xlmeta_compatibility` | `crates/filemeta/src/filemeta.rs` | Legacy meta_ver 2 objects with legacy checksums still read. |
+| `minio_generated_read_test.rs` (`#[ignore]`, `rio-v2`) | `rustfs/src/storage/minio_generated_read_test.rs` | Byte-identical plaintext reconstruction of MinIO SSE-S3 / SSE-KMS fixtures; driven by `minio-interop.yml`. |
 
-- Add a MinIO-writer fixture corpus for `xl.meta` (inline + multipart +
-  versioned + delete-marker + transitioned) and assert RustFS parses each to a
-  `FileInfo` equivalent to MinIO's, alongside the existing issue #2265 / #2288
-  fixtures in `crates/filemeta/src/filemeta.rs`.
-- Add a fixture `.metadata.bin` written by MinIO and assert
-  `BucketMetadata::unmarshal` + `parse_all_configs` load every field without
-  loss (`crates/ecstore/src/bucket/metadata.rs`).
-- Exit criterion: a CI job that fails if a real MinIO-written object or bucket
-  blob cannot be read.
+Not fixture-proven: transitioned `xl.meta`; CORS, public-access-block, and bucket-ACL configs (the SNSD corpus did not exercise them); bucket-targets credentials (MinIO stores them KMS-encrypted); the SSE-C fixture-lab lane (customer-key handout is not wired; SSE-C coverage is unit-level).
 
-#### Phase 1 status — first fixtures landed (verified 2026-07-07)
+## Out Of Scope
 
-A real MinIO `RELEASE.2025-07-23` single-drive instance wrote a bucket with
-versioning, object-lock (GOVERNANCE default), lifecycle, tagging, quota, and a
-public-download policy, plus inline / versioned / multipart objects. The on-disk
-`xl.meta` blobs are captured as hex fixtures
-(`crates/filemeta/tests/fixtures/minio/`, `crates/ecstore/tests/fixtures/minio/`).
-
-Proven by regression tests:
-
-- **Object `xl.meta` read parity** — `parses_real_minio_object_xlmeta`
-  (`crates/filemeta/src/filemeta.rs`): small inline, two-object-version + delete
-  marker, and multipart objects all parse to the expected `FileInfo`.
-- **Bucket-metadata parse parity** — `parses_real_minio_bucket_metadata_blob_without_loss`
-  (`crates/ecstore/src/bucket/metadata.rs`): the msgpack blob decodes via the
-  PascalCase MinIO field names, and `parse_all_configs` loads **all ten** config
-  types present in the corpus without loss — policy, lifecycle (**including
-  MinIO's `<ExpiryUpdatedAt>` extension**), object-lock, versioning, tagging,
-  quota, notification, encryption (SSE-S3), and replication (**including the
-  `DeleteMarkerReplication` / `ExistingObjectReplication` MinIO extensions**).
-- **Inline bucket-metadata read parity** — `reads_minio_inline_bucket_metadata_via_bitrot`
-  (`crates/ecstore/src/bucket/metadata.rs`): MinIO stores an inlined object body
-  as `[HighwayHash256 (32B)][body]`. The "`inline_data` 前缀不同" that weisd
-  raised on 2026-03-06 is exactly that bitrot prefix — **not** a format
-  incompatibility. Feeding the raw inline shard through RustFS's `BitrotReader`
-  with the default `HighwayHash256S` verifies the checksum (confirming RustFS's
-  hash matches MinIO's) and yields the exact `.metadata.bin` blob, which then
-  parses. So the object-layer inline read is compatible; the earlier "extract
-  `fi.data` directly" concern was reading the shard before the bitrot layer
-  strips its prefix.
-- **End-to-end migration** — `migrates_real_minio_bucket_metadata_end_to_end`
-  (`crates/ecstore/src/bucket/migration.rs`): on a throwaway 4-drive local
-  `ECStore`, a real MinIO `.metadata.bin` seeded under a `.minio.sys` layout is
-  migrated by `try_migrate_bucket_metadata` into `.rustfs.sys`, and the migrated
-  blob carries every config (policy / lifecycle / object-lock / versioning /
-  tagging / quota / notification / encryption / replication) byte-identical to
-  the source. This exercises the Phase 2 source adapter
-  (`MIGRATING_META_BUCKET = ".minio.sys"`) end-to-end through the object layer —
-  proven, not just present.
-
-Still to broaden: transitioned `xl.meta`; CORS, public-access-block, and bucket
-ACL configs (the SNSD test binary/`mc` did not expose these); and bucket-targets
-credentials, which MinIO stores KMS-encrypted (a documented partial). These run
-as ordinary crate tests, so they already execute in the normal `cargo
-test`/nextest CI jobs.
-
-### Phase 2 — MinIO source adapter for migration
-
-- Generalize the importer in `crates/ecstore/src/bucket/migration.rs` so the
-  source can be a MinIO `.minio.sys/buckets/<bucket>/.metadata.bin` layout, not
-  only the RustFS legacy meta bucket. Because the blob format matches, this is
-  mostly source-path plumbing plus IAM record normalization reuse.
-- Exit criterion: importing a MinIO backup reproduces all backlog#580
-  bucket-config items with byte-identical config payloads.
-
-### Phase 3 — Close the two partial encodings
-
-- bucket_targets: document/normalize the RustFS-only
-  `bucket_targets_config_meta_json` so a round-trip through MinIO and back does
-  not silently drop it; or fold its content into a MinIO-compatible
-  representation.
-- bucket_acl: decide whether ACL grant semantics beyond canned ACLs are in
-  scope; if not, keep the blob round-trippable but document the enforcement
-  limit (already reflected in the router compatibility matrix).
-
-### Phase 4 — Round-trip / write-back parity (non-goal for migration)
-
-Proving a MinIO binary can re-read a *RustFS-written drive set* (the reverse
-direction) is **out of scope for the migration use case**, which is one-way
-MinIO → RustFS:
-
-- RustFS's meta bucket is `.rustfs.sys` (`crates/ecstore/src/disk/mod.rs:29`);
-  MinIO looks for `.minio.sys`. A MinIO binary pointed at a RustFS drive set
-  does not find `format.json` or bucket configs and refuses the set — this is a
-  set-level divergence, not an object-format one.
-- The object-level `xl.meta` format *does* match (proven above), so the reverse
-  direction is limited by drive-set discovery, not by per-object encoding.
-- The supported flow is one-way: `try_migrate_bucket_metadata` /
-  `try_migrate_iam_config` / `format.json` migration import a MinIO layout into
-  RustFS. There is no requirement to keep a live MinIO able to serve
-  RustFS-written drives.
-
-If a true bidirectional round-trip is ever needed, it would require RustFS to
-optionally write the `.minio.sys` set layout — a separate feature, not part of
-the interop/migration story tracked here.
-
----
+- A live MinIO binary serving a RustFS-written drive set (set-level `.minio.sys` vs `.rustfs.sys` divergence). A bidirectional round-trip would require RustFS to optionally write the `.minio.sys` set layout, which is a separate feature.
+- RustFS-written SSE objects readable by MinIO.
+- KES / MinKMS / KMS-plugin-sealed MinIO objects.
+- Objects sealed with pre-DARE-v2-HMAC MinIO seal algorithms; `parse_minio_managed_sealed_key` rejects unknown algorithms and the read fails closed.
+- AWS KMS and KES wire-protocol compatibility.
 
 ## Guardrails
 
-- This document is analysis only. Any change to `crates/filemeta` or
-  `crates/ecstore/src/bucket` metadata encoding is a storage-format change and
-  must follow the migration and readiness contracts in
-  [README.md](README.md) and the ecstore layout boundary rules.
-- The version constants (`XL_META_VERSION`,
-  `BUCKET_METADATA_FORMAT`/`BUCKET_METADATA_VERSION`) are compatibility anchors.
-  Bumping any of them requires a read-compat path for the prior value and a
-  migration story, exactly as the current meta_ver 2 → 3 read path provides.
+- Any change to `crates/filemeta` or `crates/ecstore/src/bucket` metadata encoding is a storage-format change and follows the migration and readiness contracts in [README.md](README.md) and the ecstore layout boundary rules.
+- Do not bump a Version Anchor without a read path for the prior value; see [erasure-coding.md](erasure-coding.md) for the accept-older, reject-newer rule.
+- `.github/workflows/ci.yml`, `.github/workflows/cache-warm.yml`, and `ARCHITECTURE.md` cite the [rio-v2 variant lifecycle](#rio-v2-variant-lifecycle) heading; keep it when editing this file.

docs/architecture/minio-rustfs-router-compatibility.md

@@ -1,213 +1,50 @@
-# MinIO ↔ RustFS Router Compatibility Matrix
+# MinIO ↔ RustFS Router Compatibility (Exceptions Only)
 
-Tracks how RustFS covers the MinIO HTTP router surface, split into the S3
-data-plane router (`cmd/api-router.go` in MinIO: object + bucket APIs) and the
-admin control-plane router (`cmd/admin-router.go`: admin `/v3/` and `/v4/`
-APIs). Each row records the current RustFS implementation status and the
-landing point in the code so the matrix can be re-verified after refactors.
+**Use this when:** a client or `mc` call that works against MinIO fails against RustFS and you need to know whether the endpoint is missing, stubbed, or deliberately different.
+**Source of truth:** S3 plane: the `s3s::S3` trait impl in `rustfs/src/storage/ecfs.rs`. Admin plane: `make_admin_route` in `rustfs/src/admin/mod.rs`, the registration inventory `rustfs/src/admin/route_registration_test.rs`, and the route/action guardrail [admin-route-action-snapshot.md](admin-route-action-snapshot.md).
 
-This complements two neighbouring documents and does not duplicate them:
-
-- [s3-compatibility-matrix.md](s3-compatibility-matrix.md) — the release-facing
-  S3 compatibility claim and the Ceph s3tests lists that gate it.
-- [admin-route-action-snapshot.md](admin-route-action-snapshot.md) — the admin
-  route/handler/authorization-action migration guardrail (the source of truth
-  for exact route patterns and auth contracts).
-
-Refs rustfs/backlog#596 rustfs/backlog#603.
+This document lists only exceptions. Anything not listed here is implemented with MinIO-equivalent behavior. For the s3tests-level claim see [s3-compatibility-matrix.md](s3-compatibility-matrix.md).
 
 ## Status Legend
 
 | Status | Meaning |
 |---|---|
-| 已实现 (implemented) | Handler is registered and performs the real operation. |
-| 部分兼容 (partial) | Registered and functional, but a documented subset of the MinIO behavior is rejected or unsupported. |
-| 已注册未完成 (registered, incomplete) | Route is registered but the handler returns `NotImplemented` (a behavior contract, not a real implementation). |
-| 缺失 (missing) | No RustFS route/handler for the MinIO endpoint. |
-| 行为不一致 (behavior differs) | Implemented but intentionally diverges from MinIO's response contract. |
+| 缺失 (missing) | No RustFS route or handler for the MinIO endpoint. |
+| 部分兼容 (partial) | Registered and functional, but a documented subset of MinIO behavior is rejected. |
+| 已注册未完成 (registered, incomplete) | Route is registered; the handler returns `NotImplemented` as a behavior contract. |
+| 行为不一致 (behavior differs) | Implemented, but intentionally diverges from MinIO's response contract. |
 
-Prefixes: RustFS registers admin routes under the canonical `/rustfs/admin`
-prefix and accepts `/minio/admin` as a compatibility alias via router
-canonicalization (see
-[admin-route-action-snapshot.md](admin-route-action-snapshot.md)). Admin paths
-below are shown relative to that prefix (e.g. `/v3/info`).
+Admin paths are relative to the canonical `/rustfs/admin` prefix; `/minio/admin` is accepted as an alias via router canonicalization.
 
----
+## S3 Data Plane
 
-## Part 1 — S3 Data Plane (MinIO `cmd/api-router.go`)
+All `s3s::S3` trait methods are implemented in `rustfs/src/storage/ecfs.rs` except the following.
 
-RustFS implements the S3 surface through the `s3s` service trait in
-`rustfs/src/storage/ecfs.rs`, delegating to use-case layers under
-`rustfs/src/app/`. Line numbers are indicative landing points on the branch
-this matrix was written against and may drift; the file paths are stable.
-
-### Bucket-level operations
-
-| MinIO / S3 operation | Status | RustFS landing point |
+| S3 operation | Status | Detail |
 |---|---|---|
-| CreateBucket | 已实现 | `rustfs/src/storage/ecfs.rs` (`create_bucket`) |
-| DeleteBucket | 已实现 | `rustfs/src/storage/ecfs.rs` (`delete_bucket`) |
-| HeadBucket | 已实现 | `rustfs/src/storage/ecfs.rs` (`head_bucket`) |
-| ListBuckets | 已实现 | `rustfs/src/storage/ecfs.rs` (`list_buckets`) |
-| GetBucketLocation | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_location`) |
-| ListObjects (v1) | 已实现 | `rustfs/src/storage/ecfs.rs` (`list_objects`) |
-| ListObjectsV2 | 已实现 | `rustfs/src/storage/ecfs.rs` (`list_objects_v2`) |
-| ListObjectVersions | 已实现 | `rustfs/src/storage/ecfs.rs` (`list_object_versions`) |
-| ListMultipartUploads | 已实现 | `rustfs/src/storage/ecfs.rs` (`list_multipart_uploads`) |
-| Get/PutBucketVersioning | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_versioning`, `put_bucket_versioning`) |
-| Get/Put/DeleteBucketPolicy | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_policy`, `put_bucket_policy`, `delete_bucket_policy`) |
-| GetBucketPolicyStatus | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_policy_status`) |
-| Get/Put/DeleteBucketTagging | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_tagging`, `put_bucket_tagging`, `delete_bucket_tagging`) |
-| Get/Put/DeleteBucketLifecycle | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_lifecycle_configuration`, `put_bucket_lifecycle_configuration`, `delete_bucket_lifecycle`) |
-| Get/Put/DeleteBucketReplication | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_replication`, `put_bucket_replication`, `delete_bucket_replication`) |
-| Get/Put/DeleteBucketEncryption | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_encryption`, `put_bucket_encryption`, `delete_bucket_encryption`) |
-| Get/PutObjectLockConfiguration | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_object_lock_configuration`, `put_object_lock_configuration`) |
-| Get/Put/DeletePublicAccessBlock | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_public_access_block`, `put_public_access_block`, `delete_public_access_block`) |
-| Get/Put/DeleteBucketCors | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_cors`, `put_bucket_cors`, `delete_bucket_cors`) |
-| GetBucketNotificationConfiguration | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_notification_configuration`) |
-| PutBucketNotificationConfiguration | 已实现 | `rustfs/src/storage/ecfs.rs` (`put_bucket_notification_configuration`) |
-| Get/PutBucketRequestPayment | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_request_payment`, `put_bucket_request_payment`) |
-| Get/PutBucketLogging | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_logging`, `put_bucket_logging`) |
-| Get/Put/DeleteBucketWebsite | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_website`, `put_bucket_website`, `delete_bucket_website`) |
-| Get/PutBucketAccelerateConfiguration | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_accelerate_configuration`, `put_bucket_accelerate_configuration`) |
-| GetBucketAcl | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_bucket_acl`) |
-| PutBucketAcl | 部分兼容 | `rustfs/src/storage/ecfs.rs` (`put_bucket_acl`) — canned-ACL headers only; XML grant policies return `NotImplemented`. |
-| GetBucketReplicationMetrics | 缺失 | No S3-path handler; replication metrics are exposed via the admin API `/v3/replicationmetrics` instead. |
-| GetBucketOwnershipControls | 缺失 | Not implemented (matches the "bucket ownership controls: planned" note in `s3-compatibility-matrix.md`). |
-| Put/DeleteBucketOwnershipControls | 缺失 | Not implemented. |
+| GetBucketReplicationMetrics | 缺失 | No `get_bucket_replication_metrics`; replication metrics are exposed via admin `/v3/replicationmetrics`. |
+| GetBucketOwnershipControls | 缺失 | No handler; s3tests entries remain in `scripts/s3-tests/unimplemented_tests.txt`. |
+| PutBucketOwnershipControls, DeleteBucketOwnershipControls | 缺失 | No handler. |
+| DeleteBucketNotification, DeleteBucketLogging, DeleteBucketRequestPayment, DeleteBucketAccelerate | 部分兼容 | No distinct DELETE handlers; clear the config by writing an empty configuration through the PUT path. |
+| PutBucketAcl, PutObjectAcl | 部分兼容 | Canned-ACL headers only; XML grant bodies return `NotImplemented` (`put_bucket_acl`, `put_object_acl`). |
+| GetObjectTorrent | 行为不一致 | `get_object_torrent` returns `404 NoSuchKey` by design, not `501 NotImplemented`, so clients degrade gracefully. |
 
-Note on delete verbs: several S3 sub-resource DELETE operations
-(DeleteBucketNotification, DeleteBucketLogging, DeleteBucketRequestPayment,
-DeleteBucketAccelerate) are not exposed as distinct handlers; the corresponding
-config is cleared by writing an empty configuration through the PUT path. Treat
-these as 部分兼容 at the client level.
+## Admin Control Plane
 
-### Object-level operations
+Every route asserted in `rustfs/src/admin/route_registration_test.rs` is registered. Exceptions:
 
-| MinIO / S3 operation | Status | RustFS landing point |
+| MinIO admin family | Status | Detail |
 |---|---|---|
-| GetObject | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_object`) |
-| PutObject | 已实现 | `rustfs/src/storage/ecfs.rs` (`put_object`) |
-| DeleteObject | 已实现 | `rustfs/src/storage/ecfs.rs` (`delete_object`) |
-| DeleteObjects (multi-delete) | 已实现 | `rustfs/src/storage/ecfs.rs` (`delete_objects`) |
-| HeadObject | 已实现 | `rustfs/src/storage/ecfs.rs` (`head_object`) |
-| CopyObject | 已实现 | `rustfs/src/storage/ecfs.rs` (`copy_object`) |
-| GetObjectAcl | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_object_acl`) |
-| PutObjectAcl | 部分兼容 | `rustfs/src/storage/ecfs.rs` (`put_object_acl`) — canned-ACL headers only; XML grants return `NotImplemented`. |
-| Get/Put/DeleteObjectTagging | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_object_tagging`, `put_object_tagging`, `delete_object_tagging`) |
-| GetObjectAttributes | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_object_attributes`) |
-| Get/PutObjectLegalHold | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_object_legal_hold`, `put_object_legal_hold`) |
-| Get/PutObjectRetention | 已实现 | `rustfs/src/storage/ecfs.rs` (`get_object_retention`, `put_object_retention`) |
-| RestoreObject (POST restore) | 已实现 | `rustfs/src/storage/ecfs.rs` (`restore_object`) |
-| SelectObjectContent | 已实现 | `rustfs/src/storage/ecfs.rs` (`select_object_content`) → `rustfs/src/app/select_object.rs` |
-| CreateMultipartUpload | 已实现 | `rustfs/src/storage/ecfs.rs` (`create_multipart_upload`) |
-| UploadPart / UploadPartCopy | 已实现 | `rustfs/src/storage/ecfs.rs` (`upload_part`, `upload_part_copy`) |
-| CompleteMultipartUpload | 已实现 | `rustfs/src/storage/ecfs.rs` (`complete_multipart_upload`) |
-| AbortMultipartUpload | 已实现 | `rustfs/src/storage/ecfs.rs` (`abort_multipart_upload`) |
-| ListParts | 已实现 | `rustfs/src/storage/ecfs.rs` (`list_parts`) |
-| PostObject (POST form upload) | 已实现 | Routed via the POST-object marker into the put-object path (`rustfs/src/app/object/put.rs`). See the "POST Object form upload checksum handling: planned" note in `s3-compatibility-matrix.md`. |
-| GetObjectTorrent | 行为不一致 | `rustfs/src/storage/ecfs.rs` (`get_object_torrent`) — returns `404 NoSuchKey` by design (not `501 NotImplemented`) so clients degrade gracefully. |
+| Batch jobs (`/v3/start-job`, `/v3/list-jobs`, `/v3/status-job`, `/v3/describe-job`, `/v3/cancel-job`) | 已注册未完成 | `rustfs/src/admin/handlers/batch_job.rs`: `start-job` returns `NotImplemented` for known job types (`KNOWN_JOB_TYPES`) and `InvalidRequest` for unknown ones; `list-jobs` returns an empty list; status/describe/cancel return no-such-job. See [kms-bulk-rekey-contract.md](kms-bulk-rekey-contract.md) for why `keyrotate` must keep refusing. |
+| Service control (`POST /v3/service`) | 行为不一致 | `ServiceHandle` in `rustfs/src/admin/handlers/system.rs`: `restart` and `stop` both initiate graceful shutdown (the process manager must relaunch; no in-process restart); `freeze` / `unfreeze` toggle a global freeze flag under `ServiceFreezeAdminAction`. `rustfs/src/admin/route_policy.rs` still classifies the route as deferred `NotImplemented`. |
+| Inspect data (`GET|POST /v3/inspect-data`) | 行为不一致 | `InspectDataHandler` in `system.rs` returns the raw bytes of one exact `volume` + `file`, size-capped, instead of MinIO's encrypted raw-drive-file archive. The bounded archive lives at `POST /v4/inspect/archive` (`rustfs/src/admin/handlers/inspect_archive.rs`). `route_policy.rs` still classifies the v3 route as deferred `NotImplemented`. |
+| Pools decommission / cancel / clear | 部分兼容 | `rustfs/src/admin/handlers/pools.rs` returns `NotImplemented` when endpoints are not initialized (single-pool or uninitialized clusters). |
+| `/v3/top/drives`, `/v3/top/net` | 缺失 | Only `/v3/top/locks` is registered (`rustfs/src/admin/handlers/diagnostics.rs`). |
+| Bucket / site replication per-object diff | 缺失 | `/v3/replicationmetrics` and site-replication status exist; no diff endpoint. |
+| MRF (most-recent-failures) replication metrics breakdown | 缺失 | Only the generic `/v3/metrics` stream and replication metrics wire (`rustfs/src/admin/replication_metrics_wire.rs`). |
 
-For the gate-level view of which of these are covered by executable s3tests,
-defer to [s3-compatibility-matrix.md](s3-compatibility-matrix.md); this table is
-the router/handler view, not the test-list view.
+Formerly-missing families that are now registered and therefore not exceptions: `/v3/healthinfo`, `/v3/obdinfo`, `/v3/force-unlock`, `/v3/top/locks`, `/v3/speedtest*`, `/v3/log`, `/v3/trace`, `/v3/profile`, `/v3/profiling/*`, `/v3/idp/{ldap|openid}/*`, `/v3/idp-config/*`.
 
----
+## Update Rule
 
-## Part 2 — Admin Control Plane (MinIO `cmd/admin-router.go`)
-
-Router assembly is `rustfs/src/admin/mod.rs::register_admin_routes`; the exact
-route patterns, handler ownership, and authorization actions are the guardrail
-in [admin-route-action-snapshot.md](admin-route-action-snapshot.md). This table
-maps MinIO admin route families to RustFS status.
-
-### Implemented / registered families
-
-| MinIO admin family | Status | RustFS landing point |
-|---|---|---|
-| STS / is-admin probe | 已实现 | `rustfs/src/admin/handlers/sts.rs`, `is_admin.rs` |
-| User lifecycle (list/add/info/remove/status) | 已实现 | `rustfs/src/admin/handlers/user_lifecycle.rs`, `user.rs` |
-| Groups | 已实现 | `rustfs/src/admin/handlers/group.rs` |
-| Service accounts / access keys | 已实现 | `rustfs/src/admin/handlers/service_account.rs` |
-| Canned policies + builtin policy attach/detach + policy-entities | 已实现 | `rustfs/src/admin/handlers/policies.rs` |
-| IAM import/export | 已实现 | `rustfs/src/admin/handlers/user_iam.rs`, `user.rs` |
-| Account info | 已实现 | `rustfs/src/admin/handlers/account_info.rs` |
-| Config KV (get/set/del/help/history/restore + `/v3/config`) | 已实现 | `rustfs/src/admin/handlers/config_admin.rs` |
-| Server info / storageinfo / datausageinfo | 已实现 | `rustfs/src/admin/handlers/system.rs` |
-| Metrics stream (`/v3/metrics`) | 已实现 | `rustfs/src/admin/handlers/metrics.rs` via `system.rs` |
-| Runtime capabilities (`/v4/runtime/capabilities`) | 已实现 | `rustfs/src/admin/handlers/system.rs` |
-| Pools list/status | 已实现 | `rustfs/src/admin/handlers/pools.rs` |
-| Pools decommission/cancel/clear | 部分兼容 | `rustfs/src/admin/handlers/pools.rs` — returns `NotImplemented` when endpoints are not initialized (single-pool / uninitialized clusters). |
-| Rebalance start/status/stop | 已实现 | `rustfs/src/admin/handlers/rebalance.rs` |
-| Heal + background-heal status | 已实现 | `rustfs/src/admin/handlers/heal.rs` |
-| Tier (list/stats/verify/add/edit/remove/clear) | 已实现 | `rustfs/src/admin/handlers/tier.rs` |
-| Quota (legacy + bucket-scoped + stats/check) | 已实现 | `rustfs/src/admin/handlers/quota.rs` |
-| Bucket metadata export/import | 已实现 | `rustfs/src/admin/handlers/bucket_meta.rs` |
-| Scanner status | 已实现 | `rustfs/src/admin/handlers/scanner.rs` |
-| Notification targets (list/arns/put/reset) | 已实现 | `rustfs/src/admin/handlers/event.rs` |
-| Audit targets (list/put/reset) | 已实现 | `rustfs/src/admin/handlers/audit.rs` |
-| Module switches | 已实现 | `rustfs/src/admin/handlers/module_switch.rs` |
-| Plugin catalog + instances (`/v4/plugins/*`) | 已实现 | `rustfs/src/admin/handlers/plugins_catalog.rs`, `plugins_instances.rs` |
-| Extension catalog + instances (`/v4/extensions/*`) | 已实现 | `rustfs/src/admin/handlers/extensions.rs` |
-| Object ZIP download (`/v3/zip-downloads`) | 已实现 | `rustfs/src/admin/handlers/object_zip_download.rs` |
-| Cluster snapshot (`/v4/cluster/snapshot`) | 已实现 | `rustfs/src/admin/handlers/cluster_snapshot.rs` |
-| Bucket-level remote targets (list/metrics/set/remove) | 已实现 | `rustfs/src/admin/handlers/replication.rs` |
-| Site replication (add/remove/info/status/peer/resync + devnull/netperf) | 已实现 | `rustfs/src/admin/handlers/site_replication.rs` |
-| Admin profiling (`/debug/pprof/profile`, `/debug/pprof/status`) | 已实现 | `rustfs/src/admin/handlers/profile_admin.rs`, `profile.rs` |
-| TLS debug (`/debug/tls/status`) | 已实现 | `rustfs/src/admin/handlers/tls_debug.rs`, `profile.rs` |
-| KMS management / dynamic / keys | 已实现 | `rustfs/src/admin/handlers/kms_management.rs`, `kms_dynamic.rs`, `kms_keys.rs` |
-| OIDC public + config | 已实现 | `rustfs/src/admin/handlers/oidc.rs` |
-| Table catalog (Iceberg) | 已实现 | `rustfs/src/admin/handlers/table_catalog/mod.rs` |
-
-### Registered-but-incomplete
-
-| MinIO admin family | Status | RustFS landing point |
-|---|---|---|
-| Service restart/stop (`POST /v3/service`) | 已注册未完成 | `rustfs/src/admin/handlers/system.rs` — handler returns `NotImplemented`. |
-| Inspect data (`GET|POST /v3/inspect-data`) | 已注册未完成 | `rustfs/src/admin/handlers/system.rs` — handler returns `NotImplemented`. |
-
-These registered-but-`NotImplemented` routes are behavior contracts; per the
-migration rules in [admin-route-action-snapshot.md](admin-route-action-snapshot.md),
-implementing or removing them is a behavior-change PR.
-
----
-
-## Gaps Only — Missing Admin Endpoints (follow-up checklist)
-
-The following MinIO admin `/v3/` route families have **no** RustFS registration
-today. This is the actionable checklist for closing admin-API parity. Verified
-against `rustfs/src/admin/mod.rs` and `rustfs/src/admin/handlers/` on the branch
-this doc was written on.
-
-- [ ] **Server profiling start/stop** — MinIO `/v3/profile` (bulk profiling
-  session). RustFS only exposes `/debug/pprof/profile` and
-  `/debug/pprof/status`, which are a different, single-shot pprof surface.
-- [ ] **Health info** — MinIO `/v3/healthinfo` (cluster health report / subnet
-  diagnostics). No RustFS route.
-- [ ] **LDAP / generic IDP config CRUD** — MinIO `/v3/idp/{ldap|openid}/...`
-  config management. RustFS exposes OIDC config under `/v3/oidc/*` only; there
-  is no LDAP IDP config route.
-- [ ] **Bucket / site replication diff** — MinIO replication-diff endpoints.
-  RustFS exposes `/v3/replicationmetrics` (metrics) and site-replication
-  status, but no per-object diff.
-- [ ] **MRF metrics** — MinIO's most-recent-failures replication metrics
-  breakdown. RustFS has only the generic `/v3/metrics` stream.
-- [ ] **Batch jobs** — MinIO `/v3/batch`, `/v3/list-batch-jobs`, job
-  describe/cancel. No RustFS batch API.
-- [ ] **Distributed locks introspection** — MinIO `/v3/force-unlock` and
-  `/v3/top/locks`. No RustFS locks-management API.
-- [ ] **Speedtest / perf** — MinIO `/v3/speedtest` (object/drive/net perf).
-  RustFS has `netperf`/`devnull` **only** inside the site-replication family,
-  not as standalone admin speedtest endpoints.
-- [ ] **Console log stream** — MinIO `/v3/log` (kstream / log search). No RustFS
-  route.
-- [ ] **Top introspection** — MinIO `/v3/top/locks`, `/v3/top/drives`,
-  `/v3/top/net`. No RustFS unified `top` family.
-- [ ] **Trace stream** — MinIO `/v3/trace`. A `trace.rs` handler skeleton
-  exists under `rustfs/src/admin/handlers/` but its registration function is
-  **not** called from `register_admin_routes`, so no route is live.
-
-When one of these lands, register it in `rustfs/src/admin/mod.rs`, extend
-`rustfs/src/admin/route_registration_test.rs`, update
-[admin-route-action-snapshot.md](admin-route-action-snapshot.md) with the
-route/handler/action rows, and move the item out of this checklist.
+When an exception above changes state, edit its row here in the same PR that changes the handler, and extend `rustfs/src/admin/route_registration_test.rs` and [admin-route-action-snapshot.md](admin-route-action-snapshot.md) for admin routes. Do not add "implemented" rows to this document; absence from this list is the implemented claim.

docs/architecture/obs-ecstore-dependency-inventory.md

@@ -1,65 +1,35 @@
 # Observability ECStore Dependency Inventory
 
-This inventory closes the first `rustfs/backlog#735` step: make every
-observability dependency on ECStore visible before introducing traits or moving
-dependency direction.
+**Use this when:** adding, removing, or moving any `rustfs_ecstore` or `rustfs_storage_api` reference inside `crates/obs`.
+**Source of truth:** the `use` block at the top of `crates/obs/src/metrics/storage_api.rs`; the guard in `scripts/check_architecture_migration_rules.sh`.
 
-No behavior or crate movement is planned in inventory PRs. The current boundary
-is `crates/obs/src/metrics/storage_api.rs`; all direct `rustfs_ecstore` and
-`rustfs_storage_api` source references in `rustfs-obs` must stay in that file
-until the contracts below are extracted.
+`rustfs-obs` still depends on `rustfs-ecstore` (`crates/obs/Cargo.toml`). Every direct reference is confined to one boundary file so the dependency can later be replaced by provider traits without touching collectors.
 
 ## Dependency Inventory
 
-| Current symbol in `crates/obs/src/metrics/storage_api.rs` | Consumed by | Classification | Purpose |
-|---|---|---|---|
-| `rustfs_ecstore::api::storage::ECStore` as `ObsStore` | `stats_collector.rs` | Type dependency | Concrete object-store handle used to call storage admin methods and data-usage loaders. |
-| `rustfs_storage_api::{BucketOperations, BucketOptions, StorageAdminApi}` | `stats_collector.rs` | Type and trait dependency | Method-resolution and associated type contracts for bucket listing, backend info, and storage info. |
-| `rustfs_ecstore::api::runtime::object_store_handle` | `stats_collector.rs` | Runtime dependency | Resolves the currently published object-store handle for metric collection. |
-| `rustfs_ecstore::api::data_usage::load_data_usage_from_backend` | `stats_collector.rs` | Behavior dependency | Loads bucket/object usage and is projected into obs-local DTOs before collectors consume it. |
-| `rustfs_ecstore::api::capacity::{get_total_usable_capacity, get_total_usable_capacity_free}` | `stats_collector.rs` | Behavior dependency | Computes usable and free capacity from ECStore storage info. |
-| `rustfs_ecstore::api::bucket::metadata_sys::get_quota_config` | `stats_collector.rs` | Behavior dependency | Reads per-bucket quota limits used in bucket usage metrics. |
-| `rustfs_ecstore::api::bucket::bandwidth::monitor::Monitor` | `runtime_sources.rs`, `stats_collector.rs` | Type and runtime dependency | Reads replication bandwidth reports from the global bucket monitor handle. |
-| `rustfs_ecstore::api::runtime::bucket_monitor` | `runtime_sources.rs` | Runtime dependency | Resolves the global bucket bandwidth monitor for metric collection. |
-| `rustfs_ecstore::api::bucket::replication::get_global_replication_stats` | `storage_api.rs` snapshot helpers | Runtime dependency | Reads replication status, transfer, failure, and site-replication stats, then projects them into obs-local snapshot DTOs. |
-| `rustfs_ecstore::api::bucket::lifecycle::bucket_lifecycle_ops::{GLOBAL_ExpiryState, GLOBAL_TransitionState}` | `runtime_sources.rs`, `stats_collector.rs` | Runtime dependency | Reads lifecycle expiry and transition queue counters. |
-| `rustfs_ecstore::api::error::Result` as `ObsEcstoreResult` | `stats_collector.rs` | Type dependency | Preserves ECStore error propagation while data-usage behavior remains ECStore-owned. |
+The authoritative list is the `pub(crate) use rustfs_ecstore::api::...` block in `crates/obs/src/metrics/storage_api.rs`; it is not copied here. Each import belongs to one of three coupling categories:
 
-## Classification
+| Category | Covers | Examples (aliases defined in the boundary file) |
+|---|---|---|
+| Type coupling | Concrete ECStore types and storage-api traits used for method resolution | `ObsStore`, `ObsEcstoreResult`, `ObsBucketBandwidthMonitor`, the `rustfs_storage_api` trait imports |
+| Runtime handle coupling | Resolving process-wide handles for metric collection | object-store handle, bucket monitor, expiry and transition state handles (`rustfs_ecstore::api::runtime::*`), replication stats read inside the snapshot helpers |
+| Behavior coupling | ECStore-owned computations whose output is projected into obs-local DTOs | data-usage loading, compression totals, quota lookup, usable-capacity math |
 
-The remaining coupling is not just a dependency declaration problem:
-
-- type coupling: `ObsStore`, `ObsEcstoreResult`, `ObsBucketBandwidthMonitor`,
-  `StorageAdminApi`, `BucketOperations`, and `BucketOptions`;
-- runtime handle coupling: object-store handle, bucket monitor, replication
-  stats inside snapshot helpers, expiry state, and transition state;
-- behavior coupling: data-usage loading, quota lookup, and capacity math.
-
-Removing `rustfs-ecstore` from `crates/obs/Cargo.toml` is unsafe until those
-three categories have replacement contracts and compile coverage.
+Collectors consume only the aliases and the obs-local DTOs. Removing `rustfs-ecstore` from `crates/obs/Cargo.toml` is unsafe until all three categories have replacement contracts and compile coverage.
 
 ## Extraction Plan
 
-1. Keep all direct ECStore and storage-api imports centralized in
-   `crates/obs/src/metrics/storage_api.rs`.
-2. Keep projecting ECStore data-usage and replication stats output into
-   obs-local DTOs before collectors consume it.
-3. Introduce obs-owned provider traits for storage info, bucket info, quota,
-   data usage, replication, bandwidth, and lifecycle queue snapshots.
-4. Implement those traits in ECStore or an ECStore-owned adapter crate after the
-   trait shapes are covered by focused tests.
-5. Remove the `rustfs-ecstore` dependency from `rustfs-obs` only after metrics
-   behavior is unchanged through the provider traits.
+1. Keep all direct ECStore and storage-api imports centralized in `crates/obs/src/metrics/storage_api.rs`.
+2. Keep projecting ECStore data-usage and replication stats into obs-local DTOs before collectors consume them.
+3. Introduce obs-owned provider traits for storage info, bucket info, quota, data usage, replication, bandwidth, and lifecycle queue snapshots.
+4. Implement those traits in ECStore or an ECStore-owned adapter crate once the trait shapes are covered by focused tests.
+5. Remove the `rustfs-ecstore` dependency from `rustfs-obs` only after metrics behavior is unchanged through the provider traits.
 
 ## Guardrails
 
-The architecture guard enforces this inventory boundary:
+Enforced by `scripts/check_architecture_migration_rules.sh`:
 
-- `crates/obs/src/metrics/storage_api.rs` is the only `rustfs-obs` source file
-  allowed to reference `rustfs_ecstore` or `rustfs_storage_api`;
-- raw replication stats handles and ECStore replication stat methods must stay
-  behind the snapshot helpers in `crates/obs/src/metrics/storage_api.rs`;
-- `rustfs-obs` must not add `storage_compat.rs` or `ecstore_compat.rs`
-  passthrough bridges;
-- future extraction PRs must update this inventory and the guard in the same
-  reviewed change when a dependency category is removed.
+- `crates/obs/src/metrics/storage_api.rs` is the only `rustfs-obs` source file allowed to reference `rustfs_ecstore` or `rustfs_storage_api`.
+- Raw replication stats handles and ECStore replication stat methods stay behind the snapshot helpers in that file.
+- `rustfs-obs` must not add passthrough bridge modules (a second `storage_api.rs`, an `ecstore_compat.rs`, or similar) that re-export ECStore items to other crates.
+- An extraction PR that removes a dependency category updates this inventory and the guard in the same change.

docs/architecture/overview.md

@@ -1,60 +1,20 @@
 # RustFS Architecture Evolution
 
-This document set tracks the architecture migration from
-[`rustfs/backlog#660`](https://github.com/rustfs/backlog/issues/660).
+**Use this when:** you need the historical framing of the architecture-migration program or the phase order that the migration contracts assume.
+**Source of truth:** [README.md](README.md) is the index of architecture documents; the per-topic contracts it lists are authoritative.
 
 ## Baseline
 
-- Baseline branch: `upstream/main`
-- Baseline commit: `61f0dfbc40f748be313be84d834d8259cf3e19c9`
-- Baseline title: `fix(ecstore): invalidate wiped disk id cache (#3251)`
-- First migration PR type: `docs-only`
+The architecture-migration program (`rustfs/backlog#660`) closed in 2026-07. Its original baseline commit predates the current `main` lineage and is no longer reachable from `main`; treat it as historical. The guardrails the program introduced remain enforced by `scripts/check_architecture_migration_rules.sh`.
 
 ## Core Principle
 
-Cut wrong dependency directions with directories and contracts first, migrate global
-state in small steps next, and split crates only after boundaries are stable. Storage
-hot-path behavior must not drift during this migration.
-
-## Architecture Documents
-
-- [`runtime-lifecycle.md`](runtime-lifecycle.md): runtime, AppContext,
-  startup/readiness, and shutdown contracts.
-- [`readiness-matrix.md`](readiness-matrix.md): request-surface behavior,
-  runtime dependency readiness, probe semantics, and preservation rules.
-- [`s3-tables-support-matrix.md`](s3-tables-support-matrix.md): supported,
-  preview, reference-only, and not-claimed S3 Tables and Iceberg REST Catalog
-  surfaces.
-- [`storage-control-data-plane.md`](storage-control-data-plane.md): boundaries
-  between StorageCore, ECStore, ClusterControlPlane, and BackgroundControllers.
-- [`background-services-inventory.md`](background-services-inventory.md): current
-  scanner, heal, lifecycle, replication, config reload, metrics, and shutdown
-  surface before BackgroundController work.
-- [`background-controller-contract.md`](background-controller-contract.md):
-  desired/current/status/reconcile vocabulary and lifecycle boundaries for
-  future read-only BackgroundController work.
-- [`crate-boundaries.md`](crate-boundaries.md): PR types, crate direction,
-  compatibility rules, and migration guardrails.
-- [`global-state-crate-split-plan.md`](global-state-crate-split-plan.md): late
-  global-state cleanup, runtime-source boundaries, fallback removal rules, and
-  crate-split evaluation criteria.
-- [`obs-ecstore-dependency-inventory.md`](obs-ecstore-dependency-inventory.md):
-  observability-to-ECStore dependency inventory, classification, and extraction
-  guardrails.
-- [`ecstore-config-consumer-inventory.md`](ecstore-config-consumer-inventory.md):
-  current `ecstore::config::{Config, KV, KVS}` definitions, consumers,
-  migration risks, and do-not-change contract.
-- [`ecstore-api-facade-inventory.md`](ecstore-api-facade-inventory.md): current
-  `rustfs_ecstore::api` facade groups, external consumer boundaries, shrink
-  rules, and split dependency inventory.
-- [`config-model-boundary-adr.md`](config-model-boundary-adr.md): target crate,
-  module path, dependency rules, and verification gates for moving the pure
-  server-config model.
-- [`compat-cleanup-register.md`](compat-cleanup-register.md): temporary
-  compatibility code that must be removed later.
+Cut wrong dependency directions with directories and contracts first, migrate global state in small steps next, and split crates only after boundaries are stable. Storage hot-path behavior must not drift during this migration.
 
 ## Phase Order
 
+Historical sequencing of the migration phases. All phases are closed; the diagram is kept because later documents refer to phase names.
+
 ```mermaid
 flowchart LR
     G["Phase 0: Baseline and guardrails"]
@@ -80,10 +40,4 @@ flowchart LR
     GS --> CR
 ```
 
-The first implementation sequence is conservative:
-
-1. Record baseline and migration context.
-2. Establish PR and compatibility rules.
-3. Add dependency and loss-prevention checks in a separate `ci-gate` PR.
-4. Inventory `ecstore::config::{Config, KV, KVS}` before moving any code.
-5. Decide the config model boundary before extracting or migrating consumers.
+The document index is [README.md](README.md). The ECStore facade boundary that the storage phases converged on is described in [ecstore-api-facade-inventory.md](ecstore-api-facade-inventory.md).

docs/architecture/placement-repair-invariants.md

@@ -1,53 +1,38 @@
 # Placement And Repair Invariants
 
-This inventory covers `G-012` for `rustfs/backlog#666`. It records the current
-object placement, readiness, lock quorum, scanner, and repair boundaries that
-later scheduler or topology work must preserve.
+**Use this when:** changing anything that resolves an object to a pool, set, or disk, or that admits scanner or heal work; these are the behaviors later scheduler or topology work must preserve.
+**Source of truth:** `Sets::get_disks_by_key` / `get_hashed_set_index` in `crates/ecstore/src/core/sets.rs`; `DistributionAlgoVersion` in `crates/ecstore/src/layout/format.rs`; `crc_hash` / `sip_hash` in `crates/utils/src/hash.rs`; `ScannerCycleBudget` in `crates/scanner/src/scanner_budget.rs`; `crates/scanner/src/scanner_heal_admission_baseline.rs`.
 
 ## Object To Set Hash Rule
 
-Objects reach a set through `Sets::get_disks_by_key`, which calls
-`get_hashed_set_index` on the object key:
+Objects reach a set through `Sets::get_disks_by_key`, which calls `get_hashed_set_index` on the object key:
 
 - `DistributionAlgoVersion::V1` uses `crc_hash(input, set_count)`.
-- `DistributionAlgoVersion::V2` and `V3` use
-  `sip_hash(input, set_count, format_id_bytes)`.
-- The format ID is part of the V2/V3 distribution seed, so changing the seed,
-  object key, set count, or algorithm changes placement.
+- `DistributionAlgoVersion::V2` and `V3` use `sip_hash(input, set_count, format_id_bytes)`.
+- The format ID is part of the V2/V3 distribution seed, so changing the seed, object key, set count, or algorithm changes placement.
 
-Preservation rule: every object read, write, list, heal, repair, and
-decommission path that resolves a set for an existing object must preserve the
-same object key and format distribution algorithm.
+Preservation rule: every object read, write, list, heal, repair, and decommission path that resolves a set for an existing object must preserve the same object key and format distribution algorithm.
 
 ## Pool, Set, And Disk Assignment Boundary
 
 Pool selection is separate from set hashing:
 
-- Existing objects are discovered across pools and resolved to the best current
-  pool candidate before reads or updates continue.
-- New object writes select an available pool from current per-pool free-space
-  inputs after suspended or rebalancing pools are skipped.
-- Set selection inside a pool still uses the object-to-set hash rule above.
-- Disk index assignment comes from endpoint and format metadata, not from a
-  scheduler decision.
+- Existing objects are discovered across pools and resolved to the best current pool candidate before reads or updates continue.
+- New object writes select an available pool from current per-pool free-space inputs after suspended or rebalancing pools are skipped.
+- Set selection inside a pool uses the object-to-set hash rule above.
+- Disk index assignment comes from endpoint and format metadata, not from a scheduler decision.
 
-Boundary rule: schedulers may influence admission, worker concurrency, or
-buffer sizing, but they must not rewrite pool, set, or disk indexes.
+Boundary rule: schedulers may influence admission, worker concurrency, or buffer sizing, but they must not rewrite pool, set, or disk indexes.
 
 ## Readiness And Lock Quorum Boundary
 
-Runtime readiness currently checks storage and lock health independently:
+Runtime readiness checks storage and lock health independently:
 
-- Storage readiness requires every observed set to meet write quorum based on
-  the set drive count and storage class data/parity shape.
-- Lock readiness aggregates per-set lock-client host quorum and fails fast if
-  any set loses quorum.
-- Object and bucket mutations acquire namespace locks through the existing
-  storage lock wrappers before changing object or bucket state.
+- Storage readiness requires every observed set to meet write quorum based on the set drive count and storage class data/parity shape.
+- Lock readiness aggregates per-set lock-client host quorum and fails fast if any set loses quorum.
+- Object and bucket mutations acquire namespace locks through the existing storage lock wrappers before changing object or bucket state.
 
-Boundary rule: readiness and lock quorum must stay set-aware. A global healthy
-disk count or global connected-host count is not sufficient when any individual
-set is below quorum.
+Boundary rule: readiness and lock quorum must stay set-aware. A global healthy disk count or global connected-host count is not sufficient when any individual set is below quorum.
 
 ## Scanner Budget Preservation
 
@@ -55,43 +40,45 @@ Scanner cycles are bounded by `ScannerCycleBudget`:
 
 - Runtime budget cancels the child token after the configured duration.
 - Object budget cancels after the configured object count.
-- Directory budget rejects additional directories and cancels with the
-  directories reason.
+- Directory budget rejects additional directories and cancels with the directories reason.
 - Partial-cycle metrics and checkpoints use the budget reason.
 
-Preservation rule: later scheduler work can change how scan cycles are admitted
-only if it preserves the budget reason, checkpoint reason, and child-token
-cancellation behavior.
+Preservation rule: later scheduler work can change how scan cycles are admitted only if it preserves the budget reason, checkpoint reason, and child-token cancellation behavior.
 
 ## Heal Admission Preservation
 
-Scanner and background repair work enter the heal manager through explicit
-admission:
+Scanner and background repair work enter the heal manager through explicit admission:
 
-- Scanner object heal requests are low priority and may be accepted, merged,
-  rejected as full, or dropped.
-- Required/high-priority heal candidates escalate on non-admission instead of
-  silently disappearing.
-- Heal queue admission deduplicates queued and active work unless the request
-  explicitly forces admission.
-- Full queues can drop low-priority work or displace lower-priority work for a
-  higher-priority request according to current manager rules.
+- Scanner object heal requests are low priority and may be accepted, merged, rejected as full, or dropped.
+- Required/high-priority heal candidates escalate on non-admission instead of silently disappearing.
+- Heal queue admission deduplicates queued and active work unless the request explicitly forces admission.
+- Full queues can drop low-priority work or displace lower-priority work for a higher-priority request according to current manager rules.
 
-Preservation rule: repair scheduling changes must keep admission outcomes
-observable and must not convert rejected or dropped repair work into silent
-success.
+Preservation rule: repair scheduling changes must keep admission outcomes observable and must not convert rejected or dropped repair work into silent success.
+
+### Scanner/heal admission entry points
+
+No cluster-wide coordinator or second generation token exists for scanner/heal admission; each entry point keeps its own guard. `crates/scanner/src/scanner_heal_admission_baseline.rs` `include_str!`s the scanner sources and asserts the named guards are still present, so a rename or guard removal fails that test instead of silently leaving this table stale. It also encodes the investigation matrix (scanner read and heal read may overlap; heal write conflicts with scanner reads; data-movement write conflicts with all work; independent sets stay concurrent) without claiming production enforces it.
+
+| Work | Entry point | Current guard | Fallback / namespace semantics |
+|---|---|---|---|
+| Scanner read/list | `nsscanner_disk` in `crates/scanner/src/scanner_io/io_disk.rs` | Per-disk `start_scan()` guard; bucket lifecycle/replication/object-lock reads precede `scan_data_folder` | Scanner keeps its local disk and durable cursor; no HealManager set-level admission is consulted |
+| Scanner metadata read | Object-size and metadata branches in `crates/scanner/src/scanner_folder.rs` | Scanner cycle budget and per-disk scan marker | Corrupt metadata records the pending scanner ledger; MRF is a hint, not the durable owner |
+| Scanner heal admission | `send_required_scanner_heal_request` in `crates/scanner/src/scanner_folder.rs` | Manager queue dedup and pending ledger (`update_pending_scanner_heal_after_admission`) | MRF `Enqueued` / `Coalesced` is ledger-only; rejected MRF keeps immediate heal plus ledger |
+| Heal auto scan | `start_auto_disk_scanner` in `crates/heal/src/heal/manager/auto_scan.rs` | Queue-first then active-task check; replacement recovery blocklist | Scanning disks remain candidates when degraded quorum needs them; they are not globally excluded |
+| Heal object read | `heal_object` in `crates/ecstore/src/set_disk/ops/heal.rs` | Namespace write lock (`get_write_lock`) unless `no_lock`; reads file info before commit | The namespace lock is object-scoped and does not claim scanner cycle ownership |
+| Disk selection | `get_online_disks_with_healing_and_info` in `crates/ecstore/src/set_disk/ops/locking.rs` | Healing disks are ordered after new disks; scanning disks may remain candidates | Degraded/quorum fallback is preserved |
+| Data movement | `wait_for_data_movement_admission` in `crates/ecstore/src/data_movement/backpressure.rs` | Storage-owned backpressure on foreground pressure; no second coordinator | Any future admission token must be validated at the final metadata/format/delete commit (see [unified-object-generation.md](unified-object-generation.md)) |
+
+Rules: cancellation or a local lease alone is not a fence; if a fixture ever demonstrates a stale destructive write, the fix extends the storage-owned generation/admission primitive and validates the token at the final commit rather than adding a coordinator.
 
 ## Behavior Change Gates
 
 Any later placement or repair PR must use the following gates:
 
-- Placement gate: prove object-to-set hashing is unchanged for existing object
-  keys and format algorithms.
-- Pool gate: prove pool selection does not choose suspended or rebalancing
-  pools unless the existing path already allows it.
+- Placement gate: prove object-to-set hashing is unchanged for existing object keys and format algorithms.
+- Pool gate: prove pool selection does not choose suspended or rebalancing pools unless the existing path already allows it.
 - Quorum gate: prove storage readiness and lock readiness remain per-set.
 - Scanner gate: prove scan budget reason and checkpoint mapping remain stable.
-- Heal gate: prove low-priority scanner heal, forced heal, duplicate merge, and
-  queue-full outcomes remain distinct.
-- Rollback gate: if a new scheduler sidecar is disabled, placement and repair
-  must fall back to the current direct ECStore/scanner/heal behavior.
+- Heal gate: prove low-priority scanner heal, forced heal, duplicate merge, and queue-full outcomes remain distinct.
+- Rollback gate: if a new scheduler sidecar is disabled, placement and repair must fall back to the current direct ECStore/scanner/heal behavior.

docs/architecture/readiness-matrix.md

@@ -1,8 +1,7 @@
 # Readiness Matrix
 
-This document records the current request and dependency behavior around
-startup readiness. It is a behavior-preservation baseline for architecture
-migration work, not a new readiness policy.
+**Use this when:** changing what a request surface does before storage or IAM is ready, changing probe semantics, or adding a runtime dependency that readiness must wait for.
+**Source of truth:** `rustfs/src/server/readiness.rs` (probe paths, `Retry-After`), `crates/common/src/readiness.rs` (`StorageReady`, `IamReady`, `FullReady`), `crates/config/src/constants/health.rs` (`RUSTFS_HEALTH_*` gates). This matrix is a behavior-preservation baseline, not a new readiness policy.
 
 ## Request Behavior Matrix
 
@@ -23,7 +22,7 @@ migration work, not a new readiness policy.
 | IamReady | Inline IAM bootstrap or deferred IAM recovery publication. | Yes. | Deferred recovery can publish IAM readiness after HTTP has already started. |
 | Lock quorum | Per-set write quorum readiness. | Yes. | Do not replace the distributed lock quorum check with node count or endpoint count. |
 | Peer health | `peer_health_ready` runtime status. | Only when `RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` is enabled. | The gate is disabled by default; unknown peer health degrades readiness only when enabled. |
-| KMS compatibility | KMS health compatibility readiness. | Only when the KMS compatibility readiness check is enabled. | KMS startup fatality and health reporting remain separate from pure docs work. |
+| KMS compatibility | KMS health compatibility readiness. | Only when `RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE` is enabled (default off; `crates/config/src/constants/health.rs`). | When enabled, `/health/ready` additionally requires the KMS service to be running if a global KMS manager exists. KMS startup fatality and health reporting remain separate from pure docs work. |
 
 Effective `FullReady` is:
 

docs/architecture/runtime-capability-contracts.md

@@ -1,70 +1,28 @@
 # Runtime Capability Contracts
 
-This document records the `rustfs/backlog#660` PR-08 and PR-09 contract slice.
-It adds read-only observability and topology snapshot shapes to
-`rustfs-storage-api` without coupling the contract crate to runtime, ECStore,
-admin routes, profiling, or observability implementation crates.
+**Use this when:** changing the read-only observability or topology snapshot contracts in `rustfs-storage-api`, their RustFS providers, or the `storage_classes` payload of `GET /rustfs/admin/v4/runtime/capabilities`.
+**Source of truth:** `ObservabilitySnapshot` in `crates/storage-api/src/observability.rs`; `TopologySnapshot` in `crates/storage-api/src/topology.rs`; `CapabilityState` and `CapabilitySnapshotError` in `crates/storage-api/src/capability.rs`; providers in `rustfs/src/runtime_capabilities.rs`; storage-class constants in `crates/ecstore/src/config/storageclass.rs`.
 
-## Observability Snapshot Contract
+## Snapshot Contracts
 
-`ObservabilitySnapshot` records:
+Field lists live on the defining types and are not repeated here.
 
-- Runtime telemetry capability state.
-- Userspace CPU and memory profiling capability state.
-- Process, system, and cgroup memory sampling state.
-- Platform support for target triple, OS, architecture, allocator, eBPF, and
-  NUMA capability.
+| Contract | Defining type | RustFS provider | Rule |
+|---|---|---|---|
+| Observability | `ObservabilitySnapshot` | `RustFsObservabilitySnapshotProvider` | Reports runtime telemetry, profiling, memory-sampling, platform, allocator, eBPF, and NUMA capability as `CapabilityState` values without starting telemetry, profiling, allocator reclaim, or memory-observability workers. |
+| Topology | `TopologySnapshot` | `EndpointTopologySnapshotProvider` | Maps `EndpointServerPools` into pool/set/disk indexes, optional stable IDs, and optional zone/rack/node/media/NUMA labels without changing endpoint construction, placement, readiness, locks, or ECStore metadata. Local file endpoint paths are never used as disk IDs or labels; extra labels go in the `additional` map so future inventory labels need no ECStore type leakage. |
 
-Unsupported, disabled, and unknown states are represented by `CapabilityState`
-instead of failing snapshot construction. The contract is intentionally read
-only and does not replace existing profiling routes, telemetry APIs, exporter
-pipelines, or startup behavior.
-
-## Topology Snapshot Contract
-
-`TopologySnapshot` records:
-
-- Pool, set, and disk identity indexes plus optional stable IDs.
-- Optional zone, rack, node, media, NUMA, and additional labels.
-- Topology-wide profiling, NUMA, failure-domain label, and media-label
-  capability states.
-- Per-disk media, failure-domain, NUMA, and profiling capability states.
-
-Missing labels are represented as absent `Option` values. Extra topology labels
-belong in the `additional` label map, so future inventory labels do not require
-ECStore type leakage.
+Unsupported, disabled, and unknown states are values of `CapabilityState`, not construction failures. Missing labels are `None`. Providers map implementation failures into `CapabilitySnapshotError` before crossing the contract boundary. Neither contract replaces existing profiling routes, telemetry APIs, exporter pipelines, or startup behavior.
 
 ## Boundary Rules
 
-- No `rustfs-ecstore`, `rustfs-obs`, Axum, KMS, admin route, OTEL, eBPF, or
-  profiling implementation dependency is added to `rustfs-storage-api`.
-- No placement, membership, NUMA pinning, profiling, startup, admin route, or
-  exporter behavior changes are part of this contract slice.
-- Providers must map implementation failures into `CapabilitySnapshotError`
-  before crossing the contract boundary.
-
-## RustFS Provider Slice
-
-`rustfs/src/runtime_capabilities.rs` wires the contracts to RustFS runtime
-owners through read-only providers:
-
-- `RustFsObservabilitySnapshotProvider` maps current dial9, profiling,
-  memory-sampling, platform, allocator, eBPF, and NUMA capability state without
-  starting telemetry, profiling, allocator reclaim, or memory-observability
-  workers.
-- `EndpointTopologySnapshotProvider` maps `EndpointServerPools` into pool, set,
-  and disk topology snapshots without changing endpoint construction,
-  placement, readiness, locks, or ECStore metadata. Local file endpoint paths are
-  intentionally not used as disk IDs or labels.
-
-Unsupported or unavailable runtime capabilities are reported as `unsupported`
-or `unknown` contract states instead of activating fallback behavior.
+- `rustfs-storage-api` gains no dependency on `rustfs-ecstore`, `rustfs-obs`, Axum, KMS, admin routes, OTEL, eBPF, or profiling implementation crates.
+- Providers are read-only. Adding or changing a provider changes no placement, membership, NUMA pinning, profiling, startup, admin-route, or exporter behavior.
+- Unsupported or unavailable runtime capabilities are reported as `unsupported` or `unknown`; they never activate fallback behavior.
 
 ## Storage-Class Write Contract
 
-Authenticated clients discover the storage-class write contract from
-`GET /rustfs/admin/v4/runtime/capabilities`. The additive
-`storage_classes` object is versioned independently from the route:
+Authenticated clients discover the storage-class write contract from `GET /rustfs/admin/v4/runtime/capabilities`. The additive `storage_classes` object is versioned independently from the route:
 
 ```json
 {
@@ -77,15 +35,11 @@ Authenticated clients discover the storage-class write contract from
 }
 ```
 
-`supported_write_classes` is the complete client-selectable write allowlist.
-Any other value fails before object or multipart mutation with the stable S3
-error named by `unsupported_write_error`. `legacy_label_behavior` means
-non-transitioned historical label-only metadata is reported as its effective
-local class; actual lifecycle transition tier names remain unchanged.
+| Field | Meaning |
+|---|---|
+| `supported_write_classes` | The complete client-selectable write allowlist. Any other value fails before object or multipart mutation with the S3 error named by `unsupported_write_error`. |
+| `unsupported_write_error` | Stable S3 error code (`UNSUPPORTED_WRITE_ERROR` in `crates/ecstore/src/config/storageclass.rs`). |
+| `legacy_label_behavior` | Non-transitioned historical label-only metadata is reported as its effective local class; lifecycle transition tier names are unchanged. |
+| `contract_version` | Consumers must branch on it before assigning meaning to future fields. |
 
-The values are sourced from
-[`crates/ecstore/src/config/storageclass.rs`](../../crates/ecstore/src/config/storageclass.rs),
-which also owns write validation and response normalization. Consumers must
-branch on `contract_version` before assigning meaning to future fields. The
-admin route continues to require `ServerInfoAdminAction`; capability discovery
-does not weaken authentication or authorization.
+Values, write validation, and response normalization are owned by `crates/ecstore/src/config/storageclass.rs`. The route continues to require `ServerInfoAdminAction`; capability discovery does not weaken authentication or authorization.

docs/architecture/runtime-lifecycle.md

@@ -1,120 +1,57 @@
 # Runtime And Lifecycle Contracts
 
-Runtime and lifecycle work must preserve startup ordering, readiness behavior, and
-shutdown semantics.
+**Use this when:** moving or reordering anything in `rustfs/src/startup_*.rs`, changing readiness publication, or touching shutdown ordering.
+**Source of truth:** the `rustfs/src/startup_*.rs` modules listed below; readiness semantics in [readiness-matrix.md](readiness-matrix.md); global-state targets in [global-state-inventory.md](global-state-inventory.md).
+
+Runtime and lifecycle work must preserve startup ordering, readiness behavior, and shutdown semantics.
 
 ## Startup And Readiness
 
-- HTTP can listen early, but normal requests must remain behind readiness gates.
-- Effective `FullReady = storage_ready && iam_ready && lock_quorum_ready &&
-  peer_health_ready`; `peer_health_ready` is true by default unless
-  `RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` is enabled.
-- Boot phases must keep the old fatal and non-fatal boundaries.
-- AppContext migration keeps context-first lookup with global fallback until the
-  global path is proven unused.
+- HTTP can listen early, but normal requests stay behind the readiness gate.
+- The `FullReady` formula, its dependencies, and the `RUSTFS_HEALTH_PEER_READY_CHECK_ENABLE` gate are defined once in [readiness-matrix.md](readiness-matrix.md); do not restate them elsewhere.
+- Boot phases keep the existing fatal and non-fatal boundaries.
+- AppContext migration keeps context-first lookup with global fallback until the global path is proven unused.
 - Notify and audit lifecycle behavior must not drift during lifecycle movement.
-- IAM and KMS startup, deferred recovery, and fatal boundary behavior must not be
-  changed by pure movement PRs.
-- Request-surface and dependency details are tracked in
-  [`readiness-matrix.md`](readiness-matrix.md).
+- IAM and KMS startup, deferred recovery, and fatal-boundary behavior must not be changed by pure movement PRs.
 
-## Service Registry Scope
+## Startup Module Ownership
 
-`ServiceRegistry` is only for lifecycle and shutdown ordering. It must not become a
-general dependency injection container.
+Each module owns one concern; orchestration order is owned by `startup_services`, `startup_lifecycle`, and `startup_shutdown`. Movement PRs may pass handles between modules but must not reorder the steps a module owns.
 
-Allowed responsibilities:
-
-- Register start and stop order.
-- Expose read-only status snapshots.
-- Coordinate graceful shutdown.
-
-Disallowed responsibilities:
-
-- Construct arbitrary dependencies for business logic.
-- Hide globals behind a service-locator API.
-- Change startup side effects while moving code.
+| Module (`rustfs/src/`) | Owns |
+|---|---|
+| `startup_entrypoint.rs` | CLI command dispatch into preflight and the runtime lifecycle. |
+| `startup_preflight.rs` | License init, external env compatibility, runtime foundation bootstrap. |
+| `startup_runtime.rs` | Runtime foundation orchestration; outbound TLS fatal boundary when configured material fails to load. |
+| `startup_runtime_hooks.rs` | Startup diagnostics, profiling hook dispatch, default crypto provider installation. |
+| `startup_tls_material.rs` | Outbound TLS material loading, global publication, generation recording, TLS metrics init. |
+| `startup_runtime_sources.rs` | Process-local runtime source publication (port, buffer profile, KMS manager, TLS generation). |
+| `startup_fs_guard.rs` | Unsupported-filesystem policy enforcement for endpoint paths. |
+| `startup_deadlock.rs` | Deadlock detector state logging. |
+| `startup_server.rs` | HTTP listener start and `ServiceStateManager` publication. |
+| `startup_storage.rs` | Endpoints, local disks, ECStore, lock clients, global config, background replication init; `StorageReady`. |
+| `startup_bucket_metadata.rs` | Bucket metadata system init, legacy meta-bucket import (`try_migrate_bucket_metadata`, `try_migrate_iam_config`), resync intents. |
+| `startup_iam.rs` | IAM init, deferred recovery, `IamReady` publication. |
+| `startup_auth.rs` | OIDC and federated identity setup. |
+| `startup_notification.rs` | Notification system and bucket notification configuration. |
+| `startup_audit.rs` | Event notifier and audit system start. |
+| `startup_observability.rs` | Auto-tuner, update check, server info, compression totals. |
+| `startup_background.rs` | Scanner, heal, bitrot self-test, workload-admission provider publication. |
+| `startup_protocols.rs` | FTP/FTPS/SFTP/WebDAV sidecar start and shutdown senders. |
+| `startup_optional_runtime_sidecars.rs` | Handles, shutdown planning, and shutdown execution for optional sidecars that are not readiness boundaries (currently protocol servers only). New sidecars enter here with explicit shutdown handles and status snapshots, not ad hoc work in `startup_services`. |
+| `startup_services.rs` | Orchestration order of runtime service startup: KMS, optional runtimes, audit, metadata, IAM, auth, notification, background services, observability. |
+| `startup_lifecycle.rs` | Ready publication, global init-time publication, scanner startup, shutdown-signal wait, shutdown delegation, final stopped-state log. |
+| `startup_shutdown.rs` | The shutdown sequence (see below). |
+| `startup_embedded.rs`, `startup_embedded_optional.rs` | Embedded-mode reuse of the phase owners above (see below). |
 
 ## Shutdown Lifecycle Boundary
 
-`startup_shutdown` owns the main shutdown sequence after the process receives a
-shutdown signal. Startup modules may pass handles into this boundary, but they
-must not reorder runtime-token cancellation, background service shutdown,
-optional runtime shutdown planning, notifier/audit/profiling shutdown, HTTP
-shutdown, optional runtime waits, or final service-state publication.
+`startup_shutdown` owns the main shutdown sequence after the process receives a shutdown signal. Startup modules may pass handles into this boundary, but they must not reorder runtime-token cancellation, background service shutdown, optional runtime shutdown planning, notifier/audit/profiling shutdown, HTTP shutdown, optional runtime waits, or final service-state publication.
 
-## Startup Lifecycle Boundary
+## Embedded Startup Reuse
 
-`startup_lifecycle` owns the ready-to-shutdown orchestration after runtime
-services initialize. Service modules may return initialized handles into this
-boundary, but they must not reorder ready publication, global init-time
-publication, scanner startup, shutdown-signal wait, shutdown delegation, or the
-final stopped-state log.
-
-## Startup Service Component Boundary
-
-`startup_service_components` owns individual runtime service startup component
-helpers while `startup_services` preserves their orchestration order. Migration
-PRs must not change KMS, optional runtime, audit, metadata, IAM, auth,
-notification, background service, or observability startup ordering while moving
-these helpers.
-
-## Optional Runtime Boundary
-
-`startup_optional_runtime_sidecars` owns startup handles, shutdown planning, and
-shutdown execution for optional runtime services that are not readiness
-boundaries. `startup_optional_runtimes` remains a compatibility handoff for the
-old module path. The current owner set is protocol servers only. Future optional
-sidecars must enter this boundary with explicit shutdown handles and status
-snapshots instead of adding ad hoc startup or shutdown work to
-`startup_services`.
-
-## Startup Runtime Hook Boundary
-
-`startup_runtime_hooks` owns runtime hook side effects that wrap the startup
-foundation but are not TLS material loading: startup diagnostics, profiling
-hook dispatch, and default crypto provider installation. `startup_runtime`
-preserves BOOT-006 orchestration and outbound TLS fatal behavior, while
-`startup_profiling` remains a compatibility handoff for the old profiling hook
-path.
-
-## Startup TLS Material Boundary
-
-`startup_tls_material` owns configured outbound TLS material loading, global
-outbound TLS publication, generation recording, and TLS metrics initialization.
-`startup_runtime` still owns BOOT-006 ordering and must preserve the fatal
-boundary when configured TLS material fails to load.
-
-## Embedded Startup Phase Reuse
-
-Embedded startup should use the same startup server and storage phase owners for
-listen context, endpoint/local disk setup, storage runtime setup, readiness
-publication, and replication startup. Embedded-specific behavior still owns its
-stable-port requirement, one-shot global initialization guard placement, S3-only
-HTTP listener, and non-fatal KMS/audit/notification policy.
-
-## Embedded Runtime Service Reuse
-
-Embedded runtime service setup should share startup service helpers for optional
-service initialization, bucket metadata/IAM setup, notification setup, and
-shutdown cleanup. Embedded-specific behavior still owns warning-only
-KMS/audit/notification failures, no binary-only background sidecars, no state
-manager, and the one-shot server handle cleanup used by embedded shutdown.
-
-## Embedded Lifecycle Publication Reuse
-
-Embedded ready publication should share startup lifecycle helpers for IAM
-readiness publication, global init-time publication, and ready-state logging.
-Embedded-specific behavior still owns server handle construction, endpoint
-address normalization, and process-local shutdown cleanup.
+Embedded startup reuses the same phase owners as the binary: server and storage phases for listen context, endpoint/local disk setup, storage runtime setup, readiness publication, and replication startup; service helpers for optional service init, bucket metadata/IAM setup, notification setup, and shutdown cleanup; lifecycle helpers for IAM readiness publication, global init-time publication, and ready-state logging. Embedded-specific behavior that stays in `startup_embedded*.rs`: stable-port requirement, one-shot global initialization guard placement, S3-only HTTP listener, warning-only KMS/audit/notification failures, no binary-only background sidecars, no state manager, server handle construction, endpoint address normalization, and process-local one-shot shutdown cleanup.
 
 ## AppContext Foundation
 
-Early AppContext work should split resolver files and add compatibility tests before
-boot extraction or consumer migration. This keeps the migration context-first while
-preserving the old global fallback path during transition.
-
-AppContext remains a context-first facade, not a full replacement for every
-process global. New migration work must keep fallback reads inside owner-local
-runtime-source boundaries and follow the global-state target inventory in
-[`global-state-inventory.md`](global-state-inventory.md).
+AppContext is a context-first facade, not a full replacement for every process global. Resolver files are split and covered by compatibility tests before boot extraction or consumer migration, so the old global fallback path keeps working during transition. New migration work keeps fallback reads inside owner-local runtime-source boundaries and follows the target inventory in [global-state-inventory.md](global-state-inventory.md).

docs/architecture/s3-compatibility-matrix.md

@@ -1,44 +1,30 @@
 # S3 Compatibility Matrix
 
-This matrix records the user-facing S3 compatibility claim for RustFS and ties
-it to the executable Ceph s3tests lists under `scripts/s3-tests/`.
+**Use this when:** writing or checking a user-facing S3 compatibility claim, or moving a Ceph s3tests case between lists.
+**Source of truth:** the test lists under `scripts/s3-tests/` and the runner `scripts/s3-tests/run.sh`; counts are derived from those files and are not recorded here.
 
 ## Current Claim
 
-RustFS provides broad S3 API compatibility for supported features. It does not
-claim complete coverage of every standard or vendor-specific S3 behavior.
-
-The root README should use the same wording: supported S3-compatible clients and
-features are covered by the compatibility matrix and test lists.
+RustFS provides broad S3 API compatibility for supported features. It does not claim complete coverage of every standard or vendor-specific S3 behavior. The root README uses the same wording: supported S3-compatible clients and features are covered by the compatibility matrix and test lists.
 
 ## Test List Sources
 
-| List | Purpose | Current count | Source |
-|---|---:|---:|---|
-| Implemented tests | Standard S3 tests expected to pass and used by the default local s3tests run. | 452 | `scripts/s3-tests/implemented_tests.txt` |
-| Lifecycle behavior tests | Expiration behavior cases gated by the dedicated `s3-lifecycle-behavior-tests` lane (debug-accelerated day + scanner enabled). | 5 | `scripts/s3-tests/lifecycle_behavior_tests.txt` |
-| Unimplemented tests | Standard S3 features planned but not yet implemented. | 17 | `scripts/s3-tests/unimplemented_tests.txt` |
-| Excluded tests | Vendor-specific or intentionally unsupported behavior excluded from RustFS compatibility gating. | 273 | `scripts/s3-tests/excluded_tests.txt` |
+| List | Purpose | Source |
+|---|---|---|
+| Implemented tests | Standard S3 tests expected to pass; the default local s3tests run. | `scripts/s3-tests/implemented_tests.txt` |
+| Lifecycle behavior tests | Days-based expiration cases gated by the `s3-lifecycle-behavior-tests` lane in `.github/workflows/ci.yml`. | `scripts/s3-tests/lifecycle_behavior_tests.txt` |
+| Unimplemented tests | Standard S3 features not yet passing. | `scripts/s3-tests/unimplemented_tests.txt` |
+| Excluded tests | Vendor-specific or intentionally unsupported behavior excluded from RustFS gating. | `scripts/s3-tests/excluded_tests.txt` |
 
-Counts ignore blank lines and comments.
-
-The lifecycle behavior lane runs real Days-based expiration cases that need
-`RUSTFS_ILM_DEBUG_DAY_SECS` (Ceph `lc_debug_interval` equivalent) and an enabled
-background scanner; it cannot share the default single-server gate because a
-global debug day would also shrink the `x-amz-expiration` header asserted by the
-`test_lifecycle_expiration_header_*` cases. See `scripts/s3-tests/run.sh`
-(`IMPLEMENTED_TESTS_FILE` override) and the `s3-lifecycle-behavior-tests` job in
-`.github/workflows/ci.yml`.
+Counts ignore blank lines and comments; compute them from the files. The lifecycle lane runs separately because its cases need `RUSTFS_ILM_DEBUG_DAY_SECS` and an enabled scanner, and a global debug day would also shrink the `x-amz-expiration` header asserted by `test_lifecycle_expiration_header_*`; see `IMPLEMENTED_TESTS_FILE` in `scripts/s3-tests/run.sh`.
 
 ## Supported Coverage
 
-The implemented test list currently covers the common object-storage surface:
-
 | Area | Status | Evidence |
 |---|---|---|
 | Bucket create/delete/list/head | Supported | `implemented_tests.txt` |
 | Object put/get/delete/copy/head | Supported | `implemented_tests.txt` |
-| CopyObject checksums (CRC32, CRC32C, CRC64NVME, SHA1, SHA256, MD5, SHA512, XXHASH3, XXHASH64, XXHASH128), including source preservation and explicit override | Supported in the first RustFS release containing this change | `crates/e2e_test/src/copy_object_checksum_test.rs` |
+| CopyObject checksums (CRC32, CRC32C, CRC64NVME, SHA1, SHA256, MD5, SHA512, XXHASH3, XXHASH64, XXHASH128), including source preservation and explicit override | Supported | `crates/e2e_test/src/copy_object_checksum_test.rs` |
 | ListObjects/ListObjectsV2 prefix, delimiter, marker, max-keys | Supported | `implemented_tests.txt` |
 | Multipart upload create/upload/complete/abort and selected multipart copy/checksum/object-attribute behavior | Supported | `implemented_tests.txt` |
 | Bucket and object tagging | Supported | `implemented_tests.txt` |
@@ -50,33 +36,25 @@ The implemented test list currently covers the common object-storage surface:
 | SSE-C and selected SSE-KMS edge cases | Supported | `implemented_tests.txt` |
 | Selected versioning, object-lock, checksum, CORS, raw request, and conditional write behavior | Supported | `implemented_tests.txt` |
 
-"Supported" for the SSE row means RustFS encrypts and decrypts its own objects. It does not mean RustFS can read objects another implementation encrypted: objects MinIO wrote with SSE-S3, SSE-KMS, or SSE-C are not readable by RustFS today, which matters when migrating. See [MinIO file-format interoperability, Part C](minio-file-format-compat.md#part-c--server-side-encryption-sse) and rustfs/backlog#1638.
+"Supported" for the SSE row means RustFS encrypts and decrypts its own objects. MinIO SSE objects (SSE-S3, SSE-KMS, SSE-C) are not readable in default builds; see [minio-file-format-compat.md Part C](minio-file-format-compat.md#part-c--server-side-encryption-sse) for the `rio-v2` migration build.
 
-## Planned Standard Coverage
+## Not Yet Passing
 
-These are standard S3 areas that remain planned work and must not be described
-as already complete:
+Standard S3 areas that must not be described as complete:
 
 | Area | Status | Evidence |
 |---|---|---|
-| Bucket access logging | Planned | `unimplemented_tests.txt` |
-| POST Object form upload checksum handling | Planned | `unimplemented_tests.txt` |
-| Bucket ownership controls | Planned | `unimplemented_tests.txt` |
+| Bucket access logging | Handlers exist (`get_bucket_logging`, `put_bucket_logging` in `rustfs/src/storage/ecfs.rs`); the `test_*bucket_logging*` s3tests cases are still listed as unimplemented | `unimplemented_tests.txt` |
+| POST Object form upload checksum handling | Not yet passing | `unimplemented_tests.txt` |
+| Bucket ownership controls | No handler | `unimplemented_tests.txt` |
 | Multipart upload listing and part lookup compatibility edge cases | Not part of default gate | `excluded_tests.txt` |
 | IAM-account or multi-storage-class dependent cases | Not part of default gate | `unimplemented_tests.txt` |
 | Tenanted bucket policy edge cases | Needs investigation | `unimplemented_tests.txt` |
 
 ## Intentional Exclusions
 
-`excluded_tests.txt` contains tests that should not block the RustFS
-compatibility gate. They fall into two classes:
-
-- vendor-specific or non-portable behavior not required for RustFS S3
-  compatibility;
-- intentionally unsupported product behavior, such as ACL authorization.
+`excluded_tests.txt` holds tests that must not block the compatibility gate: vendor-specific or non-portable behavior, and intentionally unsupported product behavior such as ACL authorization.
 
 ## Update Rule
 
-When a planned S3 feature is implemented, move its passing test entries from
-`unimplemented_tests.txt` to `implemented_tests.txt`, update this matrix, and
-avoid changing README wording beyond the supported coverage.
+When a feature starts passing, move its test entries from `unimplemented_tests.txt` to `implemented_tests.txt` and update the row here in the same PR. Do not change README wording beyond the supported coverage. Handler-level status (missing, stubbed, or diverging endpoints) is tracked in [minio-rustfs-router-compatibility.md](minio-rustfs-router-compatibility.md).

docs/architecture/s3-tables-support-matrix.md

@@ -1,327 +1,152 @@
 # S3 Tables Support Matrix
 
-This matrix records the RustFS S3 Tables surfaces that are supported,
-previewed, referenced, or intentionally not claimed. It is the release-facing
-boundary for the Iceberg REST Catalog work in RustFS.
+**Use this when:** writing a release note, README claim, or client-compatibility statement about RustFS S3 Tables / Iceberg REST Catalog, or deciding whether a feature is supported, preview, or not claimed.
+**Source of truth:** the table-catalog handlers under `rustfs/src/admin/handlers/table_catalog/`; the conformance scripts and their README in `scripts/table-catalog/`; the durable-backing cutover procedure in [docs/operations/s3-tables-cutover-runbook.md](../operations/s3-tables-cutover-runbook.md).
 
-RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on
-top of the RustFS S3 data plane. This document does not claim full parity with
-the AWS S3 Tables control-plane API or with every vendor-specific Iceberg
-catalog extension.
+RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on top of the RustFS S3 data plane. It does not claim parity with the AWS S3 Tables control-plane API or with vendor-specific Iceberg catalog extensions.
 
 ## Status Labels
 
 | Label | Meaning |
 |---|---|
 | Automated | Covered by a runnable RustFS script or server test. |
-| Manual/live harness | RustFS can generate pinned client package inputs, commands, expected outputs, and CI opt-in gates for a live endpoint, but the live run is not enabled by default in CI. |
-| Generated harness | RustFS can generate client configuration or probe input, but live execution is not automated in CI. |
+| Manual/live harness | RustFS generates pinned client inputs, commands, expected outputs, and CI opt-in gates for a live endpoint; the live run is not enabled by default in CI. |
+| Generated harness | RustFS generates client configuration or probe input; live execution is not automated. |
 | Supported | Implemented server-side and covered by focused RustFS tests. |
-| Preview / controlled | Implemented behind explicit operator action or a run-once endpoint. No automatic background claim is made. |
-| Documented, not automated | Configuration or behavior is documented, but the live client run is not automated. |
-| Reference only | Kept as a compatibility reference. RustFS does not claim live interoperability yet. |
-| Not claimed | Out of scope for the current S3 Tables implementation. |
+| Preview / controlled | Implemented behind explicit operator action or a run-once endpoint; no automatic background claim. |
+| Documented, not automated | Behavior is documented; the live client run is not automated. |
+| Reference only | Compatibility reference; live interoperability is not claimed. |
+| Not claimed | Out of scope for the current implementation. |
 
 ## Endpoint And Profile Matrix
 
 | Surface | Status | Notes |
 |---|---|---|
-| `/iceberg/v1` | Supported | Canonical RustFS Iceberg REST Catalog prefix. Default REST signing name is `s3`. |
-| `/_iceberg/v1` | Supported compatibility alias | MinIO AIStor-style alias. The smoke profile defaults to REST signing name `s3tables`. |
-| S3 object data plane | Supported | Data, metadata, manifest, and delete files remain ordinary S3 objects, with table-aware policy checks for table warehouse paths. |
-| Table bucket enablement | Supported | A regular RustFS bucket can be enabled for table catalog use and then addressed as the REST catalog warehouse. |
-| Catalog-vended table credentials | Automated when enabled | Disabled by default. When enabled, LoadTable vends credentials only when `X-Iceberg-Access-Delegation` contains the exact `vended-credentials` token; the dedicated credentials endpoint uses the same issuer path. |
-| AWS S3 Tables endpoint shape | Profile generator | Generates the AWS catalog URI and S3 Tables warehouse ARN shape for migration docs. Full AWS S3 Tables API parity is not claimed. |
-| MinIO AIStor Tables profile | Profile generator plus RustFS alias smoke | RustFS exposes the alias shape, but does not claim all AIStor private extensions. |
-| Cloudflare R2 Data Catalog profile | Profile generator | Generates the catalog URI and warehouse-name shape for migration docs. Live RustFS interoperability is not claimed. |
-| Alibaba OSS Tables profile | Profile generator | Generates provider endpoint, `acs:osstables` warehouse ARN, `osstables` signing-name, and `https://oss-{region}.aliyuncs.com` S3FileIO endpoint shapes for migration docs. Live RustFS interoperability is not claimed. |
+| `/iceberg/v1` | Supported | Canonical REST Catalog prefix; default REST signing name `s3`. |
+| `/_iceberg/v1` | Supported compatibility alias | MinIO AIStor-style alias; smoke profile defaults to signing name `s3tables`. |
+| S3 object data plane | Supported | Data, metadata, manifest, and delete files are ordinary S3 objects with table-aware policy checks on warehouse paths. |
+| Table bucket enablement | Supported | A regular bucket is enabled for catalog use and addressed as the REST catalog warehouse. |
+| Catalog-vended table credentials | Automated when enabled | Disabled by default. LoadTable vends credentials only when `X-Iceberg-Access-Delegation` contains the exact `vended-credentials` token; the dedicated credentials endpoint uses the same issuer path. |
+| AWS S3 Tables endpoint shape | Profile generator | Generates the AWS catalog URI and warehouse ARN shape for migration docs. API parity not claimed. |
+| MinIO AIStor Tables profile | Profile generator plus alias smoke | Alias shape only; AIStor private extensions not claimed. |
+| Cloudflare R2 Data Catalog profile | Profile generator | Catalog URI and warehouse-name shape only; live interop not claimed. |
+| Alibaba OSS Tables profile | Profile generator | Endpoint, `acs:osstables` warehouse ARN, `osstables` signing name, and `https://oss-{region}.aliyuncs.com` S3FileIO endpoint shapes only; live interop not claimed. |
 
 ## Client And Engine Matrix
 
-| Client or engine | Status | Current RustFS claim |
+| Client or engine | Status | Claim |
 |---|---|---|
-| PyIceberg | Automated | Creates namespace and table, appends rows, reloads, scans, probes metadata-location, refs, views, maintenance, diagnostics, and optional catalog-vended table credentials with an exact-prefix data-plane scope check. |
-| Spark Iceberg REST catalog | Manual/live harness | RustFS can generate pinned Spark/Iceberg package inputs, REST catalog properties, SQL, run commands, expected `row_count=2`, and a CI opt-in gate for namespace creation, table creation, append, refresh, count, and cleanup. Live Spark execution and commit-conflict probing are still manual validation items unless explicitly enabled in the runner. |
-| Trino Iceberg REST catalog | Manual/live harness | RustFS can generate catalog properties and a read-only `SELECT COUNT(*)` command for a table created by PyIceberg or Spark. Write compatibility is not claimed. |
-| DuckDB Iceberg 1.5.5 | Automated | `duckdb_smoke.py` verifies the metadata-location read path and generic REST Catalog single-table create, insert, update, delete, merge, schema evolution, snapshots, concurrent writers, normal drop, PyIceberg cross-read, `/iceberg` with `s3` signing, and `/_iceberg` with `s3tables` signing. Staged create, purge-on-drop, and format v3 are verified as fail-closed boundaries. DuckDB's endpoint-disabled two-table mode is exercised without claiming cross-table atomicity. AWS `ENDPOINT_TYPE S3_TABLES` and catalog-vended credential integration are not claimed. |
-| StarRocks Iceberg REST catalog | Documented, not automated | External catalog read-path reference only. Write compatibility is not claimed. |
-| Databend | Manual/live harness | RustFS can generate an S3 stage read probe for table data files. RustFS does not claim Databend Iceberg REST Catalog integration yet. |
-| Snowflake Open Catalog / Iceberg integrations | Generated harness | RustFS can generate an operator-adapted external volume/catalog SQL template. Live RustFS interoperability is not claimed. |
+| PyIceberg | Automated | Namespace and table create, append, reload, scan, metadata-location, refs, views, maintenance, diagnostics, optional vended credentials with an exact-prefix data-plane scope check. |
+| Spark Iceberg REST catalog | Manual/live harness | Pinned package inputs, catalog properties, SQL, expected `row_count=2`, and a CI opt-in gate for create/append/refresh/count/cleanup. Live execution and commit-conflict probing remain manual unless enabled in the runner. |
+| Trino Iceberg REST catalog | Manual/live harness | Catalog properties and a read-only `SELECT COUNT(*)` against a PyIceberg- or Spark-created table. Write compatibility not claimed. |
+| DuckDB Iceberg 1.5.5 | Automated | `duckdb_smoke.py` covers metadata-location read, single-table create/insert/update/delete/merge, schema evolution, snapshots, concurrent writers, drop, PyIceberg cross-read, and both signing profiles. Staged create, purge-on-drop, and format v3 are verified fail-closed. Two-table mode runs without claiming cross-table atomicity. AWS `ENDPOINT_TYPE S3_TABLES` and vended-credential integration not claimed. |
+| StarRocks Iceberg REST catalog | Documented, not automated | External catalog read-path reference only. |
+| Databend | Manual/live harness | S3 stage read probe for table data files only; Iceberg REST integration not claimed. |
+| Snowflake Open Catalog / Iceberg integrations | Generated harness | Operator-adapted external volume/catalog SQL template only; live interop not claimed. |
 
 ## Live Evidence And Operations Matrix
 
-| Area | Status | Current RustFS claim |
+| Area | Status | Claim |
 |---|---|---|
-| Live conformance evidence | Automated for PyIceberg and DuckDB | `engine_compatibility.py --print-live-evidence-schema` defines the required evidence schema and claim promotion boundaries. `pyiceberg_smoke.py --live-evidence-output` and `duckdb_smoke.py --live-evidence-output` write validated client evidence records after successful live smoke runs. |
-| Production operations guide | Generated harness | `engine_compatibility.py --print-operations-guide` records command, evidence, pass criteria, and fail-closed signals for live conformance, durable backing cutover, maintenance, recovery, permissions, credential vending, and unsupported-claim governance. |
-| Vendor compatibility gap audit | Generated harness | `engine_compatibility.py --print-vendor-audit` records provider source URLs, catalog path and warehouse shapes, signing/auth models, error/permission/maintenance validation categories, and not-claimed boundaries for AWS S3 Tables, MinIO AIStor Tables, Cloudflare R2 Data Catalog, and Alibaba OSS Tables. |
-| Client claim promotion | Automated for scoped clients | PyIceberg and DuckDB claims remain bounded by their repeatable smoke entrypoints and recorded versions. Spark can be promoted only with recorded manual/live evidence; Trino remains read-only; Snowflake and vendor profiles remain reference-only without repeatable live evidence. |
+| Live conformance evidence | Automated for PyIceberg and DuckDB | `engine_compatibility.py --print-live-evidence-schema` defines the evidence schema and promotion boundaries; both smoke scripts write validated evidence records via `--live-evidence-output`. |
+| Production operations guide | Generated harness | `engine_compatibility.py --print-operations-guide` records commands, evidence, pass criteria, and fail-closed signals for conformance, cutover, maintenance, recovery, permissions, credential vending, and claim governance. |
+| Vendor compatibility gap audit | Generated harness | `engine_compatibility.py --print-vendor-audit` records provider URLs, path and warehouse shapes, auth models, validation categories, and not-claimed boundaries for the four vendor profiles. |
+| Client claim promotion | Automated for scoped clients | PyIceberg and DuckDB claims are bounded by their smoke entrypoints and recorded versions; Spark needs recorded live evidence; Trino stays read-only; Snowflake and vendor profiles stay reference-only. |
 
 ## Catalog API Matrix
 
 | Area | Status | Covered behavior |
 |---|---|---|
-| Catalog config | Supported | `GET /v1/config` advertises RustFS catalog defaults and only the supported OpenAPI REST paths in `endpoints`. RustFS administration, maintenance, migration, diagnostics, refs, and metadata-location extensions remain available but are not presented as standard Iceberg REST endpoints. |
-| Table bucket discovery | Supported | `PUT` and `GET /v1/buckets/{warehouse}` enable and inspect table bucket state. |
-| Namespaces | Supported | Create, list, load, existence check, and drop namespace routes are registered on both catalog prefixes. List responses support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Namespace identifiers are limited to 512 ASCII characters so persisted paths and stateless continuation tokens remain bounded. |
-| Tables | Supported | Create, register, list, load, existence check, rename, commit, metadata-location get/update, and drop table routes are registered on both catalog prefixes. Object-backed rename uses a bucket-scoped persistent fence, recoverable intent, and conditional publication of the destination, source tombstone, and warehouse index; the source identifier is reusable only through an ETag-conditional tombstone replacement. Table and view listings support Iceberg REST `pageSize`/`pageToken` pagination with context-bound tokens and bounded catalog-store reads. Commit identifiers must match the URL resource; unknown requirements, updates, and snapshot operations fail as bad requests; staged create, register overwrite, purge-on-drop, and v3-only encryption-key updates return an explicit unsupported-operation response. Standard statistics, partition statistics, and schema/spec cleanup updates are accepted. |
-| Commit CAS | Supported | Single-table commits validate base metadata, expected version token, referenced object existence, warehouse scope, and Iceberg commit requirements before advancing the current metadata pointer. Externally supplied metadata transitions preserve monotonic column, partition, and sequence assignment watermarks and immutable definitions for retained schemas, partition specs, sort orders, and snapshots. Standard commits preserve the normal commit-token file name and use an immutable-table-scoped fallback when rename followed by source-name reuse would otherwise collide at the same generation and commit ID. The catalog does not advertise `idempotency-key-lifetime`; clients must treat standard mutation-wide `Idempotency-Key` semantics as unsupported. |
-| Commit recovery | Supported | Commit log, idempotency lookup, diagnostics, and recovery routes expose staged/finalization gaps and repair safe idempotency gaps without moving the table pointer. |
-| Snapshot refs | Supported | Refs can be listed, created or replaced, and deleted through catalog commits. `main` is protected and refs with explicit retention require forced delete. |
-| Iceberg views | Supported | Basic create, list, load, replace, existence check, and drop routes persist view metadata with view-scoped authorization. Replace identifiers must match the URL resource, `schema-id: -1` resolves to the last added schema, one commit timestamp is used consistently, and only Iceberg view format version 1 is accepted. |
-| LoadTable and table credentials endpoint | Supported | LoadTable keeps the client-provided mode unless the request negotiates `vended-credentials`. Successful vending returns one temporary session for both the table warehouse prefix and the exact current metadata location. Missing credential permission falls back to metadata-only LoadTable with an explicit reason; issuer failures remain errors. Negotiated and dedicated credential responses set `Cache-Control: no-store, private`, `Pragma: no-cache`, and `Expires: 0`. |
-| Catalog diagnostics and export | Supported | Exposes recovery state, consistency state, backing manifest, recoverable commit-log WAL state, strong backing migration target, single-active-writer policy, and scale validation matrix. |
-| Catalog import and rollback | Supported | Import/register and online rollback use catalog validation and commit paths rather than direct pointer mutation. Online rollback accepts only a forward-safe metadata target that preserves assignment watermarks and retained definitions. Restoring an older target that lowers those watermarks is an offline disaster-recovery operation and requires every writer to be stopped. |
-| External catalog bridge | Supported operator path | Operator-supplied metadata pointer sync/import is supported for external catalog identity boundaries. Online vendor SDK polling and policy mirroring are not claimed. |
-| Multi-table transactions | Not claimed | RustFS currently claims single-table commit atomicity only. |
+| Catalog config | Supported | `GET /v1/config` advertises defaults and only the supported OpenAPI REST paths in `endpoints`; RustFS extensions (administration, maintenance, migration, diagnostics, refs, metadata-location) are not presented as standard endpoints. |
+| Table bucket discovery | Supported | `PUT` / `GET /v1/buckets/{warehouse}` enable and inspect table bucket state. |
+| Namespaces | Supported | Create, list, load, exists, drop on both prefixes; `pageSize`/`pageToken` pagination with context-bound tokens; identifiers limited to 512 ASCII characters. |
+| Tables | Supported | Create, register, list, load, exists, rename, commit, metadata-location get/update, drop on both prefixes. Rename uses a bucket-scoped persistent fence, recoverable intent, and conditional publication; the source name is reusable only via an ETag-conditional tombstone replacement. Commit identifiers must match the URL; unknown requirements/updates fail as bad requests; staged create, register overwrite, purge-on-drop, and v3-only encryption-key updates return an explicit unsupported-operation response. |
+| Commit CAS | Supported | Single-table commits validate base metadata, version token, referenced object existence, warehouse scope, and Iceberg requirements before advancing the pointer; external metadata transitions preserve monotonic assignment watermarks and immutable retained definitions. `idempotency-key-lifetime` is not advertised; mutation-wide `Idempotency-Key` semantics are unsupported. |
+| Commit recovery | Supported | Commit log, idempotency lookup, diagnostics, and recovery routes expose and repair finalization gaps without moving the pointer. |
+| Snapshot refs | Supported | List, create/replace, delete via commits; `main` is protected; refs with explicit retention need forced delete. |
+| Iceberg views | Supported | Create, list, load, replace, exists, drop with view-scoped authorization; only view format version 1. |
+| LoadTable and table credentials endpoint | Supported | Vending only on negotiated `vended-credentials`; one temporary session scoped to the warehouse prefix and current metadata location; missing credential permission falls back to metadata-only with an explicit reason; responses carry `Cache-Control: no-store, private`. |
+| Catalog diagnostics and export | Supported | Recovery state, consistency, backing manifest, WAL state, migration target, single-active-writer policy, scale validation matrix. |
+| Catalog import and rollback | Supported | Import/register and online rollback go through validation and commit paths. Online rollback accepts only forward-safe targets; restoring an older target that lowers watermarks is an offline disaster-recovery operation with all writers stopped. |
+| External catalog bridge | Supported operator path | Operator-supplied metadata pointer sync/import. Vendor SDK polling and policy mirroring not claimed. |
+| Multi-table transactions | Not claimed | Single-table commit atomicity only. |
 
 ## Data Plane And Credential Matrix
 
 | Area | Status | Covered behavior |
 |---|---|---|
-| Table-aware S3 policy bridge | Supported | Ordinary S3 actions against table warehouse paths are checked through the table data-plane bridge so table policy cannot be bypassed by direct object access. |
-| Reserved catalog protection | Supported | Catalog-reserved internal prefixes are protected from ordinary object mutation. |
-| Static S3 credentials | Automated | The default PyIceberg smoke path uses configured S3 credentials for REST signing and object data-plane access. |
-| Catalog-vended credentials | Automated when enabled | `rustfs-vended-credentials` verifies the returned table prefix, then checks `PutObject`, `HeadObject`, `GetObject`, and `DeleteObject` inside the prefix and denies access outside the prefix. |
-| Credential lifetime | Supported | Vended credential TTL is server-side and clamped to a short-lived range. |
-| No-long-term-data-credential bootstrap | Not claimed | The current credential-vending flow still uses the configured principal for catalog setup before table-scoped credentials are requested. |
+| Table-aware S3 policy bridge | Supported | Ordinary S3 actions on warehouse paths are checked through the table bridge; table policy cannot be bypassed by direct object access. |
+| Reserved catalog protection | Supported | Catalog-reserved prefixes are protected from ordinary object mutation. |
+| Static S3 credentials | Automated | Default PyIceberg smoke path. |
+| Catalog-vended credentials | Automated when enabled | `rustfs-vended-credentials` verifies the returned prefix, then checks Put/Head/Get/DeleteObject inside it and denies access outside it. |
+| Credential lifetime | Supported | Server-side TTL clamped to a short-lived range. |
+| No-long-term-data-credential bootstrap | Not claimed | Catalog setup still uses the configured principal before table-scoped credentials are requested. |
 
 ## Maintenance Matrix
 
-| Capability | Status | Current RustFS claim |
+| Capability | Status | Claim |
 |---|---|---|
-| Metadata retention dry-run | Supported | Reports retained metadata and deletion candidates without moving the table pointer. |
-| Metadata cleanup delete | Supported | Deletes only candidates that pass the safety window and current-pointer checks. |
-| Ordinary bucket lifecycle expiry | Disabled for table buckets | Table bucket objects are excluded from ordinary lifecycle expiration, including already queued expiry work. Snapshot expiration and orphan cleanup remain catalog maintenance operations so referenced Iceberg files cannot be deleted outside publication fencing. |
-| Snapshot expiration planning | Supported | Produces expiration plans with retained and candidate snapshots. |
-| Snapshot expiration commit | Preview / controlled | Can manually commit safe snapshot expiration through the catalog. Stale plans fail closed. |
-| Manifest/data/delete reachability cleanup | Supported | Reads manifest-list and manifest Avro references, reports reachable objects, and deletes only unreferenced table objects that pass the safety window. |
-| Maintenance scheduler run endpoint | Preview / controlled | Lets an external scheduler durably queue one maintenance job per table, reuse an active queued job, and recover expired queued leases before requeuing. |
-| Maintenance worker run endpoint | Preview / controlled | Supports queued-job claim, run-once execution, current-job backpressure, retry deferral, lease expiry recovery, and heartbeat updates. |
-| Maintenance scheduler guardrails | Preview / controlled | Exposes disabled, paused, ready, queued-job handoff, active-job backpressure, retry deferral, quarantine boundary, recommended actions, and recent maintenance job audit timeline state for external schedulers and operators. |
-| Maintenance audit events | Preview / controlled | Job reports and scheduler job summaries include structured audit events for planning, worker transitions, heartbeats, lease expiry recovery, and mutating quarantine operations. |
-| Maintenance quarantine operations | Preview / controlled | Lets operators inspect, release, retry, or abandon the current quarantined maintenance job without moving the table pointer. |
-| Compaction planning | Preview / controlled | Plans partition-local and sort-order-local binpack candidates for Parquet files and does not mix data files from different partition directories or sort orders in one rewrite group. |
-| Delete-file or row-level compaction planning | Preview / controlled | Manifests with position or equality delete files produce machine-readable row-level planning and force the compaction report into manual review before any rewrite can run. |
-| Compaction commit | Preview / controlled | Can commit a safe partition-local Parquet rewrite through the catalog while preserving Iceberg data file sort order IDs in the rewritten manifest. |
-| Built-in periodic scheduler | Not claimed | Operators can trigger scheduler and worker ticks, but continuous in-process scheduling is not claimed. |
-| Delete-file or row-level compaction execution | Not claimed | RustFS does not rewrite delete files or execute row-level compaction; those cases remain manual-review maintenance items. |
+| Metadata retention dry-run | Supported | Reports retained metadata and deletion candidates without moving the pointer. |
+| Metadata cleanup delete | Supported | Deletes only candidates passing the safety window and current-pointer checks. |
+| Ordinary bucket lifecycle expiry | Disabled for table buckets | Table bucket objects are excluded from lifecycle expiration, including queued work; snapshot expiration and orphan cleanup stay inside catalog maintenance. |
+| Snapshot expiration planning | Supported | Plans with retained and candidate snapshots. |
+| Snapshot expiration commit | Preview / controlled | Manual commit through the catalog; stale plans fail closed. |
+| Manifest/data/delete reachability cleanup | Supported | Reads manifest-list and manifest Avro references; deletes only unreferenced objects passing the safety window. |
+| Maintenance scheduler run endpoint | Preview / controlled | Durably queues one job per table, reuses an active queued job, recovers expired leases. |
+| Maintenance worker run endpoint | Preview / controlled | Claim, run-once, backpressure, retry deferral, lease expiry recovery, heartbeats. |
+| Maintenance scheduler guardrails | Preview / controlled | Disabled/paused/ready state, handoff, backpressure, quarantine boundary, recommended actions, recent job audit timeline. |
+| Maintenance audit events | Preview / controlled | Structured events for planning, worker transitions, heartbeats, lease recovery, quarantine mutations. |
+| Maintenance quarantine operations | Preview / controlled | Inspect, release, retry, or abandon the quarantined job without moving the pointer. |
+| Compaction planning | Preview / controlled | Partition-local and sort-order-local binpack candidates for Parquet; never mixes partitions or sort orders in one group. |
+| Delete-file or row-level compaction planning | Preview / controlled | Position or equality delete files force machine-readable planning into manual review. |
+| Compaction commit | Preview / controlled | Commits a safe partition-local Parquet rewrite preserving sort order IDs. |
+| Built-in periodic scheduler | Not claimed | Ticks are operator-triggered; no continuous in-process scheduling. |
+| Delete-file or row-level compaction execution | Not claimed | Manual-review maintenance item. |
 
 ## Recovery And Strong Backing Matrix
 
-| Area | Status | Current RustFS claim |
+| Area | Status | Claim |
 |---|---|---|
-| Single-table CAS | Supported | The table pointer advances only through expected-token and expected-metadata-location validation. |
-| Idempotent retry | Supported | Repeated commit IDs can return the already finalized result or surface recoverable finalization gaps. |
-| Commit publication fencing | Supported with rolling-upgrade gate | Existing deployments retain exact object guards so older writers cannot mutate referenced files during publication. Set `RUSTFS_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED=true` only after every serving node supports table and table-bucket publication fences. In scalable mode, active table warehouse prefixes must not overlap, ordinary lifecycle expiry remains disabled for table buckets, and first enablement, first publication, drop, and warehouse relocation are serialized by the table-bucket fence. |
-| Post-CAS finalization recovery | Supported | Diagnostics and recovery can repair stale or missing idempotency indexes without changing the current table pointer. |
-| Catalog export | Supported | Exposes table state, commit recovery state, and backing migration information for operator inspection. |
-| Strong backing state transfer | Supported | Object-backed table bucket, namespace, table, view, commit-log, and idempotency state can be materialized into the durable strong snapshot. The transfer is deterministic, ETag-CAS protected, idempotent after an interrupted finalization, validates candidate state through the restart decoder before publication, preserves resource-backed implicit namespaces, and fails closed when an inactive explicit namespace conflicts with active descendants or resources. Snapshot hydration requires a stable non-empty ETag, caps the encoded snapshot at 64 MiB, shares state and reload serialization across requests in one server context, and rejects disappearance or format-version rollback after observation. Configured durable-strong mode rejects a missing snapshot on its first catalog access after startup; only object-backed migration may initialize an empty target. |
-| Durable backing migration preflight | Supported | `GET /iceberg/v1/{warehouse}/catalog/migration` and the `/_iceberg/v1` alias inspect object-backed catalog inventory, recovery blockers, warehouse prefix index readiness, active table/view identifier collisions, persistent write-fence state, target snapshot agreement, and whether every table bucket is ready for cutover. |
-| Durable backing migration execution | Preview / controlled | `POST /iceberg/v1/{warehouse}/catalog/migration` fences table-bucket registry changes, acquires a persistent per-bucket write fence, records whether a global strong snapshot existed before publication, drains in-flight catalog mutations, materializes the target snapshot, and reports `ready_to_enable_durable_strong`. Retries and `DELETE` may restore a known-absent initial target after an ambiguous first write, but fail closed if a previously existing or materialized global snapshot disappears. `DELETE` releases the bucket fence only while its target state has not advanced, and releases the registry fence after the last bucket is cancelled. Both mutations require `admin:MigrateTableCatalog`. |
-| Strong snapshot rolling compatibility | Supported | Durable strong control-plane reads snapshot versions 1 and 2, writes version 1 by default, and writes version 2 only after both the requested and fleet-confirmed gates are enabled. A running process rejects any lower-format snapshot after observing a higher format. Once version 2 is fleet-confirmed, table data-plane resolution fails closed until the persisted snapshot is version 2; a missing table-bucket entry also fails closed instead of bypassing table-aware authorization. |
-| Disaster recovery rehearsal | Manual/live harness | `failure_coverage.py --print-disaster-recovery-rehearsal` generates an operator runbook covering catalog export, diagnostics, safe recovery repair, rollback/import, durable backing migration dry-run, post-recovery loadTable, and table data-plane policy probes. |
-| Scale and fault rehearsal | Manual/live harness | `failure_coverage.py --print-scale-fault-rehearsal` generates an opt-in runbook for concurrent writer stress, maintenance scheduler lease recovery, durable backing cutover preflight, recovery/rollback/import under load, and post-run evidence capture. |
-| Durable strong snapshot backing cutover | Preview / controlled | Operators can select the ETag-CAS snapshot backing with `RUSTFS_TABLE_CATALOG_BACKING=durable-strong` only after every table bucket reports `SNAPSHOT_MATERIALIZED` and `ready_to_enable_durable_strong: true`. This mode does not claim a separate external KV/WAL service, and object-only advanced operations fail closed. Version 1 backing manifests retain the legacy `STRONG_KV_WAL` and `CUT_OVER_LINEARIZABLE_READS` wire labels for client compatibility; those labels do not expand the implementation claim. |
+| Single-table CAS | Supported | Pointer advances only through expected-token and expected-metadata-location validation. |
+| Idempotent retry | Supported | Repeated commit IDs return the finalized result or surface recoverable finalization gaps. |
+| Commit publication fencing | Supported with rolling-upgrade gate | Exact object guards protect referenced files during publication. Set `RUSTFS_TABLE_CATALOG_PUBLICATION_FENCE_FLEET_CONFIRMED=true` only after every serving node supports table and table-bucket fences. In scalable mode, warehouse prefixes must not overlap and enablement, first publication, drop, and relocation are serialized by the table-bucket fence. |
+| Post-CAS finalization recovery | Supported | Repairs stale or missing idempotency indexes without changing the pointer. |
+| Catalog export | Supported | Table state, commit recovery state, and backing migration information. |
+| Strong backing state transfer | Supported | Object-backed catalog state is materialized into the durable strong snapshot deterministically, ETag-CAS protected, idempotent after interrupted finalization, and validated through the restart decoder before publication; conflicts between inactive explicit namespaces and active descendants fail closed. Hydration requires a stable non-empty ETag, caps the snapshot at 64 MiB, and rejects disappearance or format-version rollback after observation. Configured durable-strong mode rejects a missing snapshot on first access; only object-backed migration may initialize an empty target. |
+| Durable backing migration preflight | Supported | `GET /iceberg/v1/{warehouse}/catalog/migration` (and the alias) reports inventory, recovery blockers, prefix-index readiness, identifier collisions, fence state, target agreement, and per-bucket cutover readiness. |
+| Durable backing migration execution | Preview / controlled | `POST /iceberg/v1/{warehouse}/catalog/migration` fences registry changes, acquires a persistent per-bucket write fence, drains in-flight mutations, materializes the snapshot, and reports `ready_to_enable_durable_strong`. `DELETE` cancels only while the target has not advanced. Both mutations require `admin:MigrateTableCatalog`. Procedure: [s3-tables-cutover-runbook.md](../operations/s3-tables-cutover-runbook.md). |
+| Strong snapshot rolling compatibility | Supported | Reads snapshot versions 1 and 2; writes version 1 by default and version 2 only after both `RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2` and `RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED` are set. A process rejects a lower format after observing a higher one; once v2 is fleet-confirmed, data-plane resolution fails closed until the persisted snapshot is v2. |
+| Disaster recovery rehearsal | Manual/live harness | `failure_coverage.py --print-disaster-recovery-rehearsal`. |
+| Scale and fault rehearsal | Manual/live harness | `failure_coverage.py --print-scale-fault-rehearsal`. |
+| Durable strong snapshot backing cutover | Preview / controlled | `RUSTFS_TABLE_CATALOG_BACKING=durable-strong` only after every table bucket reports `SNAPSHOT_MATERIALIZED` and `ready_to_enable_durable_strong: true`. No separate external KV/WAL service is claimed; the legacy `STRONG_KV_WAL` and `CUT_OVER_LINEARIZABLE_READS` labels in v1 manifests are wire-compatibility labels, not claims. |
 | Single active writer region | Supported policy | Diagnostics publish single-active-writer semantics and read-only replica limits. |
-| Active-active multi-region writes | Not claimed | A table must not accept independent concurrent writers in multiple active regions. |
-
-## Durable Backing Cutover Runbook
-
-Use the migration dry-run before changing the table catalog backing for a
-warehouse:
-
-1. Take an object-backed catalog backup and record the current metadata pointer
-   and version token for representative tables.
-2. Run `GET /iceberg/v1/{warehouse}/catalog/migration` with a principal that has
-   `GetTableCatalogAction` on each table bucket. Treat every `blockers` entry as
-   fail-closed; repair commit recovery state and backfill the warehouse prefix
-   index before continuing.
-3. Before the migration `POST`, drain every catalog writer that predates the
-   durable-backing migration fence and restart it on a fence-aware release. An
-   older writer does not recognize the persisted fence and can otherwise
-   mutate the object-backed source after the snapshot inventory is captured.
-   Keep all catalog writers on the fence-aware release until cutover completes.
-4. Inventory object-only advanced operations, including maintenance workers,
-   catalog recovery, export, diagnostics, and external catalog bridge writes.
-   Quiesce mutating operations before cutover and confirm that each required
-   operation is supported by durable-strong mode; unsupported operations fail
-   closed after cutover rather than continuing against object-backed state.
-5. Run `POST /iceberg/v1/{warehouse}/catalog/migration` with
-   `admin:MigrateTableCatalog`. This acquires the exclusive migration fence to
-   drain in-flight fence-aware mutations, persists the source fence while
-   exclusivity is held, and then copies the catalog state.
-6. Repeat the preflight and materialization for every table bucket. Do not set
-   `RUSTFS_TABLE_CATALOG_BACKING=durable-strong` until the preflight reports
-   `SNAPSHOT_MATERIALIZED`, no blockers, and
-   `ready_to_enable_durable_strong: true`.
-7. Restart with durable strong backing enabled, then verify catalog config,
-   table and view loads, commit idempotency, and table data-plane policy
-   resolution before admitting writers.
-8. Before restarting into durable-strong mode, `DELETE` on the migration
-   endpoint can remove a migration-created target bucket snapshot and release
-   the source fence. After the durable-strong state advances, cancellation
-   fails closed; recovery requires an operator-selected restore or reverse
-   migration instead of restarting against the stale object-backed pointer.
-9. Preserve the object-backed catalog backup until durable strong backing has
-   passed the operator's retention window.
-10. Keep strong snapshot writes on version 1 during a rolling binary upgrade.
-   After every catalog writer can read version 2, set both
-   `RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2=true` and
-   `RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED=true`, then restart
-   the catalog writers. Perform a controlled catalog write or migration
-   materialization and confirm that the persisted snapshot is version 2 before
-   serving table data-plane traffic. Setting only one gate does not change the
-   write format.
-11. After any version 2 snapshot is persisted, do not roll catalog writers back
-   to a binary that only reads version 1. Current binaries preserve version 2
-   even when the gates are later disabled. A running process rejects restored
-   version 1 content after observing version 2, but cannot distinguish an older
-   snapshot with the same format version from a deliberate restore. The format
-   high-water mark is process-local: restoring any older snapshot and restarting
-   every writer is a privileged disaster-recovery rollback that cannot be
-   inferred from the restored object alone. Recovery must restore a compatible
-   binary and a snapshot selected through the operator recovery procedure.
-12. Migration preflight rejects an active table/view identifier collision before
-    it writes a migration fence. A pre-existing version 1 strong snapshot with
-    such a collision is loaded in cleanup-only quarantine. Ambiguous reads fail
-    closed; each cleanup mutation must reduce the collision set, and unrelated
-    writes remain blocked until all collisions are removed. Drain catalog
-    writers that predate cleanup quarantine before starting this repair, and
-    complete cleanup before the first version 2 write. Restoring any version 1
-    snapshot after a writer has observed version 2 fails closed instead of
-    replacing the in-process catalog state.
+| Active-active multi-region writes | Not claimed | A table must not accept independent concurrent writers in multiple regions. |
 
 ## Production Failure Coverage
 
-Positive client smoke proves a client can use a table. Production failure probes
-prove RustFS does not silently advance table state when a failure happens.
+Failure probes prove RustFS does not silently advance table state on failure. Tracked cases: stale commit token or base metadata returns a conflict without advancing the pointer; missing metadata, manifest, data, or delete objects fail closed before commit or maintenance; concurrent writers produce one CAS winner and retryable conflicts; catalog and S3 permission denials prevent data-plane bypass; stale maintenance plans fail closed before deletion or commit; post-CAS finalization gaps are visible and safely recoverable; external catalog sync conflicts leave pointer, token, and generation unchanged; backing migration stays blocked until WAL and recovery replay are clean.
 
-The tracked failure cases are:
-
-- stale commit token or stale base metadata returns a conflict without advancing
-  the table pointer
-- missing metadata, manifest, data, or delete objects fail closed before commit
-  or maintenance can advance state
-- concurrent writers produce a single winning CAS and retryable conflicts for
-  stale writers
-- table catalog and ordinary S3 permission denials prevent data-plane bypass
-- stale maintenance plans fail closed before object deletion or catalog commit
-- post-CAS finalization gaps are visible through diagnostics and safe recovery
-- external catalog sync conflicts leave pointer, token, and generation unchanged
-- backing migration remains blocked until WAL and recovery replay are clean
-
-Do not promote a failure case from a required live probe or load test to an
-automated claim until the exact RustFS build, client version, and expected
-response shape are recorded.
+Do not promote a failure case from live probe or load test to an automated claim until the exact RustFS build, client version, and expected response shape are recorded.
 
 ## Unsupported Or Not Claimed
 
-RustFS does not currently claim:
+Full AWS S3 Tables control-plane parity; full MinIO AIStor private extensions; full Cloudflare R2 Data Catalog or Alibaba OSS Tables interoperability; built-in periodic maintenance scheduling; active-active multi-region writes; multi-table transactions; no-long-term-data-credential bootstrap; online vendor SDK polling; external catalog policy mirroring; delete-file rewrite or row-level compaction execution; built-in SQL execution; Delta Lake or Hudi; end-to-end SQL row-level DML validation through Spark, Trino, or another engine.
 
-- full AWS S3 Tables control-plane API parity
-- full MinIO AIStor Tables private extension parity
-- full Cloudflare R2 Data Catalog interoperability
-- full Alibaba OSS Tables interoperability
-- built-in periodic maintenance scheduling; external schedulers can queue maintenance jobs and workers can claim them, but RustFS does not claim a continuous in-process scheduler
-- active-active multi-region table writes
-- multi-table transactions
-- no-long-term-data-credential table bootstrap
-- online external catalog vendor SDK polling
-- external catalog policy mirroring
-- delete-file rewrite or row-level compaction execution
-- built-in SQL query execution
-- Delta Lake or Hudi table format support
-- end-to-end SQL row-level DML validation through Spark, Trino, or another SQL engine
+## Verification
 
-## Verification Commands
-
-Use these commands when updating this matrix, release notes, or client
-compatibility claims:
-
-```bash
-python3 scripts/table-catalog/test_pyiceberg_smoke.py
-python3 scripts/table-catalog/test_engine_compatibility.py
-python3 scripts/table-catalog/test_duckdb_smoke.py
-python3 scripts/table-catalog/test_failure_coverage.py
-python3 scripts/table-catalog/pyiceberg_smoke.py --print-client-matrix
-python3 scripts/table-catalog/pyiceberg_smoke.py --print-engine-compatibility
-python3 scripts/table-catalog/pyiceberg_smoke.py --print-production-failure-coverage
-python3 scripts/table-catalog/pyiceberg_smoke.py --print-vendor-profiles
-python3 scripts/table-catalog/pyiceberg_smoke.py --print-production-readiness
-python3 scripts/table-catalog/engine_compatibility.py --print-vendor-audit
-python3 scripts/table-catalog/engine_compatibility.py --print-spark-config
-python3 scripts/table-catalog/engine_compatibility.py --print-duckdb-rest-sql
-python3 scripts/table-catalog/engine_compatibility.py \
-  --profile aws-s3tables \
-  --region us-east-1 \
-  --account-id 123456789012 \
-  --table-bucket analytics \
-  --print-spark-config
-python3 scripts/table-catalog/engine_compatibility.py \
-  --metadata-location s3://rustfs-s3table-smoke/tables/table-id/metadata/v1.metadata.json \
-  --print-live-conformance \
-  --cleanup
-python3 scripts/table-catalog/engine_compatibility.py --print-live-evidence-schema
-python3 scripts/table-catalog/pyiceberg_smoke.py \
-  --endpoint http://127.0.0.1:9000 \
-  --bucket rustfs-s3table-smoke \
-  --replace \
-  --cleanup \
-  --rustfs-build rustfs-v1.0.0-beta.8 \
-  --git-sha "$(git rev-parse HEAD)" \
-  --catalog-backing durable-strong \
-  --live-evidence-output /tmp/rustfs-pyiceberg-live-evidence.json
-python3 scripts/table-catalog/engine_compatibility.py \
-  --warehouse rustfs-s3table-smoke \
-  --namespace smoke \
-  --table events \
-  --print-operations-guide
-python3 scripts/table-catalog/failure_coverage.py \
-  --warehouse rustfs-s3table-smoke \
-  --namespace smoke \
-  --table events \
-  --print-failure-probes
-python3 scripts/table-catalog/failure_coverage.py \
-  --warehouse rustfs-s3table-smoke \
-  --namespace smoke \
-  --table events \
-  --table-warehouse-location s3://rustfs-s3table-smoke/tables/table-id \
-  --print-disaster-recovery-rehearsal
-python3 scripts/table-catalog/failure_coverage.py \
-  --warehouse rustfs-s3table-smoke \
-  --namespace smoke \
-  --table events \
-  --table-warehouse-location s3://rustfs-s3table-smoke/tables/table-id \
-  --writer-count 8 \
-  --maintenance-worker-count 2 \
-  --iteration-count 50 \
-  --print-scale-fault-rehearsal
-```
+Commands for updating this matrix, release notes, or client claims are maintained in [scripts/table-catalog/README.md](../../scripts/table-catalog/README.md); the unit tests are `scripts/table-catalog/test_*.py`.
 
 ## Release Claim Guidance
 
-Use conservative release wording that matches the matrix.
+Acceptable: "RustFS includes a core Iceberg REST Catalog-based S3 Tables implementation with PyIceberg and DuckDB smoke coverage, table-aware S3 data-plane policy checks, controlled maintenance, catalog recovery diagnostics, manual conformance input for Spark, Trino, Databend, and Snowflake, production-failure probe harnesses, disaster-recovery and scale/fault rehearsal probes, and a machine-readable operations evidence guide."
 
-Acceptable wording:
+Do not claim: "RustFS is fully compatible with AWS S3 Tables."
 
-> RustFS includes a core Iceberg REST Catalog-based S3 Tables implementation
-> with PyIceberg and DuckDB smoke coverage, table-aware S3 data-plane policy checks,
-> controlled maintenance, catalog recovery diagnostics, manual conformance
-> input for Spark, Trino, Databend, and Snowflake, production-failure
-> probe harnesses, disaster-recovery and scale/fault rehearsal probes, and a
-> machine-readable production operations evidence guide.
-
-Do not claim:
-
-> RustFS is fully compatible with AWS S3 Tables.
-
-Any stronger vendor or engine claim needs a repeatable live validation harness,
-the exact client versions used, and the expected response shapes recorded in the
-table-catalog inventories.
+Any stronger vendor or engine claim needs a repeatable live harness, the exact client versions, and the expected response shapes recorded in the table-catalog inventories.
 
 ## Related
 
 - [Table catalog conformance scripts](../../scripts/table-catalog/README.md)
+- [Durable backing cutover runbook](../operations/s3-tables-cutover-runbook.md)
 - [Admin route action snapshot](admin-route-action-snapshot.md)
 - [Runtime capability contracts](runtime-capability-contracts.md)

docs/architecture/scanner-heal-admission.md

@@ -1,29 +0,0 @@
-# Scanner/Heal admission Phase 0 baseline
-
-This document records the current entry points and safety boundaries for backlog #1939. It is an inventory and test contract, not a lease design. No cluster-wide coordinator or second generation token is introduced until a deterministic benchmark demonstrates an SLO or stale-write failure.
-
-## Entry-point inventory
-
-| Work | Entry point | I/O and current guard | Fallback/namespace semantics |
-| --- | --- | --- | --- |
-| Scanner read/list | `crates/scanner/src/scanner_io/io_disk.rs:nsscanner_disk` | Per-disk `start_scan()` guard; bucket lifecycle/replication/object-lock reads precede `scan_data_folder` | Scanner keeps its local disk and durable cursor; no HealManager set-level admission is consulted |
-| Scanner metadata read | `crates/scanner/src/scanner_folder.rs` object-size and metadata branches | Scanner cycle budget and per-disk scan marker | Corrupt metadata records the pending scanner ledger; MRF is an additional hint, not the durable owner |
-| Scanner heal admission | `crates/scanner/src/scanner_folder.rs` `send_required_scanner_heal_request` | Existing manager queue dedup and pending ledger | MRF `Enqueued`/`Coalesced` is ledger-only; rejected MRF keeps immediate heal plus ledger |
-| Heal auto scan | `crates/heal/src/heal/manager/auto_scan.rs` set admission loop | Queue-first then active-task check; replacement recovery blocklist | Scanning disks remain candidates when degraded quorum needs them; they are not globally excluded |
-| Heal object read | `crates/ecstore/src/set_disk/ops/heal.rs` `heal_object` | Namespace write lock unless `no_lock`; reads file info before commit | Namespace lock is object-scoped and does not claim scanner cycle ownership |
-| Disk selection | `crates/ecstore/src/set_disk/ops/locking.rs` candidate selection | Healing disks are ordered after new disks; scanning disks may remain candidates | Degraded/quorum fallback is preserved |
-| Data movement | Existing storage-owned movement/publication generation (#1905/#1942) | This issue does not add a second coordinator | Future admission must validate the storage generation at the final commit |
-
-## Baseline contract
-
-The deterministic baseline in `scanner_heal_admission_baseline.rs` encodes the investigation matrix only: ScannerRead+HealRead may overlap, HealWrite conflicts with scanner reads, DataMovementWrite conflicts with all work, and independent set identities remain concurrent. It does not claim that production currently enforces the matrix.
-
-The production facts that must be measured before Phase 1 are scanner p99, heal p99, cursor/checkpoint delay, queue and pending-ledger depth, and starvation by set. The benchmark matrix must include restart recovery, degraded quorum/scanning-disk fallback, urgent replacement heal, and at least two independent sets.
-
-The executable fixture uses a fixed eight-sample restart/degraded sequence so the baseline is reproducible without wall-clock noise: two sets each receive ScannerRead, HealRead, HealWrite and a follow-up ScannerRead. Its expected synthetic p99 is 420 microseconds, maximum modeled backlog is 2, two HealWrite samples are deferred, and the independent second set still services three reads. These are fixture values, not production SLO claims; production benchmark output must replace them with measured p99, backlog and per-set wait distributions.
-
-The inventory test reads the current source files and asserts the named guards/fallback branches are still present (`start_scan`, pending-ledger admission, Heal queue/active checks, namespace `get_write_lock`, and scanning-disk re-append). A source rename or guard removal therefore fails the baseline instead of silently leaving stale documentation.
-
-Commit-time generation-fencing, lease-expiry, and lock-order tests are intentionally deferred until a Phase-0 fixture demonstrates a stale write or an SLO violation; arithmetic-only placeholders would stay green if production paths regressed.
-
-If a future fixture demonstrates stale destructive writes, the fix must extend the storage-owned generation/admission primitive and validate the token at the final metadata/format/delete commit. Cancellation or a local lease alone is not a fence.

docs/architecture/storage-control-data-plane.md

@@ -1,7 +1,7 @@
 # Storage, Control Plane, And Background Controllers
 
-This document defines migration boundaries for the storage hot path and adjacent
-control-plane responsibilities.
+**Use this when:** adding a storage API surface, a cluster read model, or a background-service status/reconcile surface, and you need to know which layer owns it and what must not drift.
+**Source of truth:** `crates/storage-api` (trait contracts), `crates/ecstore/src/api/mod.rs` (facade groups, `api::cluster`), `crates/ecstore/src/cluster/` (control plane), [background-controller-contract.md](background-controller-contract.md) (controller vocabulary).
 
 ## Storage API Contracts
 
@@ -42,9 +42,11 @@ It maps existing endpoint pools into the shared storage-api topology contract an
 an ECStore-owned static membership snapshot. It must not expose local disk paths,
 start health checks, mutate endpoint ownership, or change placement/readiness.
 The same facade also owns static pool-state, local-node storage, and peer-health
-status projections. Peer health remains explicitly unknown until a later slice
-wires real health signals; this document does not authorize background probes or
-RPC-based health checks.
+status projections. `peer_health_snapshot` in
+`crates/ecstore/src/cluster/control_plane.rs` projects the internode health
+tracker's per-node reachability (`PEER_HEALTH_REACHABLE` /
+`PEER_HEALTH_UNREACHABLE`, or not-reported); the facade itself starts no probes
+and issues no RPC-based health checks.
 Readiness impact for storage, lock quorum, peer health, probes, admin routes,
 RPC, and the S3 data plane is recorded in
 [`readiness-matrix.md`](readiness-matrix.md).

docs/architecture/unified-object-generation.md

@@ -1,384 +1,112 @@
-# Unified Per-Object Generation Authority
+# Object Transaction UUID And Generation-Fencing Contract
 
-Establishes a **single per-object generation authority** that spans object
-commit, GET snapshots, garbage collection, and quota accounting, and pins the
-transport, encoding, proto-evolution, and mixed-version contracts that every
-consumer must obey.
+**Use this when:** adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against "the current version of an object", or when adding a field that rides internode RPC or `xl.meta`.
+**Source of truth:** `assign_object_transaction_epoch` in `crates/ecstore/src/set_disk/ops/object.rs` and `crates/ecstore/src/set_disk/ops/multipart.rs`; `FileInfo::set_object_transaction_epoch` in `crates/filemeta/src/fileinfo.rs`; `commit_rename_data_dir` and `RenameConvergence` in `crates/ecstore/src/set_disk/core/io_primitives.rs`; `PreparedPoolReadFallbackBarrier` in `crates/ecstore/src/store/rebalance.rs`; `crates/protos/src/node.proto`; env constants in `crates/config/src/constants/object.rs` and `crates/config/src/constants/internode.rs`.
 
-This is a **design and contract document**. It changes no storage code. It is
-the shared prerequisite for five implementation sub-issues under the
-[#1307](https://github.com/rustfs/backlog/issues/1307) adversarial-review
-program:
-[#1312](https://github.com/rustfs/backlog/issues/1312) (commit fencing),
-[#1313](https://github.com/rustfs/backlog/issues/1313) (read lease),
-[#1314](https://github.com/rustfs/backlog/issues/1314) (prepared pool read),
-[#1318](https://github.com/rustfs/backlog/issues/1318) (quota reservation), and
-[#1323](https://github.com/rustfs/backlog/issues/1323) (old-dir GC).
+Design tracking lives in `rustfs/backlog#1326`. This document holds only the invariants.
 
-Tracks [rustfs/backlog#1326](https://github.com/rustfs/backlog/issues/1326).
+## Authority
 
-## Why one authority
+The target contract requires **one per-object commit identity** consumed by commit fencing, read leases, cleanup, prepared reads, and quota settlement. No consumer may mint a second value and call it the same generation.
 
-The #1307 adversarial-review verdict (issuecomment-4992565957) found that the
-five sub-issues each reach for their own generation / fencing / lease token to
-solve the same underlying problem — **commit mutual-exclusion plus snapshot
-lifetime**. Left independent, they diverge and punch through one another:
+What exists today is an **object transaction UUID**, not the target authority:
 
-- #1323 old-dir GC can reclaim a directory still referenced by a #1313 lease if
-  the two disagree on what "current generation" means.
-- #1312 fence epoch and #1318 quota reservation token, if derived from two
-  different monotonic sources, cannot be compared — a late commit fenced on one
-  plane can still settle quota on the other.
+| Property | Current implementation |
+|---|---|
+| Minting | `assign_object_transaction_epoch` mints a random non-nil UUID for PUT and CompleteMultipartUpload when the object-transaction gate is active. |
+| Persistence | Written through `FileInfo::set_object_transaction_epoch` into the version's internal metadata map under the dual-key contract (`x-rustfs-internal-*` / `x-minio-internal-*`). |
+| Fence check | The coordinator reads the current UUID (or `Absent`) and revalidates exact equality immediately before `rename_data`. |
+| Cleanup | Old-data cleanup receipts carry the committed UUID; reconciliation deletes only when the receipt UUID still equals the current object UUID. |
 
-The fix is a single authority with one selected comparison rule, one persistence
-semantics, and one transport binding, that every consumer references rather than
-re-derives.
+This is an equality-CAS fence and cleanup identity. It is not a monotonic epoch, is not minted by the distributed lock grant, and is not compared atomically at each disk's `xl.meta` commit point. Documents and issues must call it the *object transaction UUID*, not proof that the generation authority exists.
 
-## Target authority and the current bounded token
+### Authority modes (one must be selected)
 
-The target contract still requires **one per-object commit identity** consumed
-by commit fencing, read leases, cleanup, prepared reads, and quota settlement.
-No consumer may mint a second value and call it the same generation.
-
-The concrete ordering semantics are not settled, however. The original #1326
-proposal requires a total-ordered, monotonic lock-grant epoch. Current main does
-not implement that proposal. PR #6077 instead implements an opaque transaction
-identity:
-
-- `assign_object_transaction_epoch` mints a random non-nil UUID for PUT and
-  CompleteMultipartUpload when the object-transaction gate is active.
-- The UUID is written through `FileInfo::set_object_transaction_epoch` into the
-  dual internal metadata map.
-- The coordinator reads the current UUID (or `Absent`) and revalidates exact
-  equality immediately before `rename_data`.
-- Old-data cleanup receipts carry the committed UUID and reconciliation deletes
-  only when the receipt UUID still equals the current object UUID.
-
-This is a useful **equality-CAS fence and cleanup identity**. It is not a
-monotonic epoch, is not minted by the distributed lock grant, and is not
-compared atomically at each disk's `xl.meta` commit point. Until the decision
-below is made, documents and issue checklists must call it the *object
-transaction UUID* rather than use it as proof that the target generation
-authority exists.
-
-### Ordering decision required
-
-Before #1313, #1314, or a unified quota binding can consume the authority, one
-of these contracts must be selected and tested:
-
-1. **Total-ordered fencing epoch.** A lock grant returns a durable per-object
-   `(term, counter)` (or another specified total-order type). Every disk rejects
-   a lower epoch at the atomic metadata commit point. The value never regresses
-   across lock-plane restart, failover, or minority recovery.
-2. **Opaque commit-generation identity.** Consumers compare only exact identity;
-   no `<` / `>` semantics are permitted. The authoritative commit must perform
-   an atomic expected-generation CAS, and all lease, cleanup, prepared-read, and
-   quota contracts must be rewritten in terms of “references this exact
-   generation,” not “lower/newer generation.”
-
-The current UUID implementation proves neither a durable total order nor a
-per-disk atomic expected-generation CAS, so it does not by itself decide between
-these options.
-
-### Persistence semantics if total order is selected
-
-A total-ordered epoch must be **monotonic across lock-plane restart and
-failover**. The distributed lock entry remains in-memory; deriving a counter
-from that entry alone would reset it after restart. The chosen source therefore
-must be either quorum-persisted before grant or derived from a durable term whose
-full `(term, counter)` comparison cannot regress. This requirement does not
-apply to an opaque UUID as an ordering rule; the opaque alternative instead
-requires atomic expected-identity comparison and durable crash recovery.
-
-## Consumer binding contracts
-
-### Current implementation snapshot (2026-08-31, main@9ee7b1221)
-
-This table separates code that exists on current main from the target contract.
-Closing an implementation issue does not imply that its token is already the
-unified authority.
-
-| Surface | Current main | Gap against this contract |
+| Mode | Contract | Persistence requirement |
 |---|---|---|
-| PUT / CompleteMultipartUpload (#1312, PR #6077) | Owned commit tasks retain the relevant guards; an opt-in gate persists a random object transaction UUID and performs a quorum metadata equality recheck before rename | no lock-grant monotonic source; no per-disk atomic epoch/CAS comparison; the live proof is the reused remote-version-state fleet proof, not a dedicated generation capability |
-| Old-data cleanup (#1323, PR #6077) | JSON receipt carries transaction UUID, old dir, and committed dir; reconciliation is gated and requires UUID equality | no generation-bound read lease is consulted, so this is crash cleanup fencing rather than the full #1313/#1323 lease lifetime contract |
-| Read lease (#1313) | short-term streaming/multipart path holds the namespace read lock through EOF/drop; deterministic part-boundary coverage is tracked by PR #6887 | no cross-node generation-bound lease registry, TTL reclamation, or crash recovery |
-| Prepared pool read (#1314) | PR #6889 tracks a pool-local prepared identity and fails closed/refetches when pool state changes | not merged on this snapshot; pool-local identity is not a cross-pool generation authority; black-box mixed-version/rebalance coverage remains open |
-| Quota reservation (#1318) | durable per-bucket ledger plus independent snapshot-lease mutation-fence tokens; issue closed after PR #6058 | reservation and settle are not bound to the object transaction UUID; the independent fence must be reconciled with the selected authority or explicitly proven to be a separate, non-generation arbitration domain |
-| Internode integrity (#1327, #1541, #1542) | v2/v3 HMAC binds audience, exact method, timestamp, nonce, canonical body digest, and receiver boot epoch; body-bound RPC policy has exact-set coverage | signature/body/replay strict switches remain default-off rollout gates; generation enforcement cannot treat an unrelated fleet-version proof as proof that these strict contracts converged |
+| Total-ordered fencing epoch | A lock grant returns a durable per-object `(term, counter)`; every disk rejects a lower epoch at the atomic metadata commit point; the value never regresses across lock-plane restart, failover, or minority recovery. | Quorum-persisted before grant, or derived from a durable term whose full comparison cannot regress. The in-memory distributed lock entry alone is insufficient. |
+| Opaque commit-generation identity | Consumers compare exact identity only; no `<` / `>` semantics. The authoritative commit performs an atomic expected-generation CAS; lease, cleanup, prepared-read, and quota contracts are phrased as "references this exact generation". | Atomic expected-identity comparison plus durable crash recovery. |
 
-| Consumer | How it binds generation | Key invariant |
-|---|---|---|
-| #1312 commit fence | selected generation is checked at `rename`, rollback restore/delete, and cleanup mutation points using the chosen ordered or exact-CAS rule | a stale writer is rejected on **all** disks; an already-ACK'd write is never rolled back |
-| #1313 read lease | lease binds the exact generation observed at read time; GC runs only after every lease referencing that generation is released | lease is visible across nodes; a crashed reader's lease is reclaimed by TTL |
-| #1323 old-dir GC | cleanup job carries the committed generation; before deleting `old_dir` it confirms that no lease for the generation owning that directory remains | `old_dir != committed_dir`; a still-referenced directory is never deleted |
-| #1314 prepared pool read | the `PreparedPoolRead` bundle carries the generation resolved during pool lookup; the chosen pool's reader setup reuses it only after a match | generation mismatch forces a fallback to full metadata fanout |
-| #1318 quota reservation | reservation / settle record binds the exact object generation (and an ordered epoch too, if that option is selected) | a late commit cannot settle quota for a different committed generation |
+The current UUID proves neither a durable total order nor a per-disk atomic CAS, so it does not decide between the modes.
 
-### Fence coverage is three disk-write points, not one (#1312 B2)
+## Consumer Binding
 
-Checking the generation only before the `rename` fanout is insufficient. The
-authoritative commit sequence is `tmp sync → data-dir rename → xl.meta commit →
-directory sync` in `crates/ecstore/src/disk/local.rs`, and there are two further
-detachable disk-write points in
-`crates/ecstore/src/set_disk/core/io_primitives.rs`:
+| Consumer | Binds generation how | Key invariant | Current state |
+|---|---|---|---|
+| Commit fence (PUT / CompleteMultipartUpload) | Checked at `rename`, rollback restore/delete, and cleanup mutation points using the selected rule | A stale writer is rejected on **all** disks; an already-ACK'd write is never rolled back | Opt-in UUID equality recheck before rename; no per-disk atomic comparison |
+| Read lease | Lease binds the exact generation observed at read time; GC runs only after every lease on that generation is released | Lease visible across nodes; crashed reader's lease reclaimed by TTL | Streaming/multipart GET holds the namespace read lock through EOF/drop (part-boundary coverage: `#6887`); no cross-node generation-bound registry |
+| Old-dir GC | Cleanup job carries the committed generation and confirms no lease owns `old_dir` before deleting | `old_dir != committed_dir`; a still-referenced directory is never deleted | UUID receipt equality (`#6077`); no lease consultation |
+| Prepared pool read | The prepared bundle carries the generation resolved during pool lookup; the chosen pool reuses it only after a match | Mismatch forces fallback to full metadata fanout | `PreparedPoolReadFallbackBarrier` (`#6889`) is a pool-local identity that fails closed / refetches on pool state change; it is not a cross-pool authority |
+| Quota reservation | Reserve / settle record binds the exact object generation (and the ordered epoch too, if selected) | A late commit cannot settle quota for a different committed generation | Durable per-bucket ledger with independent snapshot-lease fence tokens (`#6058`); not bound to the transaction UUID |
 
-- **Rollback restore/delete** — on quorum failure each disk can restore backup
-  metadata or delete the failed version. A stale writer's rollback must compare
-  the expected generation, otherwise it can overwrite or delete the winner's
-  already-committed metadata.
-- **`commit_rename_data_dir`** — a cancel-then-detach disk-write point; the
-  coordinator's "reap all child tasks" must explicitly include it so a cancelled
-  writer cannot bypass fence/lease and keep deleting directories.
+## Fence Coverage: Three Disk-Write Points
 
-If generation is validated only after data-dir rename, a fenced writer
-may already have renamed its data-dir into the object path, leaving a staged
-orphan. Either move the fence ahead of the data-dir rename, or declare that
-orphan an acceptable residue accounted for by GC metrics — the white-box
-acceptance "no background disk write after release" must be rewritten
-accordingly.
+Checking generation only before the `rename` fanout is insufficient. The commit sequence is `tmp sync → data-dir rename → xl.meta commit → directory sync` in `crates/ecstore/src/disk/local.rs`, and `crates/ecstore/src/set_disk/core/io_primitives.rs` has two further detachable disk-write points:
 
-Current PR #6077 performs a quorum metadata equality recheck before rename and
-reaps owned commit work. That closes important cancellation windows, but it is
-not evidence that every disk mutation above performs the selected generation
-comparison atomically. The writer inventory and per-point CAS/ordering proof
-remain acceptance work for #1326 even though #1312 is closed.
+1. **Rollback restore/delete.** On quorum failure each disk can restore backup metadata or delete the failed version. A stale writer's rollback must compare the expected generation, or it can overwrite or delete the winner's committed metadata. Panic, cancel, and timeout outcomes must be reaped into coordinator convergence rather than skip rollback through an early return.
+2. **`commit_rename_data_dir`.** A cancel-then-detach disk-write point; the coordinator's "reap all child tasks" must include it so a cancelled writer cannot bypass fence or lease and keep deleting directories.
 
-### Post-commit convergence is orthogonal to the fence (#1321)
+If generation is validated only after the data-dir rename, a fenced writer may already have renamed its data-dir into the object path, leaving a staged orphan. Either move the fence ahead of the data-dir rename, or declare that orphan an accepted residue accounted for by GC metrics.
 
-The same `SetDisks::rename_data` path already returns a post-commit
-convergence classification (`RenameConvergence`, rustfs/backlog#1321) that
-tells the caller whether the *committed* replicas need heal to converge —
-`AllSuccessIdentical` (no heal), `PartialCommit` (a replica failed/offline),
-`SignatureDivergent` (committed replicas' version signatures differ), or
-`Unknown` (no signature was produced, e.g. >10 versions — scanner-backstopped).
-This replaced an earlier `Option<Vec<u8>>` heuristic under which any
-version signature looked like "needs heal", so every healthy multipart
-completion self-enqueued.
+`RenameConvergence` (`AllSuccessIdentical` / `PartialCommit` / `SignatureDivergent` / `Unknown`) is a *post-commit* heal signal on the same `rename_data` path; the fence is a *commit* gate. They compose: the fence decides whether a convergence is produced, `RenameConvergence` classifies it. A fence-aware convergence variant would be an additive enum change.
 
-Convergence is a *post-commit* signal (the write landed; do the replicas need
-reconciliation), whereas the #1312 fence is a *commit* gate (a stale epoch is
-rejected before the write lands, surfaced through the existing `Result::Err`
-channel). They compose on the one `rename_data` path rather than competing:
-the fence decides whether a convergence is produced at all, and
-`RenameConvergence` classifies it once produced. A future fence-aware
-convergence variant, if ever needed, is an additive change to that enum and
-does not disturb the epoch comparison at the disk-write points above.
+## Transport And Security
 
-## Transport and security contract
+Generation and derived tokens (lease, reservation) cross node boundaries in internode RPC bodies; every such flow must be signature-bound.
 
-Generation and all derived tokens (lease, reservation) cross node boundaries in
-internode RPC bodies. Every such flow must be signature-bound.
+| Rule | Detail |
+|---|---|
+| HMAC scope | Target audience, exact service/method, timestamp, nonce, canonical body digest, receiver replay (boot) epoch. The receiver consumes the nonce in a bounded replay cache; a transmitted-but-unconsumed nonce is not replay protection. |
+| Current substrate | RPC v2/v3 in `crates/ecstore/src/cluster/rpc/http_auth.rs` binds all of the above. Body-bound policy covers mutating disk RPCs including `RenameData`, whose versioned canonical body includes every `RenameDataRequest` field, so the `FileInfo` metadata map carrying the UUID is authenticated. |
+| Strict switches | `RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`, `RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, `RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` (`crates/config/src/constants/internode.rs`) are default-off rollout gates governed by [compat-cleanup-register.md](compat-cleanup-register.md). A generation capability may claim strong transport binding only after the relevant strict modes have converged fleet-wide. |
+| Acceptance tests per consumer | Method substitution, canonical body tamper, nonce replay, receiver restart, stripped-strict-metadata negatives. |
 
-### RPC signature binding (#1312 B3, #1313, #1318)
+## Encoding Rules
 
-**Requirement.** The canonical body carrying a generation or derived token must
-be folded into the internode HMAC. The authenticated scope binds the target
-audience, exact service/method, timestamp, nonce, canonical body digest, and
-receiver replay epoch. The receiver must consume the nonce in a bounded replay
-cache; transmitting a nonce without receiver-side consumption is not replay
-protection.
+| Rule | Reason |
+|---|---|
+| **Do not bump `XL_META_VERSION` or `XL_HEADER_VERSION`** (`crates/filemeta/src/filemeta.rs`). | `decode_xl_headers` in `crates/filemeta/src/filemeta/codec.rs` rejects newer values outright; a bump makes every new `xl.meta` unreadable by rolling-upgrade old nodes and by MinIO. See [minio-file-format-compat.md](minio-file-format-compat.md). |
+| **Do not add generation as a `FileInfo` struct field.** | Internode RPC serializes `FileInfo` with two msgpack encoders: positional-array encoding for the `read_version` family (a new positional field breaks mixed-version decode) and `encode_msgpack_named` (named-map) for `rename_data` in `rustfs/src/storage/rpc/node_service/disk.rs`. A field would have to be correct under both plus the JSON compatibility twin. Use the metadata map, which rides every encoder unchanged. |
+| **Metadata-map dual key.** | The UUID lives under `x-rustfs-internal-*` / `x-minio-internal-*`; missing, malformed, nil, or conflicting dual values fail closed when fencing is active. |
+| **No sidecar unless atomic.** | An epoch sidecar outside `xl.meta` is admissible only if it commits at the same atomic/CAS point as `xl.meta` with a specified crash-recovery protocol. None is implemented. |
+| **Regression guard.** | The real-MinIO `xl.meta` interop fixtures in `crates/filemeta/src/filemeta.rs` must keep passing: objects written by a new node stay readable by old RustFS nodes and by MinIO in both upgrade directions. |
 
-**Current substrate (verified on main).** The original legacy-only description
-is obsolete:
+### Wire-encoding window (JSON and msgpack)
 
-- RPC v2 binds target audience, exact method, POST, timestamp, nonce, and body
-  digest.
-- Body-bound policy covers mutating disk RPCs including `RenameData`; its
-  versioned canonical body includes every `RenameDataRequest` field, so the
-  `FileInfo` metadata map carrying the transaction UUID is authenticated.
-- PR #5425 extended canonical-body enforcement to implemented non-disk mutating
-  unary RPCs and added an exact policy/handler coverage partition.
-- PR #5455 added the receiver boot epoch and rotating replay scope so signatures
-  captured before a receiver restart are rejected after capability convergence.
-
-The rollout switches
-`RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`,
-`RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, and
-`RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` remain default-off for rolling
-compatibility. The compatibility register and fallback/overflow metrics govern
-their fleet convergence. Therefore a generation capability may claim strong
-transport binding only when the relevant strict modes have converged; the
-object-transaction gate's current remote-version-state fleet proof is not, by
-itself, proof of RPC signature/body/replay strictness.
-
-Acceptance for each generation consumer includes method substitution, canonical
-body tamper, nonce replay, receiver restart, and stripped-strict-metadata
-negative tests. Generation rollout must also record which strict-mode evidence
-authorized enforcement.
-
-### Encoding contract (#1312 B1)
-
-The on-disk persistence of generation must not perturb the file format:
-
-- **Do not bump `XL_META_VERSION` / `XL_HEADER_VERSION`.**
-  `crates/filemeta/src/filemeta/codec.rs` rejects `meta_ver > 3` and
-  `header_ver > 3` outright (`decode_xl_headers`), and both constants are `3`
-  (`crates/filemeta/src/filemeta.rs:53-54`). Bumping either makes every new
-  `xl.meta` unreadable by rolling-upgrade old RustFS nodes and by MinIO — a
-  total read failure, not a graceful downgrade.
-- **Do not add generation as a `FileInfo` struct field.** The internode RPC layer serializes `FileInfo` with two different msgpack encoders depending on the call site: `encode_msgpack` uses rmp_serde's default **array** (positional) encoding for the `read_version` family, where a new positional field breaks decode across mixed-version nodes; `encode_msgpack_named` uses `.with_struct_map()` (named-map) encoding for `rename_data` (`crates/ecstore/src/cluster/rpc/remote_disk.rs`), which is more tolerant but still requires `#[serde(default)]` and MinIO-side agreement. Because a `FileInfo` field would have to be correct under *both* encoders and under the JSON compatibility twin (see "Wire-encoding migration" below), do not add one — use the metadata map, which rides through every encoder unchanged.
-- **Where it lives today.** The object transaction UUID uses the version's
-  internal metadata map under the dual-key contract
-  (`x-rustfs-internal-*` / `x-minio-internal-*`) via
-  `set_object_transaction_epoch`. Missing, malformed, nil, or conflicting dual
-  values fail closed when fencing is active.
-- **Sidecars are not an equivalent alternative.** A future sidecar is admissible
-  only if it commits atomically with `xl.meta` and has a specified crash-recovery
-  protocol. No such protocol is implemented, so a sidecar cannot be selected by
-  an implementation issue merely because this document mentions one.
-- **Regression guard.** Preserve the #4377 real-MinIO `xl.meta` interop
-  regression (the fixture family around `crates/filemeta/src/filemeta.rs`):
-  objects written by a new node must still be readable by old RustFS nodes and
-  by MinIO, in both upgrade and downgrade directions.
-
-### Wire-encoding migration (JSON → msgpack) interaction
-
-The internode RPC layer retains a JSON/msgpack rolling-compatibility window, and
-generation-bearing fields must respect it.
-
-- **Dual-field transport.** Each dual-encoded RPC field exists twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes _bin` field (e.g. `file_info` #4 alongside `file_info_bin` #7 on `RenameDataRequest`). Senders emit both; receivers `decode_msgpack_or_json` prefer the `_bin` form and fall back to the JSON string only when `_bin` is empty (`crates/ecstore/src/cluster/rpc/remote_disk.rs`).
-- **Capability flags, default off.** `rustfs_protos::internode_rpc_msgpack_only()` only drops the redundant JSON copy when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` are deliberately enabled after the JSON-fallback metric reads zero fleet-wide and the convergence runbook is followed. Generation follows the same default-off, fleet-confirmed, metric-reads-zero rollout discipline, but a msgpack proof is not itself a generation capability proof.
-- **Generation must ride both encodings during the window.** If epoch lives in the version's internal metadata map, that map is carried inside `FileInfo`, so it is present in both the msgpack `_bin` and JSON copies automatically — good. But any new *top-level* generation datum must be added to **both** the msgpack and JSON representations (and, for msgpack, be safe under both the array and named-map encoders). A field added to only one encoding is silently lost the moment a peer falls back to the other — exactly the failure the JSON-fallback metric exists to catch.
-- **Signature binds a canonical form.** `RenameDataRequest` now has a versioned,
-  injective canonical-body encoder that covers both compatibility fields and is
-  authenticated independently of whichever JSON/msgpack decoder branch a peer
-  consumes. A generation-capable strict request must reject missing or
-  mismatched canonical-body metadata; it must not silently downgrade to an
-  unauthenticated JSON twin.
+- Dual-encoded RPC fields exist twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes *_bin` field (e.g. `file_info` and `file_info_bin` on `RenameDataRequest`). Senders emit both; receivers (`decode_msgpack_or_json` in `crates/ecstore/src/cluster/rpc/remote_disk.rs`) prefer `_bin` and fall back to JSON only when `_bin` is empty.
+- `rustfs_protos::internode_rpc_msgpack_only()` drops the JSON copy only when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` are set after the JSON-fallback metric reads zero fleet-wide.
+- Generation inside the `FileInfo` metadata map is carried in both copies automatically. Any new *top-level* generation datum must be added to both encodings and be safe under both msgpack encoders; a field in only one encoding is silently lost when a peer falls back.
+- `RenameDataRequest` has a versioned, injective canonical-body encoder covering both compatibility fields; a strict generation-capable request must reject missing or mismatched canonical-body metadata rather than downgrade to the unauthenticated JSON twin.
 
 ### Proto evolution
 
-No top-level proto field is required by the current metadata-map UUID. If a
-future ordered epoch or explicit expected-generation is added to proto, it uses
-**proto3 `optional`** (explicit presence). A non-optional scalar is forbidden:
-an old coordinator talking to a new disk decodes absence as a plausible zero.
+No top-level proto field is required by the metadata-map UUID. If an ordered epoch or explicit expected-generation is ever added to proto, it uses **proto3 `optional`** (explicit presence). A non-optional scalar is forbidden: an old coordinator talking to a new disk decodes absence as a plausible zero.
 
-### Mixed-version gate — one direction
+## Mixed-Version Gate: One Direction
 
-When generation enforcement is not explicitly requested, or fleet confirmation
-is absent, behavior falls back to current semantics. Fail-closed is reserved for
-an explicit administrator-confirmed strict rollout.
+When generation enforcement is not explicitly requested, or fleet confirmation is absent, behavior falls back to current semantics. Fail-closed is reserved for an explicit administrator-confirmed strict rollout.
 
-Current object transaction fencing follows that direction:
+| Flag (`crates/config/src/constants/object.rs`) | Default | Effect |
+|---|---|---|
+| `RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` | false | With either flag absent, PUT/MPU neither persists nor consumes the transaction UUID. |
+| `RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` | false | With both enabled, failure to obtain or retain the live fleet proof rejects the commit before rename. |
 
-- `RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` and
-  `RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` both default false.
-- With either flag absent, PUT/MPU does not persist or consume the transaction
-  UUID.
-- With both flags enabled, failure to obtain or retain the live fleet proof
-  rejects the commit before rename.
+The fleet proof is currently borrowed from the remote-version-state writer rollout. It proves membership/process-epoch convergence for that feature only; it does not prove an epoch type, per-disk CAS support, or RPC strict-mode convergence, and must not be treated as the final generation handshake.
 
-This is an opt-in strict gate, not a negotiated generation capability. The
-proof is currently borrowed from the remote-version-state writer rollout. It
-proves current membership/process-epoch convergence for that feature, but does
-not prove an epoch type, per-disk generation CAS support, or RPC strict-mode
-convergence. Treating it as the final handshake is forbidden without an
-explicit proof mapping for those properties.
+### Capability negotiation (target)
 
-## Capability negotiation
+Generation enforcement requires one **live fleet proof** containing at least: the selected authority version and comparison mode; the current membership/topology fingerprint and process epochs; support for every required disk mutation point; RPC signature/body/replay strict convergence; and the on-disk encoding version (the metadata-map UUID is version 1). Membership change or an old-node rejoin revokes the proof; revocation before commit fails an explicitly strict request and never rewrites or lowers a persisted generation. The proof may extend the authenticated fleet-proof machinery in `notification_sys` or the runtime capability contract; this document requires one shared token, not a mechanism.
 
-Generation enforcement requires one **live fleet proof**, not independent
-boolean guesses in each consumer. The proof contract contains at least:
+## Open Decisions
 
-1. the selected authority version and comparison mode (ordered or exact-CAS),
-2. the current membership/topology fingerprint and process epochs,
-3. support for every required disk mutation point,
-4. RPC signature/body/replay strict convergence, and
-5. the on-disk encoding version (the current metadata-map UUID is version 1).
+Blockers for calling the contract implemented:
 
-The authoritative writer enables enforcement only while every target disk in
-the set is covered by a current proof. Membership change or an old node rejoin
-revokes that proof. Revocation before commit fails an explicitly strict request;
-when strict generation was never requested, the request remains on the legacy
-path. Revocation never rewrites or lowers an already-persisted generation.
-
-The existing fleet-proof machinery in `notification_sys` may be reused if its
-authenticated statements are extended to cover the properties above. The
-runtime capability contract may instead expose the proof. This document does
-not choose the storage mechanism; it requires one token whose acquisition and
-revalidation semantics are shared by all consumers.
-
-## Implementation order
-
-Some original prerequisites have landed, but not in the originally proposed
-form. Remaining work follows this order:
-
-1. **Resolve the authority mode in #1326.** Select total order or opaque
-   exact-CAS, specify its atomic commit point, and audit PR #6077 against it.
-   Do not retrofit ordering semantics onto the existing random UUID.
-2. **Define the generation fleet proof.** Map generation enablement to the RPC
-   signature/body/replay strict proofs delivered by #1327/#1541/#1542 and to
-   the selected per-disk comparison capability. Keep all strict defaults off
-   until fallback metrics converge.
-3. **Implement #1313 generation-bound read leases.** The lease registry,
-   cross-node visibility, TTL, and crash recovery must exist before old-dir GC
-   can claim the full snapshot-lifetime guarantee. #1325 supplies the required
-   multi-node failure tests.
-4. **Bind #1314 prepared reads.** A bundle binds the exact selected generation
-   within its source pool. Cross-pool ordering is forbidden until a common
-   authority is demonstrated. Validate rebalance and mixed-version fallback in
-   the #1325 multi-pool harness.
-5. **Reconcile #1318 quota fencing.** Either bind reserve/settle/reconcile to
-   the selected object generation or document and prove that its independent
-   snapshot-lease fence is a separate arbitration domain that cannot settle a
-   different generation.
-6. **Re-audit #1323 cleanup.** The existing UUID receipt remains valid crash
-   cleanup, but full closure against active readers requires the #1313 lease
-   check and the selected generation semantics.
-
-## Open design decisions (pin before contract closure)
-
-The following decisions remain blockers for calling the contract implemented:
-
-- **Authority mode.** Choose total order or opaque exact-CAS. If total order is
-  selected, define the type, per-object scope, persistence, overflow, and
-  never-regress restart/minority-recovery tests. If opaque identity is selected,
-  define the atomic expected-generation CAS and remove all ordered wording.
-- **Complete xl.meta-writer coverage.** Enumerate commit rename, rollback
-  restore/delete, cleanup, heal, transition, restore, replication, and data
-  movement. Each path must compare/carry the selected generation or be proved
-  incapable of replacing the authoritative object identity.
-- **Rollback is an expected-generation CAS (#1312 B2).** The quorum-failure
-  rollback in `rename_data` can restore backup metadata, not just remove a
-  writer-private temporary file. It must execute only when the stored generation
-  still matches the failed writer's expected generation. Panic, cancel, and
-  timeout outcomes must be reaped into coordinator convergence rather than skip
-  rollback through an early return.
-- **Sidecar is excluded unless proven atomic.** An epoch sidecar outside `xl.meta` is only admissible if it commits at the same atomic/CAS point as `xl.meta` with a defined recovery; otherwise it opens a crash gap and must be rejected in favor of the version-internal metadata map. The earlier "metadata map or sidecar" phrasing does not treat the two as equally safe.
-- **Generation capability proof.** Decide whether to extend the current
-  authenticated fleet proof or the runtime capability contract. It must prove
-  authority version, mutation coverage, topology/process epoch, and RPC strict
-  convergence in one revalidatable token.
-- **Read-lease and GC crash recovery.** Select the cross-node registry, TTL
-  reclamation, lease-holder crash behavior, and GC-executor recovery. The
-  current cleanup receipt equality check does not answer these questions.
-- **Quota reserve → commit → settle binding.** The durable ledger's idempotency
-  exists, but its independent mutation tokens must be related to the selected
-  object generation with a concrete late-settle rejection test.
-- **PreparedPoolRead is pool-local only.** A #1314 bundle's generation validates freshness only within the pool that produced it. It cannot order commits across different pools unless a cross-pool common authority exists; absent that, the multi-pool wait cannot be short-circuited.
-- **Hot-path cost is a blocking metric.** Measure any additional consensus
-  write, fsync, fleet-proof lookup, lease operation, or centralized serialization
-  under 4 KiB and high-concurrency hot-key/hot-bucket A/B.
-- **Test infrastructure.** #1325 still lacks the complete 4-node × 4-drive,
-  2-pool, directed network-fault, and large-object budget needed for restart,
-  mixed-version, and cross-node lease acceptance.
-
-## Acceptance for this contract
-
-- [x] Architecture document exists and is linked from the architecture index.
-- [x] Transport signature, encoding, proto presence, mixed-version direction,
-  and capability-proof requirements are defined once.
-- [x] Current implementations are separated from target guarantees; a closed
-  child issue is not treated as proof of unified generation binding.
-- [ ] Authority mode and atomic comparison semantics are selected and tested.
-- [ ] #1312 / #1313 / #1314 / #1318 / #1323 bodies reference this document and
-  use the selected authority terminology.
-- [ ] #1313 and #1314 bind the selected generation and pass #1325 multi-node /
-  multi-pool failure tests.
-- [ ] #1318 either binds reserve/settle to the selected generation or provides
-  an accepted proof that its separate fence cannot cross-settle generations.
-- [ ] #1323 reconciliation checks both committed generation and active
-  generation-bound leases.
-- [ ] Generation strict enablement is backed by one live proof that includes RPC
-  signature/body/replay strict convergence and per-disk comparison support.
+1. **Authority mode.** Total order or opaque exact-CAS. Do not retrofit ordering semantics onto the existing random UUID.
+2. **Complete `xl.meta`-writer coverage.** Enumerate commit rename, rollback restore/delete, cleanup, heal, transition, restore, replication, and data movement; each path compares/carries the selected generation or is proved incapable of replacing the authoritative identity.
+3. **Rollback as expected-generation CAS.** The quorum-failure rollback in `rename_data` restores backup metadata, not just a private temp file; it must run only when the stored generation still matches the failed writer's expectation.
+4. **Generation capability proof.** Extend the fleet proof or the runtime capability contract; one revalidatable token.
+5. **Read-lease and GC crash recovery.** Cross-node registry, TTL reclamation, lease-holder crash behavior, GC-executor recovery.
+6. **Quota reserve → commit → settle binding.** Relate the ledger's independent mutation tokens to the selected generation, with a concrete late-settle rejection test, or prove the fence is a separate arbitration domain that cannot cross-settle.
+7. **Prepared reads stay pool-local.** `PreparedPoolReadFallbackBarrier` validates freshness only within the pool that produced it; cross-pool ordering requires a common authority, and the multi-pool wait cannot be short-circuited without one.
+8. **Hot-path cost is a blocking metric.** Measure any added consensus write, fsync, fleet-proof lookup, lease operation, or centralized serialization under 4 KiB and hot-key/hot-bucket A/B.
+9. **Test infrastructure.** Multi-node, multi-pool, directed network-fault, and large-object budget for restart, mixed-version, and cross-node lease acceptance.

docs/architecture/workload-admission-contracts.md

@@ -1,130 +1,40 @@
 # Workload Admission Contracts
 
-This document records the `rustfs/backlog#660` PR-05 and PR-07 scheduler
-preservation and runtime workload-class contract slice.
+**Use this when:** adding a workload class or snapshot provider, consuming admission state from a background job, or deciding whether a job can "join" admission (it cannot; see Observation Surface Only).
+**Source of truth:** `WorkloadClass`, `AdmissionState`, `WorkloadAdmissionSnapshot`, `WorkloadAdmissionRegistrySnapshot`, `WorkloadAdmissionSnapshotProvider`, and `foreground_pressure` in `crates/concurrency/src/workload.rs`; the provider and consumer files named below.
 
-## Preservation Coverage
+## Contract Shapes
 
-The `rustfs-concurrency` tests pin the current reusable admission-facing
-behavior before later snapshot extraction:
+`rustfs-concurrency` owns the read-only shapes. `WorkloadClass` enumerates the admission categories (the variants are the source of truth); `AdmissionState`, `WorkloadAdmissionSnapshot`, and `WorkloadAdmissionRegistrySnapshot` are status shapes for runtime owners to fill. They do not replace the scheduler, request guards, scanner, heal, replication, or ECStore placement behavior. `GetObjectQueueSnapshot` permit semantics (saturated, over-available, zero-total) and worker-slot over-release clamping are pinned by `rustfs-concurrency` tests; scheduler buffer/priority behavior is pinned by `rustfs-io-core` and `rustfs/src/storage/concurrency/` tests.
 
-- Worker slot over-release remains clamped by the configured worker limit.
-- `GetObjectQueueSnapshot` preserves saturated, over-available, and zero-total
-  permit semantics.
+## Class To Provider Table
 
-The former reusable scheduler and backpressure-pipe facades (and their
-preservation tests) were removed as zero-caller dead code in backlog#1025;
-scheduler buffer/priority behavior is now pinned by `rustfs-io-core` and
-`rustfs/src/storage/concurrency` tests.
+| Class | Provider (`impl WorkloadAdmissionSnapshotProvider`) | `active` / `queued` / `limit` source | Reports `Unknown` when |
+|---|---|---|---|
+| `ForegroundRead` | `ConcurrencyManager` in `rustfs/src/storage/concurrency/manager.rs` (source of truth); re-exposed unchanged by the RustFS runtime provider | disk-read permits in use / `None` (the semaphore exposes no waiter count) / configured max concurrent disk reads | the storage registry has no entry |
+| `ForegroundWrite` | none | none | always: no write-specific admission owner exposes a read-only surface yet |
+| `Metadata` | `RustFsWorkloadAdmissionSnapshotProvider` in `rustfs/src/workload_admission.rs` | `Open` once the bucket metadata runtime handle exists; no counts | bucket metadata runtime not initialized |
+| `Scanner` | same | scanner active work-unit counter / none / none | the counter is zero (idle and uninitialized are indistinguishable) |
+| `Repair` | same | heal active tasks / heal queue length / `None` (limits live behind the async heal manager state) | heal manager not initialized |
+| `Replication` | same | active regular + large-object + MRF workers / site replication queue count / `None` (limits owned by the async pool and resize policy) | replication runtime not initialized, or queue stats currently locked |
 
-## Workload Class Contract
+## Observation Surface Only
 
-`WorkloadClass` defines the required future admission categories:
+This is an observation surface only. Permit acquisition, priority assignment, buffer sizing, storage media detection, request guards, queue capacity, heal admission and priority merge/drop policy, replication worker resize and MRF handling, scanner cycle scheduling, bucket metadata loading and locks, and object write paths are unchanged by any provider. There is no runtime admission API for a background job to join; a job that needs bounded contention must bound it itself (see [kms-bulk-rekey-contract.md](kms-bulk-rekey-contract.md)).
 
-- Foreground read.
-- Foreground write.
-- Metadata.
-- Scanner.
-- Repair.
-- Replication.
+Consumers that read the snapshot to self-throttle exist, and they do not change the owners' decisions:
 
-`AdmissionState`, `WorkloadAdmissionSnapshot`, and
-`WorkloadAdmissionRegistrySnapshot` define read-only status shapes for later
-runtime owners. They do not replace the current scheduler, request guard,
-scanner, heal, replication, or ECStore placement behavior.
+| Consumer | File | Behavior |
+|---|---|---|
+| Data-movement backpressure (decommission, rebalance) | `crates/ecstore/src/data_movement/backpressure.rs` (`wait_for_data_movement_admission`, `foreground_pressure`) | Delays the next data-movement step while `ForegroundRead` or `ForegroundWrite` usage exceeds the configured high-water percent. ECStore receives the provider through `set_workload_admission_snapshot_provider` (`crates/ecstore/src/lib.rs`), published from `rustfs/src/startup_background.rs`; with no provider the step is admitted immediately. |
+| Heal manager mainline throttle | `crates/heal/src/heal/manager.rs` (`new_with_workload_provider`) | When `mainline_throttle_enable` is set, defers heal work while `ForegroundRead` or `ForegroundWrite` utilization exceeds the configured high-water percents; with no provider or the throttle disabled, heal pacing is unchanged. |
 
 ## Boundary Rules
 
-- `rustfs-concurrency` owns this reusable contract surface.
-- The contract does not depend on `rustfs-ecstore` or RustFS binary runtime
-  state.
-- No scheduler decision logic, queue capacity, Tokio runtime default, scanner
-  admission, heal admission, replication admission, placement, membership, or
-  NUMA behavior changes are part of this slice.
+- `rustfs-concurrency` owns the contract surface and does not depend on `rustfs-ecstore` or RustFS binary runtime state.
+- Adding a class or provider changes no scheduler decision logic, queue capacity, Tokio runtime default, scanner/heal/replication admission, placement, membership, or NUMA behavior.
+- Providers report `Unknown` rather than blocking or guessing when their owner is uninitialized or its stats are not immediately observable.
 
-## Set-Local Snapshot Extraction
+## Provider Composition
 
-The RustFS storage `ConcurrencyManager` now implements
-`WorkloadAdmissionSnapshotProvider` for local foreground-read admission:
-
-- `ForegroundRead` reports local disk-read permit usage through
-  `GetObjectQueueSnapshot`.
-- `active` is the number of disk-read permits currently in use.
-- `limit` is the configured maximum concurrent disk reads.
-- `queued` remains `None` because the current semaphore does not expose waiter
-  counts.
-- Scanner, repair, replication, foreground write, and metadata entries remain
-  `Unknown` until their owning runtime components expose read-only status.
-
-This is an observation surface only. Permit acquisition, priority assignment,
-buffer sizing, storage media detection, request guards, and queue behavior are
-unchanged.
-
-## Heal Repair Snapshot Extraction
-
-The RustFS integration layer now exposes a read-only repair admission snapshot
-from the heal runtime counters:
-
-- `Repair` reports the current heal active task count.
-- `queued` reports the current heal queue length.
-- `limit` remains `None` because the configured heal queue and concurrency
-  limits live behind the async heal manager state.
-- Other workload classes remain `Unknown` in this provider until their owning
-  runtime components expose read-only status.
-
-This is an observation surface only. Heal request admission, queue capacity,
-priority merge/drop policy, task scheduling, retry handling, and repair
-behavior are unchanged.
-
-## Replication Snapshot Extraction
-
-The RustFS integration layer now exposes a read-only replication admission
-snapshot from the existing replication pool and queue statistics:
-
-- `Replication` reports active regular, large-object, and MRF worker counts.
-- `queued` reports the current site replication queue count when queue stats
-  are immediately observable.
-- `limit` remains `None` because replication worker limits remain owned by the
-  async replication pool and resize policy.
-- If the replication runtime has not initialized, or queue stats are currently
-  locked, the snapshot reports `Unknown` instead of blocking or guessing.
-
-This is an observation surface only. Replication admission, queue channel
-capacity, worker resize behavior, MRF handling, target dispatch, and resync
-behavior are unchanged.
-
-## RustFS Runtime Owner Snapshot Extraction
-
-The RustFS integration layer now extends the workload admission registry with
-additional read-only owner mappings:
-
-- `ForegroundRead` reuses the storage `ConcurrencyManager` disk-read permit
-  snapshot so the RustFS-level provider exposes the same active and limit
-  counts as the storage-local provider.
-- `Scanner` reports the existing scanner active work-unit counter. When the
-  counter is zero, the snapshot remains `Unknown` because the current counter
-  cannot distinguish an idle scanner from a scanner that has not initialized.
-- `Metadata` reports `Open` once the bucket metadata runtime handle is
-  available, and `Unknown` before initialization.
-- `ForegroundWrite` remains `Unknown` until a write-specific admission owner
-  exposes a read-only surface.
-
-This is an observation surface only. Disk-read permit acquisition, scanner
-cycle scheduling, bucket metadata loading, metadata locks, object write paths,
-and queue behavior are unchanged.
-
-## Provider Composition Boundary
-
-`WorkloadAdmissionRegistrySnapshot::overlay` composes provider-owned registry
-snapshots without mutating runtime owners:
-
-- The storage concurrency provider remains the source of truth for
-  `ForegroundRead`.
-- The RustFS runtime owner provider overlays metadata, scanner, repair,
-  replication, and foreground-write status on top of the storage registry.
-- Matching workload classes are replaced by the later provider snapshot; new
-  classes are appended without reordering existing unrelated entries.
-
-This keeps the later controller/status layer consuming a single read-only
-registry while preserving the existing storage, scanner, heal, replication, and
-metadata ownership boundaries.
+`WorkloadAdmissionRegistrySnapshot::overlay` composes provider-owned registries without mutating runtime owners: the storage concurrency provider is the source of truth for `ForegroundRead`; the RustFS runtime owner provider overlays metadata, scanner, repair, replication, and foreground-write status on top; matching classes are replaced by the later snapshot and new classes are appended without reordering. `workload_admission_registry_snapshot` in `rustfs/src/workload_admission.rs` is the single composed registry consumers read.

docs/operations/atomic-object-undo.md

@@ -1,5 +1,8 @@
 # Atomic object undo precondition
 
+**Use this when:** you need the `x-rustfs-expected-current-version-id` precondition for undo-style CopyObject restores or delete-marker removal, or you are changing how it is parsed or enforced.
+**Source of truth:** `rustfs/src/app/object/shared.rs` (`expected_current_version_id` header parser), `rustfs/src/app/object/copy.rs` (CopyObject enforcement), `crates/ecstore/src/set_disk/ops/object.rs` (`expected_current_version_id` checks under the namespace write lock).
+
 RustFS supports a destination-side version precondition for the two S3
 operations used to undo changes in a versioned bucket:
 

docs/operations/authing-oidc-integration.md

@@ -1,240 +0,0 @@
-# Authing OIDC Integration Runbook
-
-This runbook helps operators connect the RustFS Console to Authing through standard OpenID Connect. The examples use the default RustFS provider id, `default`.
-
-## 1. Integration Model
-
-RustFS expects a standards-compliant OpenID Connect provider, not an Authing-specific plugin. The Authing application must provide:
-
-- issuer metadata through `.well-known/openid-configuration`
-- authorization endpoint
-- token endpoint
-- JWKS or another verifiable ID token signature path
-- authorization-code flow that returns an `id_token`
-
-The RustFS browser login flow is:
-
-1. The user opens the RustFS OIDC authorize endpoint.
-2. RustFS creates `state`, `nonce`, and a PKCE S256 challenge.
-3. The browser is redirected to Authing.
-4. Authing redirects back to RustFS with `code` and `state`.
-5. RustFS exchanges the code with `client_id`, `client_secret`, and the PKCE verifier.
-6. RustFS validates the ID token signature, issuer, audience, expiry, and nonce.
-7. RustFS reads identity and authorization claims from the ID token.
-8. RustFS maps claim values to RustFS policy names and issues one-hour STS credentials for the Console.
-
-## 2. Required Values
-
-Collect these values before deployment:
-
-| Value | Example | Notes |
-| --- | --- | --- |
-| Public RustFS browser origin | `https://rustfs.example.com` | The scheme and authority users open in the browser. |
-| Provider id | `default` | This runbook uses the default provider. |
-| RustFS callback URL | `https://rustfs.example.com/rustfs/admin/v3/oidc/callback/default` | Register this exact URL in Authing. |
-| Authing application domain | `https://example.authing.cn` | Use the value shown in the Authing application. |
-| Authing issuer | `https://example.authing.cn/oidc` | Copy the issuer from Authing; do not guess the path. |
-| Authing App ID | `<AUTHING_APP_ID>` | RustFS `client_id`. |
-| Authing App Secret | `<AUTHING_APP_SECRET>` | RustFS `client_secret`. |
-| RustFS scopes | `openid,profile,email,roles` | `openid` is required; include `roles` when Authing emits role claims. |
-
-Authing deployments can use different issuer paths, such as `/oidc` or `/oauth/oidc`. Always copy the issuer from the Authing console and verify that discovery returns the same `issuer` value.
-
-## 3. Authing Configuration
-
-### 3.1 Create the Application
-
-1. Open the Authing console.
-2. Create a self-hosted application named `RustFS Console`.
-3. Record the App ID, App Secret, application domain, issuer, and discovery URL.
-
-### 3.2 Configure OIDC
-
-Use these protocol settings:
-
-| Setting | Value |
-| --- | --- |
-| Protocol | OpenID Connect |
-| Grant type | Authorization Code |
-| Response type | `code` |
-| Token endpoint authentication | `client_secret_post` |
-| PKCE | Allow or require `S256` |
-| ID token signing algorithm | `RS256` recommended |
-
-RustFS sends the client secret in the request body. Do not configure Authing to reject `client_secret_post`.
-
-### 3.3 Register the Redirect URL
-
-Add this exact callback URL in Authing:
-
-```text
-https://rustfs.example.com/rustfs/admin/v3/oidc/callback/default
-```
-
-The scheme, host, port, path, and provider id must match the RustFS configuration.
-
-### 3.4 Map Roles to RustFS Policies
-
-RustFS does not call Authing authorization APIs. It reads `roles` or `groups` from the ID token and maps each value to a RustFS policy name.
-
-Recommended policy names:
-
-| Authing claim value | RustFS policy | Purpose |
-| --- | --- | --- |
-| `consoleAdmin` | `consoleAdmin` | Full Console, admin, KMS, and S3 access. |
-| `readwrite` | `readwrite` | S3 read/write access. |
-| `readonly` | `readonly` | S3 read-only access. |
-| `writeonly` | `writeonly` | S3 write-only access. |
-| `diagnostics` | `diagnostics` | Diagnostic admin access. |
-
-For initial validation, assign a test user the `consoleAdmin` role and confirm that the ID token contains:
-
-```json
-{
-  "roles": ["consoleAdmin"]
-}
-```
-
-`claim_prefix` only prepends a fixed string. It does not perform arbitrary role mapping. Keep Authing role values equal to RustFS policy names unless you already created policies with a fixed prefix.
-
-## 4. RustFS Configuration
-
-### 4.1 Environment Variables
-
-Set the OIDC provider and the public browser origin:
-
-```bash
-export RUSTFS_BROWSER_REDIRECT_URL="https://rustfs.example.com"
-
-export RUSTFS_IDENTITY_OPENID_ENABLE=on
-export RUSTFS_IDENTITY_OPENID_CONFIG_URL="<AUTHING_ISSUER>"
-export RUSTFS_IDENTITY_OPENID_CLIENT_ID="<AUTHING_APP_ID>"
-export RUSTFS_IDENTITY_OPENID_CLIENT_SECRET="<AUTHING_APP_SECRET>"
-export RUSTFS_IDENTITY_OPENID_SCOPES="openid,profile,email,roles"
-export RUSTFS_IDENTITY_OPENID_REDIRECT_URI="https://rustfs.example.com/rustfs/admin/v3/oidc/callback/default"
-export RUSTFS_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC=off
-export RUSTFS_IDENTITY_OPENID_DISPLAY_NAME="Authing"
-export RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM="email"
-export RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM="preferred_username"
-export RUSTFS_IDENTITY_OPENID_ROLES_CLAIM="roles"
-```
-
-For short-lived connectivity testing only, you may temporarily add:
-
-```bash
-export RUSTFS_IDENTITY_OPENID_ROLE_POLICY="consoleAdmin"
-```
-
-Do not keep `role_policy=consoleAdmin` in production unless every Authing user for this client should receive full Console access.
-
-Restart RustFS after changing OIDC settings.
-
-### 4.2 Admin Config
-
-If the deployment manages OIDC through compatible admin configuration commands, set the provider like this:
-
-```bash
-mc admin config set rustfs identity_openid \
-  enable=on \
-  config_url="<AUTHING_ISSUER>" \
-  client_id="<AUTHING_APP_ID>" \
-  client_secret="<AUTHING_APP_SECRET>" \
-  scopes="openid,profile,email,roles" \
-  redirect_uri="https://rustfs.example.com/rustfs/admin/v3/oidc/callback/default" \
-  redirect_uri_dynamic=off \
-  display_name="Authing" \
-  email_claim="email" \
-  username_claim="preferred_username" \
-  roles_claim="roles"
-
-mc admin service restart rustfs
-```
-
-`RUSTFS_BROWSER_REDIRECT_URL` is a process environment variable, not an `identity_openid` provider key. Configure it in the RustFS service environment even when the provider itself is stored through admin config.
-
-### 4.3 Redirect URL Priority
-
-RustFS builds browser-facing URLs with this priority:
-
-1. Provider `redirect_uri`, when configured, is used for the OIDC callback URL sent to Authing.
-2. `RUSTFS_BROWSER_REDIRECT_URL`, when configured, is used as the public origin for OIDC callback generation when no provider `redirect_uri` exists, and for Console success redirects and logout fallback redirects.
-3. Request headers are used only when provider dynamic redirects are enabled and no browser redirect URL is configured.
-
-For reverse-proxy or load-balancer deployments, set `RUSTFS_BROWSER_REDIRECT_URL` to avoid depending on `Host` and `X-Forwarded-Proto` for Console redirects. OIDC authorize and callback requests must still reach the same RustFS node because in-flight OIDC `state` is local to the node.
-
-## 5. Validation
-
-### 5.1 Validate Authing Discovery
-
-```bash
-AUTHING_ISSUER="<AUTHING_ISSUER>"
-
-curl -fsS "$AUTHING_ISSUER/.well-known/openid-configuration" | jq '{
-  issuer,
-  authorization_endpoint,
-  token_endpoint,
-  jwks_uri,
-  id_token_signing_alg_values_supported,
-  code_challenge_methods_supported,
-  token_endpoint_auth_methods_supported,
-  scopes_supported
-}'
-```
-
-Check that:
-
-- `issuer` exactly matches `RUSTFS_IDENTITY_OPENID_CONFIG_URL`
-- `authorization_endpoint`, `token_endpoint`, and `jwks_uri` are present
-- `code_challenge_methods_supported` includes `S256`
-- `token_endpoint_auth_methods_supported` includes `client_secret_post`
-- `scopes_supported` includes `openid`, `profile`, `email`, and any role scope you need
-
-### 5.2 Validate RustFS Provider Visibility
-
-```bash
-curl -fsS "https://rustfs.example.com/rustfs/admin/v3/oidc/providers" | jq
-```
-
-The response should include the Authing provider unless `hide_from_ui` is enabled.
-
-### 5.3 Test Browser Login
-
-Open:
-
-```text
-https://rustfs.example.com/rustfs/admin/v3/oidc/authorize/default
-```
-
-Expected flow:
-
-1. Browser redirects to Authing.
-2. The user signs in.
-3. Authing redirects to `/rustfs/admin/v3/oidc/callback/default?code=...&state=...`.
-4. RustFS validates the ID token and issues STS credentials.
-5. The browser lands on the RustFS Console and can use the expected permissions.
-
-## 6. Troubleshooting
-
-| Symptom | Common cause | Fix |
-| --- | --- | --- |
-| `/oidc/providers` does not show Authing | OIDC provider did not load, or RustFS was not restarted | Check environment variables and restart RustFS. |
-| Authing reports redirect mismatch | Callback URL differs between Authing and RustFS | Use the exact `/rustfs/admin/v3/oidc/callback/default` URL. |
-| RustFS reports missing `code` or `state` | Proxy dropped the query string | Preserve the full callback URL and query string. |
-| Token exchange fails | Wrong client secret or unsupported token auth method | Confirm `client_secret_post` is allowed. |
-| RustFS reports no `id_token` | Missing `openid` scope or non-OIDC OAuth flow | Include `openid` and use OIDC authorization code flow. |
-| ID token verification fails | Issuer, audience, signing algorithm, or JWKS mismatch | Compare discovery metadata with RustFS config; prefer `RS256`. |
-| Login succeeds but access is denied | No matching RustFS policy claim | Ensure `roles` or `groups` is in the ID token and equals a RustFS policy name. |
-| Console redirects to an internal host | Missing `RUSTFS_BROWSER_REDIRECT_URL` or incorrect proxy headers | Set `RUSTFS_BROWSER_REDIRECT_URL` to the public browser origin. |
-| Invalid or expired OIDC state | Callback reached a different RustFS node | Configure load-balancer session affinity for authorize and callback requests. |
-
-## 7. Production Checklist
-
-- [ ] RustFS and Authing use HTTPS.
-- [ ] Authing redirect URL is exact, not a broad wildcard.
-- [ ] `RUSTFS_BROWSER_REDIRECT_URL` is set to the public RustFS browser origin.
-- [ ] `RUSTFS_IDENTITY_OPENID_REDIRECT_URI` matches the registered Authing callback URL.
-- [ ] Authing emits role or group claims in the ID token.
-- [ ] Claim values match RustFS policy names.
-- [ ] `role_policy=consoleAdmin` is not used as a permanent production shortcut.
-- [ ] The load balancer preserves query strings.
-- [ ] OIDC authorize and callback requests have session affinity to the same RustFS node.

docs/operations/container-resource-detection.md

@@ -1,203 +1,70 @@
-# Container Resource Detection
+# Container resource detection
 
-RustFS automatically detects container resource limits (CPU and memory) from cgroup v1/v2. This ensures correct resource allocation and accurate metrics in containerized environments (Kubernetes, Docker, etc.).
+**Use this when:** RustFS runs under a cgroup CPU or memory limit (Kubernetes, Docker) and you need to know which limit it detected, how to override it, or which log line and metrics expose it.
+**Source of truth:** `rustfs/src/cgroup_resources.rs` (detection, `ContainerResources`, `container_resources()`), `rustfs/src/memory_observability.rs` (metrics), `rustfs/src/server/runtime.rs` (Tokio thread sizing consumer), `rustfs/src/startup_entrypoint.rs` (startup log call).
 
-## Problem
+RustFS resolves the CPU core count and the memory limit once at startup (cached in a `OnceLock`) and uses them for Tokio worker/blocking-thread sizing, the memory budget, and memory metrics. Precedence is override env var, then cgroup limit, then host value from `sysinfo`.
 
-When RustFS runs in a container, the underlying system libraries report the **host's** total CPU cores and memory, not the container's limits. This leads to:
+## Detection rules
 
-1. **Over-provisioned Tokio threads**: Too many worker and blocking threads
-2. **Incorrect memory metrics**: `rustfs_memory_usage_percent` shows host-based percentage
-3. **Memory budget errors**: Object data cache sized to host RAM instead of container limit
-4. **OOMKills**: Container exceeds its memory limit and gets killed
+| Resource | Order | Source | Rule |
+| --- | --- | --- | --- |
+| CPU | 1 | cgroup v2 `/sys/fs/cgroup/cpu.max` | `"<quota> <period>"` (or bare `"<quota>"` with period 100000) gives `ceil(quota / period)`; `"max"` or a zero quota means no limit. |
+| CPU | 2 | cgroup v1 `/sys/fs/cgroup/cpu/cpu.cfs_quota_us` with `cpu.cfs_period_us` | `ceil(quota / period)`; a zero, unparsable, or `u64::MAX` quota means no limit. |
+| CPU | 3 | host | `sysinfo` CPU count, minimum 1. |
+| Memory | 1 | cgroup v2 `/sys/fs/cgroup/memory.max` | bytes; `"max"` means no limit. |
+| Memory | 2 | cgroup v1 `/sys/fs/cgroup/memory/memory.limit_in_bytes` | bytes; values `>= 1 << 62` mean no limit. |
+| Memory | 3 | host | `sysinfo` total memory. |
 
-## Solution
+cgroup reads are compiled only for Linux; other platforms always take the host branch. `cgroup_detected` is true when at least one of the two cgroup reads returned a limit.
 
-RustFS now detects cgroup limits directly from the filesystem:
+## Environment variables
 
-- **CPU**: `/sys/fs/cgroup/cpu.max` (v2) or `/sys/fs/cgroup/cpu/cpu.cfs_quota_us` (v1)
-- **Memory**: `/sys/fs/cgroup/memory.max` (v2) or `/sys/fs/cgroup/memory/memory.limit_in_bytes` (v1)
+Names are the constants `ENV_DISABLE_CGROUP_DETECTION`, `ENV_OVERRIDE_CPU_CORES`, and `ENV_OVERRIDE_MEMORY_BYTES` in `rustfs/src/cgroup_resources.rs`.
 
-The effective resource limits are the **minimum** of host and cgroup values.
+| Variable | Accepted values | Effect |
+| --- | --- | --- |
+| `RUSTFS_DISABLE_CGROUP_DETECTION` | `1` or `true` (case-insensitive) | Skip cgroup reads; host values apply unless overridden. |
+| `RUSTFS_OVERRIDE_CPU_CORES` | integer `> 0` | Replaces the CPU core count regardless of cgroup or host. |
+| `RUSTFS_OVERRIDE_MEMORY_BYTES` | integer `> 0`, bytes | Replaces the memory limit regardless of cgroup or host. |
 
-## Detection Logic
+Non-positive or unparsable override values are ignored. Changes take effect on process restart.
 
-### CPU Detection
+## Startup log
 
-1. Read cgroup v2 `/sys/fs/cgroup/cpu.max`
-   - Format: `"$QUOTA $PERIOD"` or `"max"` (unlimited)
-   - Calculate: `cores = ceil(quota / period)`
-2. Fallback to cgroup v1 `/sys/fs/cgroup/cpu/cpu.cfs_quota_us`
-   - Calculate: `cores = ceil(quota / period)`
-3. Fallback to host CPU count from `sysinfo`
+`log_container_resources` emits exactly one of these lines with `cpu_cores` and `memory_bytes` fields (the INFO variants also carry `memory_mib`):
 
-### Memory Detection
-
-1. Read cgroup v2 `/sys/fs/cgroup/memory.max`
-   - Value in bytes or `"max"` (unlimited)
-2. Fallback to cgroup v1 `/sys/fs/cgroup/memory/memory.limit_in_bytes`
-   - Very large values (≥2^62) indicate unlimited
-3. Fallback to host memory from `sysinfo`
-
-## Environment Variables
-
-### Disable Cgroup Detection
-
-```bash
-RUSTFS_DISABLE_CGROUP_DETECTION=1
-```
-
-Disables cgroup detection entirely. Useful for testing or when cgroup filesystem is not accessible.
-
-### Override CPU Cores
-
-```bash
-RUSTFS_OVERRIDE_CPU_CORES=4
-```
-
-Overrides detected CPU cores. Takes precedence over cgroup detection.
-
-### Override Memory Limit
-
-```bash
-RUSTFS_OVERRIDE_MEMORY_BYTES=2147483648
-```
-
-Overrides detected memory limit in bytes. Takes precedence over cgroup detection.
+| Message | Level | Condition |
+| --- | --- | --- |
+| `container resources (overridden by environment variables)` | INFO | an override env var was applied; also carries `cgroup_detected` |
+| `container resources (detected from cgroup)` | INFO | no override, at least one cgroup limit read |
+| `container resources (using host values)` | DEBUG | neither override nor cgroup limit |
 
 ## Metrics
 
-### New Metrics
+Gauges emitted from `rustfs/src/memory_observability.rs`.
 
-| Metric | Description |
-|--------|-------------|
-| `rustfs_memory_effective_total_bytes` | Effective memory total (host or cgroup) |
-| `rustfs_cgroup_detected` | Whether cgroup limits were detected (1=yes, 0=no) |
-| `rustfs_cgroup_cpu_cores_limit` | Detected CPU cores limit |
-| `rustfs_cgroup_memory_limit_bytes` | Detected memory limit |
-
-### Updated Metrics
-
-| Metric | Change |
-|--------|--------|
-| `rustfs_memory_total_bytes` | Now uses effective memory (cgroup-aware) |
-| `rustfs_memory_usage_percent` | Now calculated against effective memory |
-
-## Startup Logging
-
-RustFS logs detected container resources at startup:
-
-```
-INFO container resources (detected from cgroup) cpu_cores=2 memory_bytes=1073741824 memory_mib=1024
-```
-
-or
-
-```
-INFO container resources (overridden by environment variables) cpu_cores=4 memory_bytes=2147483648 memory_mib=2048
-```
-
-## Examples
-
-### Kubernetes with Resource Limits
-
-```yaml
-resources:
-  limits:
-    cpu: "2"
-    memory: "1Gi"
-  requests:
-    cpu: "500m"
-    memory: "512Mi"
-```
-
-RustFS will detect:
-- CPU cores: 2
-- Memory: 1 GiB (1073741824 bytes)
-
-### Docker with CPU and Memory Limits
-
-```bash
-docker run --cpus=2 --memory=1g rustfs/rustfs:latest
-```
-
-RustFS will detect:
-- CPU cores: 2
-- Memory: 1 GiB
-
-### Manual Override
-
-```bash
-export RUSTFS_OVERRIDE_CPU_CORES=4
-export RUSTFS_OVERRIDE_MEMORY_BYTES=2147483648
-```
-
-RustFS will use:
-- CPU cores: 4
-- Memory: 2 GiB
+| Metric | Meaning |
+| --- | --- |
+| `rustfs_memory_effective_total_bytes{basis}` | Effective memory total. `basis` is `cgroup` when a cgroup limit was detected, else `host`; an override does not change the basis label. |
+| `rustfs_container_cpu_cores` | Effective CPU cores. |
+| `rustfs_container_memory_bytes` | Effective memory limit in bytes. |
+| `rustfs_container_cgroup_detected` | `1` when a cgroup limit was read, else `0`. |
+| `rustfs_container_overridden` | `1` when an override env var was applied, else `0`. |
+| `rustfs_memory_total_bytes`, `rustfs_memory_usage_percent` | Computed against the effective total (`record_memory_usage` in `crates/io-metrics/src/lib.rs`). |
 
 ## Troubleshooting
 
-### Cgroup Detection Not Working
+When effective values look like the host rather than the container:
 
-1. Check if cgroup filesystem is mounted:
-   ```bash
-   ls -la /sys/fs/cgroup/
-   ```
+1. Confirm detection is not disabled: `env | grep RUSTFS_DISABLE_CGROUP_DETECTION`.
+2. Find the startup line: `grep "container resources" <rustfs log>`. The `(using host values)` variant is DEBUG, so raise the log level if no variant appears.
+3. Inspect the cgroup filesystem inside the container:
 
-2. Check cgroup version:
-   ```bash
-   stat -fc %T /sys/fs/cgroup/
-   ```
-   - `cgroup2fs` = cgroup v2
-   - `tmpfs` = cgroup v1
+```bash
+stat -fc %T /sys/fs/cgroup/                                                        # cgroup2fs = v2, tmpfs = v1
+cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/memory.max                                # v2
+cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/memory/memory.limit_in_bytes # v1
+```
 
-3. Check if limits are set:
-   ```bash
-   # cgroup v2
-   cat /sys/fs/cgroup/cpu.max
-   cat /sys/fs/cgroup/memory.max
-
-   # cgroup v1
-   cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
-   cat /sys/fs/cgroup/memory/memory.limit_in_bytes
-   ```
-
-### Metrics Show Host Values
-
-If `rustfs_memory_effective_total_bytes` shows host memory instead of cgroup limit:
-
-1. Verify cgroup detection is not disabled:
-   ```bash
-   echo $RUSTFS_DISABLE_CGROUP_DETECTION
-   ```
-
-2. Check startup logs for cgroup detection:
-   ```bash
-   grep "container resources" /logs/rustfs.log
-   ```
-
-3. Use environment variable override as workaround:
-   ```bash
-   export RUSTFS_OVERRIDE_MEMORY_BYTES=1073741824
-   ```
-
-## Implementation Details
-
-### Files Modified
-
-- `rustfs/src/cgroup_resources.rs` - Core cgroup detection logic
-- `rustfs/src/container_config.rs` - Container configuration with overrides
-- `rustfs/src/memory_observability.rs` - Updated memory metrics
-- `rustfs/src/server/runtime.rs` - Updated Tokio runtime configuration
-- `rustfs/src/startup_entrypoint.rs` - Startup logging
-
-### Performance Impact
-
-- **Startup**: One-time detection adds ~1ms overhead
-- **Runtime**: Cached values, no repeated filesystem reads
-- **Memory**: Negligible (<1KB for cached values)
-
-### Thread Safety
-
-All detection functions are thread-safe and use `OnceLock` for caching.
+4. If the runtime does not expose limits to the container, pin them with `RUSTFS_OVERRIDE_CPU_CORES` / `RUSTFS_OVERRIDE_MEMORY_BYTES` and confirm `rustfs_container_overridden` reads `1`.

docs/operations/dial9-runtime-profiling.md

@@ -1,5 +1,8 @@
 # dial9 Tokio Runtime Profiling
 
+**Use this when:** you need Tokio runtime-level evidence (which task held a worker, long polls, park/unpark behaviour) that Prometheus metrics and `tracing` spans cannot provide, or you are building or running the opt-in `dial9` profiling binary.
+**Source of truth:** `crates/obs/src/telemetry/dial9/mod.rs` (session setup), `crates/obs/src/metrics/collectors/dial9.rs` (metrics), `crates/config/src/constants/runtime.rs` (`RUSTFS_RUNTIME_DIAL9_*` and defaults), `.config/make/build.mak` (`build-profiling`), `crates/obs/build.rs` (feature/cfg pairing check).
+
 `dial9-tokio-telemetry` records Tokio runtime-level events — poll start/end,
 worker park/unpark, task spawn/terminate, and optionally async backtraces of
 stalled tasks — into binary trace segments.
@@ -60,9 +63,8 @@ That is `cargo build --release --bin rustfs --features dial9` with
 dumps and S3 upload are both unavailable, for the reasons given above and below.
 
 `crates/obs/build.rs` fails the build if the `dial9` feature is enabled without
-`--cfg tokio_unstable`. This is deliberate: an environment `RUSTFLAGS` *replaces*
-the value from `.cargo/config.toml` rather than appending to it, so the flag used
-to disappear silently whenever anything else set `RUSTFLAGS`.
+`--cfg tokio_unstable`, so a mismatched `RUSTFLAGS` cannot produce a binary that
+silently records nothing.
 
 For CPU profiling with usable stacks, add `-C force-frame-pointers=yes`.
 

docs/operations/drive-timeout-tuning.md

@@ -1,5 +1,8 @@
 # Drive Timeout Tuning
 
+**Use this when:** `ListObjects`/`ListObjectsV2` on a large prefix fails with `Io error: timeout`, or RustFS runs on HDD-class, network, or throttled storage and you need to widen per-operation drive liveness budgets.
+**Source of truth:** `crates/config/src/constants/drive.rs` (`DEFAULT_DRIVE_*_TIMEOUT_SECS`, `DRIVE_TIMEOUT_PROFILE_HIGH_LATENCY_SECS`), `crates/config/src/constants/object.rs` (`DEFAULT_OBJECT_DISK_READ_TIMEOUT`), `crates/config/src/constants/capacity.rs` (`DEFAULT_CAPACITY_MAX_TIMEOUT_SECS`), `crates/ecstore/src/cache_value/metacache_set.rs` (walk stall handling and `rustfs_list_path_raw_stall_total`).
+
 This document describes the per-operation drive timeout knobs and the
 drive-timeout profile. It is written for operators running RustFS on slow or
 high-latency storage (HDD-class disks, network block devices, throttled
@@ -79,10 +82,11 @@ for the full list and defaults.
 `ListObjects`/`ListObjectsV2` on a large prefix either:
 
 - returns `500 InternalError` with `Io error: timeout`; or
-- (on older builds) returns HTTP 200 with `IsTruncated=false` after fewer keys
-  than the bucket actually holds — a **silent** truncation that S3 clients
-  (`mc`, minio-go, SDK pagination loops) cannot detect, because
-  `IsTruncated=false` is the protocol's only end-of-listing signal.
+- on builds without the failure contract below, returns HTTP 200 with
+  `IsTruncated=false` after fewer keys than the bucket actually holds — a
+  **silent** truncation that S3 clients (`mc`, minio-go, SDK pagination loops)
+  cannot detect, because `IsTruncated=false` is the protocol's only
+  end-of-listing signal.
 
 Every "missing" object remains readable by exact key via `GetObject` /
 `StatObject`; only the listing is affected.
@@ -97,21 +101,10 @@ single `readdir` exceed the budget on a perfectly healthy disk, especially on
 HDD-class or throttled storage. That trips a drive timeout, which the listing
 path escalates and surfaces to the client.
 
-### The silent variant is fixed; the loud 500 is tuned away
+### Failure contract
 
-As of the walk-stall rework (merged to `main`, first released in **1.0.0-beta.9**):
-
-- **The silent variant is eliminated.** A walk that dies mid-stream can no
-  longer be consumed as a clean end-of-listing. Once a walk has streamed any
-  entries and then stalls, the failure is recorded as a hard drive timeout and
-  escalated on that erasure set, so the client always sees an error — never a
-  well-formed short page. This is locked by the
-  `list_path_raw_returns_timeout_when_producer_fails_after_partial_entry`
-  regression test in `crates/ecstore/src/cache_value/metacache_set.rs`.
-- **The remaining 500 is an operator-tunable, not a data-integrity bug.** A
-  genuinely wide flat directory can still exhaust the default 5s stall budget on
-  slow storage and fail the listing loudly. The supported mitigation is to widen
-  the budget.
+- A walk that stalls after streaming any entries fails as a hard drive timeout escalated on that erasure set; the client always sees an error, never a well-formed short page. Locked by `list_path_raw_returns_timeout_when_producer_fails_after_partial_entry` in `crates/ecstore/src/cache_value/metacache_set.rs`.
+- The remaining `500` on a genuinely wide flat directory is an operator tunable, not a data-integrity bug: widen the stall budget as below.
 
 ### Mitigation
 
@@ -129,12 +122,6 @@ Raise the walk stall budget, or select the high-latency profile:
 -e RUSTFS_DRIVE_TIMEOUT_PROFILE=high_latency
 ```
 
-> Note: on releases at or before `1.0.0-beta.8`, the foreground listing path was
-> bounded by the *total* wall-clock knob `RUSTFS_DRIVE_WALKDIR_TIMEOUT_SECS`
-> instead of the stall budget. If you cannot upgrade, raise that knob — but
-> upgrading to `1.0.0-beta.9` or later is strongly preferred, because only the
-> newer builds convert the *silent* truncation into a detectable error.
-
 The most durable fix for pathologically wide directories is to shard keys under
 additional prefix levels so no single directory holds an enormous flat child
 set; the stall budget then never has to bound one giant `readdir`.

docs/operations/durability-modes.md

@@ -1,5 +1,8 @@
 # Durability modes (drive sync tiers)
 
+**Use this when:** choosing or debugging the fsync tier (`strict|relaxed|none|legacy-off`) for a deployment or a single bucket, or changing any write-path sync behaviour.
+**Source of truth:** `crates/ecstore/src/disk/local.rs` (`ENV_RUSTFS_DURABILITY_MODE`, `ENV_RUSTFS_DRIVE_SYNC_ENABLE`, mode resolution and per-write-point sync decisions), `crates/ecstore/src/bucket/durability.rs` (`ENV_NEW_BUCKET_DURABILITY_MODE`, per-bucket override), `rustfs/src/admin/handlers/durability.rs` (admin API), `crates/ecstore/src/bucket/metadata_sys.rs` (`BUCKET_METADATA_REFRESH_INTERVAL`).
+
 RustFS lets operators choose how much fsync work runs on the object write
 path. The default (`strict`) preserves the fully synced behavior RustFS has
 always shipped; the relaxed tiers are **opt-in** trades of power-loss
@@ -81,13 +84,13 @@ failure:
   MinIO's default posture (no per-object fsync) but means small objects have
   the widest loss window.
 - Durability of acknowledged writes therefore rests on **erasure-coded
-  redundancy across other nodes** plus the unclean-shutdown heal introduced
-  in PR #4221 converging the affected drive afterwards.
+  redundancy across other nodes** plus the unclean-shutdown heal converging
+  the affected drive afterwards.
 
 Deployment rule for `relaxed`: only multi-node clusters whose nodes sit in
 **independent power domains** (separate feeds/UPS). If all nodes can lose
-power simultaneously — the exact incident class that motivated PR #4221 —
-`relaxed` can lose recently acknowledged objects cluster-wide. Single-node
+power simultaneously, `relaxed` can lose recently acknowledged objects
+cluster-wide. Single-node
 deployments must stay on `strict`.
 
 **`none`.** No fsync on the object data path at all; acknowledged objects can
@@ -119,7 +122,7 @@ staged in tmp still commits with full `strict` durability.
 The durability mode is server-side configuration only; it cannot be raised or
 lowered by any request header.
 
-## Per-bucket durability (phase 2)
+## Per-bucket durability
 
 A bucket can override the process-wide mode with its own tier. The override
 is stored in the bucket's metadata (a `durability.json` entry in
@@ -146,8 +149,8 @@ configuration plane.
 
 ### New-bucket default
 
-A bucket created after this feature ships gets a `relaxed` override **seeded
-into its own metadata** at creation time (rustfs/backlog#1811), so it opts
+A newly created bucket gets a `relaxed` override **seeded into its own
+metadata** at creation time, so it opts
 into MinIO's default posture (object data still fdatasynced; xl.meta and
 directory-entry fsyncs left to the page cache) without touching the
 process-wide default. This is a gradual migration:
@@ -226,20 +229,10 @@ bucket on power failure.
 
 ## Performance expectations
 
-The often-quoted 26x PUT throughput delta was measured on macOS with the old
-binary switch fully **off** (equivalent to `none`/`legacy-off`), where
-`F_FULLFSYNC` heavily amplifies sync cost. `relaxed` keeps the per-shard
-fdatasync, so its gain is necessarily smaller and must be measured on the
-target platform (Linux ext4/xfs) before being relied on. Do not use `none`
-numbers to size `relaxed`.
-
-## Scope
-
-Phase 1 (rustfs/backlog#926) shipped the global, per-process tier configured
-by environment variable. Phase 2 (rustfs/backlog#938) adds the per-bucket
-override described above, configured through the admin API and stored in
-bucket metadata. `mc admin` integration for the per-bucket tier is a
-follow-up.
+Measure `relaxed` on the target platform (Linux ext4/xfs) before relying on a
+number. Throughput deltas measured with sync fully **off** (`none` or
+`legacy-off`) do not transfer: `relaxed` keeps the per-shard fdatasync, so its
+gain is necessarily smaller. Do not use `none` numbers to size `relaxed`.
 
 ## Related Recovery Guides
 

docs/operations/get-path-experimental-switches.md

@@ -1,35 +0,0 @@
-# GET Path Experimental Performance Switches
-
-This document records two experimental environment switches on the object GET
-path. Both default to **off**, are read once at startup, and exist to support
-staged performance work — they are not general tuning knobs. Until this
-document existed they were referenced only by performance harness scripts,
-which made them look like orphans during dead-code sweeps; they are kept
-deliberately (rustfs/backlog#1832).
-
-## RUSTFS_GET_SEEK_BUFFER_ENABLE
-
-- Type: boolean (`true`/`false`), default `false`.
-- Read once at startup in `rustfs/src/app/object_usecase.rs`.
-- When enabled, small GET responses may be served through an in-memory seek
-  buffer, providing seek support without re-reading the object. The seek-buffer
-  code path is unit-test gated; whether the path stays or graduates to default
-  is a post-1.0 maintainer decision — do not remove either the switch or the
-  gated path as dead code.
-
-## RUSTFS_GET_OUTPUT_HANDOFF_ATTRIBUTION_ENABLE
-
-- Type: boolean (`true`/`false`), default `false`.
-- Read once at startup in `rustfs/src/app/object_usecase.rs`.
-- When enabled, GET responses attribute output-handoff stage timing in the GET
-  stage metrics, at a small per-request bookkeeping cost. Used by the A/B
-  performance runbooks (`scripts/run_get_codec_streaming_smoke.sh`,
-  `scripts/test_get_1mib_abba_stage_metrics.sh`) to compare handoff cost
-  between configurations.
-
-## Operational guidance
-
-Leave both switches unset in production. Enable them only when following a
-performance runbook that asks for them, and unset them afterwards — both are
-startup-latched, so changing a value requires a process restart to take
-effect.

docs/operations/heal-concurrency-safety-notes-zh.md

@@ -1,115 +0,0 @@
-# Heal 并发安全说明（对象级 healing 标记对标审计结论）
-
-对应 backlog rustfs/backlog#1874（父 #1862，HS-12）。本文回答一个问题：MinIO 在 heal
-期间对对象打 `x-minio-healing:true` 元数据标记以防"heal 提交与并发删除/版本清理互毁"
-（cmd/xl-storage.go RenameData 的 healing 分支），RustFS 是否需要同款防御。
-
-**结论：不需要。** RustFS 不存在 MinIO 用 healing 标记防御的那类竞争：所有会触达同一
-`(bucket, object)` 提交面的路径都在同一把对象级 namespace 写锁上互斥，且 heal 的锁
-guard 覆盖 rename 提交全程；MinIO 需要标记的根因（RenameData 提交内部与版本清理逻辑
-交错）在 RustFS 的提交模型中不存在。RustFS 已有一个瞬态 healing 旗标用于另一目的
-（见下文 §2），并有并发不变量回归测试锁定本结论（§5）。
-
-## 1. 两个防御模型的对照
-
-MinIO：heal 时对对象写 `x-minio-healing:true`（持久元数据标记），后续任何 RenameData
-提交看到该标记就跳过版本清理/legacy purge 逻辑——防御发生在锁外，靠元数据让路。
-
-RustFS：三层防御，全部不依赖持久对象标记：
-
-1. **锁内互斥**：heal 与一切前台/后台写路径的提交点在同一把 `(bucket, object)` ns 写锁
-   上串行（分布式部署为 quorum 锁 RPC，单机为进程内锁管理器；锁粒度是对象级，version
-   恒为 None）。
-2. **提交模型隔离**：rename_data 提交内没有会与 heal 交错的版本清理逻辑；被替换旧版本
-   的 data_dir 物理删除被移出提交临界区（commit tail），且只删已被新提交替换的 unshared
-   目录。
-3. **瞬态 healing 旗标**：`FileInfo::set_healing`（crates/filemeta/src/fileinfo.rs）在
-   heal 提交的内存 FileInfo 上打 `"healing"` 内部键，rename_data 据此允许先清空 stale
-   目标 data_dir 再 rename——解决 heal 复用 data_dir 做 in-place 修复时 rename(2) 无法
-   替换非空目录的文件系统语义冲突（EEXIST/ENOTEMPTY）。该键是瞬态的，不落盘
-   （`is_skip_meta_key`），与 MinIO 的持久标记目的不同。非 heal 提交撞上非空目标
-   data_dir 会显式失败，有测试锁定两个方向的行为。
-
-## 2. 交点矩阵
-
-中心路径：`heal_object_with_explicit_version_regen`（crates/ecstore/src/set_disk/ops/heal.rs，
-下称 heal.rs）在入口取 `(bucket, object)` ns 写锁，guard 绑定到函数作用域末尾，覆盖
-quorum 元数据读取 → EC 重建 → 逐盘 rename 提交 → tmp 清理 → HEAL_RENAME_INCOMPLETE
-部分提交返回 → 孤儿 data_dir 回收的全过程。并发侧逐交点判定：
-
-| # | 并发路径 | 并发侧锁 | 判定 | 关键证据 |
-|---|---|---|---|---|
-| 1 | PUT 对象提交 | `put_object_commit` 对象写锁，rename_data 在锁内 | 同锁串行 | ops/object.rs 提交锁段 + rename 调用点 |
-| 2 | PUT 旧 data_dir tail 清理 | drop 对象锁后的 `commit_rename_data_dir`，无锁 | 无锁并发，语义安全（见 §3.1） | object.rs drop 后 tail 段；io_primitives.rs |
-| 3 | DELETE 单对象/版本 | `delete_object` 对象写锁，delete_version 在锁内 | 同锁串行 | object.rs delete_object 锁段 |
-| 4 | DELETE 批量 | 批量逐对象写锁（dist 走批量锁 RPC） | 同锁串行 | object.rs delete_objects 锁段 |
-| 5 | CompleteMultipart | 对象写锁 + upload 路径锁双锁，rename 在锁内 | 同锁串行 | ops/multipart.rs 提交锁段 |
-| 6 | CompleteMultipart tail 清理 | drop 对象锁后的旧 data_dir 删除 | 无锁并发，语义安全（见 §3.1） | multipart.rs drop 后 tail 段 |
-| 7 | AbortMultipart | 仅 multipart bucket 的 upload 路径锁 | 锁 key 不相交，但资源不相交（abort 不触对象 data_dir/xl.meta）→ 无实际交点 | multipart.rs abort 锁段 |
-| 8 | ILM expiry（含 DeleteAllVersions） | DeleteAllVersions 走 `delete_prefix_object=true` → 仍取对象锁；FreeVersionTask 显式取锁；noncurrent 批量走批量锁 | 同锁串行 | bucket_lifecycle_ops.rs 消费端链路 |
-| 9 | 纯 prefix 删除（绕锁能力面） | `delete_prefix`-only 不取子对象锁 | 无锁并发，但生产调用方为零（见 §3.2） | object.rs delete_object 锁条件 |
-| 10 | 孤儿 data_dir 回收 reclaim_orphan_data_dirs | 函数本体无锁；唯一生产调用方在 heal 锁内 | heal 流程内=锁内串行 | heal.rs 收尾调用；io_primitives.rs |
-| 11 | 旧清理 receipt 对账 reconcile_old_data_cleanup_receipts | 函数本体无锁；调用点在 heal 锁内 + epoch fence 防误删 | 锁内串行 | object.rs 对账函数 |
-| 12 | replication | 数据面为远端 HTTP 写（不落本地盘）；本地元数据回写走对象锁 | 同锁串行 / 无交点 | replication_resyncer.rs 链路 |
-| 13 | data_movement / rebalance / decommission 源清理 | 显式取对象锁 + 版本未变复核 + guard 复用（no_lock 只是复用已持锁） | 同锁串行 | data_movement/mod.rs 源清理 |
-| 14 | copy_object | 目标对象锁 / 走 put 链锁 | 同锁串行 | object.rs copy_object 锁段 |
-| 15 | 另一 heal 任务（跨 HealType/force_start） | dedup key 跨类型不相交 + force_start 跳过去重 → 任务级可并发 | 最终在 ns 写锁上串行 | heal/manager.rs dedup key 构成 |
-| 16 | admin `no_lock=true` heal | 客户端可控绕锁 | 无锁并发，明示运维选项（见 §3.3） | admin/handlers/heal.rs 透传 |
-| 17 | stale multipart 清理 | multipart bucket 的 upload 路径锁 | 资源不相交 → 无交点 | bucket_lifecycle_ops.rs 清理链路 |
-
-## 3. 残留窗口定性
-
-### 3.1 PUT/CompleteMultipart commit tail（交点 2/6）
-
-写路径提交成功、释放对象锁之后，才 best-effort 删除被替换的旧 data_dir（注释明示有意
-不阻塞下一操作）。该删除与并发 heal 对同一旧 data_dir 的读取/重建存在竞态窗口，但语义
-安全：
-
-- 删除目标是已被新提交替换的 unshared data_dir；heal 的 canonical 元数据来自 quorum
-  仲裁（ETag/mod_time），此时 quorum 已指向新版本，heal 不会把已替换版本当作 canonical
-  复活；
-- 竞态最坏后果 = heal 当轮对旧版本的一次 transient 失败/空转，重试轮自然收敛；清理
-  residue 会上报并重新入队 heal（`report_old_data_dir_cleanup`）；
-- 换盘重建等长 heal 走 per-version 显式版本请求，quorum 元数据在锁内读取，不受 tail
-  影响。
-
-### 3.2 纯 prefix 删除（交点 9）
-
-`delete_prefix && !delete_prefix_object` 的路径不取子对象锁（对象名空间锁无法保护前缀
-递归删除），与并发 heal 存在理论复活窗口（heal 在 prefix 删除进行中依据旧 quorum 元
-数据重建某版本）。全仓库核对结论：该路径的**生产调用方为零**——所有生产 `delete_prefix:
-true` 调用点均同时设置 `delete_prefix_object: true`（从而取对象锁）或在测试模块内。这
-是 API 能力面的暴露而非行为风险。若未来有调用方需要纯 prefix 删除，须在调用点证明与
-heal/scanner 的隔离（例如 bucket 级停扫围栏）。
-
-### 3.3 admin `no_lock=true`（交点 16）
-
-admin heal 请求可透传客户端 `nolock` 参数绕过 ns 锁（与 MinIO madmin 的同名选项对齐）。
-这是运维明示选项：使用即自负与并发写的竞争责任。文档化即可，不建议收紧。
-
-## 4. heal 侧自身的不变量保障
-
-- dedup key 跨 HealType 不相交（object/metadata/mrf/ecdecode/prefix 各自键面）+ admin
-  `force_start` 可跳过去重 → 同对象可能同时存在多个 heal 任务，但它们的执行体全部在
-  `heal_object` 入口的 ns 写锁上串行（生产入口均 `no_lock=false`）；
-- read-repair 的本地 TTL 预留只去重自身来源，不拦截其他来源的 heal——同样由 ns 锁兜底；
-- healing 旗标不落盘，故不存在"标记残留导致后续提交错误让路"的反向风险。
-
-## 5. 回归测试
-
-以下两个并发不变量测试随本审计加入 `crates/ecstore/src/set_disk/ops/heal.rs` 测试模块：
-
-- `heal_racing_version_delete_never_resurrects_the_deleted_version`：注入 doomed 版本
-  shard 损坏后，版本化 DELETE 与 Deep heal 真并发（同一把锁争用），断言已删除版本不被
-  复活、存活版本完好；
-- `heal_racing_unversioned_overwrites_preserves_the_last_commit`：非版本化覆盖提交（激活
-  commit tail 旧 data_dir 删除）与 Deep heal 循环竞态，断言最终 current 恰为最后一次
-  提交（etag 级一致）。
-
-## 6. 结论
-
-MinIO 的 `x-minio-healing` 是锁外元数据防御，前提是其 RenameData 提交内部存在与 heal
-交错的版本清理逻辑；RustFS 的提交模型把这类交错从根上消除（提交面锁内互斥 + 清理外
-移到 tail + tail 只删 unshared 旧目录），因此引入持久对象级 healing 标记没有对应的竞争
-可防，反而会引入 FileInfo 落盘格式变更与标记残留清理两类新成本。维持现状，本对标疑点
-关闭。

docs/operations/heal-scanner-parity-notes-zh.md

@@ -1,109 +0,0 @@
-# Heal/Scanner 配置与语义对照（MinIO parity 决策记录）
-
-对应 backlog rustfs/backlog#1878（父 #1862，批 HS-14/HS-16/HS-18）。本页沉淀三项"决策 + 文档化"结论：scanner idle 节流语义对照与迁移警告（HS-14）、单机默认扫描周期决策（HS-16）、stale multipart 与 tmp/.trash 清理三段核对（HS-18），并顺带收录 bitrot_cycle 与 alert_excess_folders 两项已确认的默认值差异。所有 MinIO 侧结论均于 2026-08 按 minio/minio master 逐源码核对（引用文件为上游路径），不转述二手资料。
-
-运行时旋钮的完整清单、状态端点与调参流程见 [Scanner Runtime Controls](scanner-runtime-controls.md)；excess 告警阈值差异见 [Scanner Excess Alerts](scanner-excess-alerts_zh.md)；heal 并发模型对照见 [Heal 并发安全说明](heal-concurrency-safety-notes-zh.md)。
-
-## 1. HS-14：scanner idle 节流语义对照
-
-### RustFS 当前语义（三因子）
-
-RustFS 的 scanner 步进节流由三个因子共同决定（crates/scanner/src/sleeper.rs）：
-
-1. **总闸 `scanner.idle_mode` / `RUSTFS_SCANNER_IDLE_MODE`（默认 `true`）**：`false` 时所有节流 sleep 全部跳过，scanner 全速推进；`true` 时按下面两因子计算 sleep。
-2. **速度档**（`scanner.speed` / `RUSTFS_SCANNER_SPEED`，默认 `default`）：档位表与 MinIO 完全一致（见下表）。目录级 sleep = `1ms × factor`（上限 `max_wait`）；对象级 sleep = `本对象处理耗时 × factor`，下限 1ms、上限 `max_wait`。
-3. **前台读退避下限**：`current_foreground_read_activity()` 取并发 GetObject 请求数（rustfs/src/storage/concurrency/request_guard.rs 的 `GetObjectGuard`）与流式读计数（`ForegroundReadGuard`）的较大值，换算为 `10ms × 活跃读数`、封顶 250ms 的下限；该下限对目录级与对象级 sleep 都生效（`.max(foreground_sleep)`），且**可以超过速度档的 `max_wait`**（自身封顶 250ms）。速度档为 `fastest`（factor=0）时预设 sleep 为 0，但只要 `idle_mode=true`，前台读下限仍然生效。
-
-| 速度档 | sleep factor | 单次 sleep 上限 | 周期间隔 |
-|---|---:|---:|---:|
-| `fastest` | 0 | 0 | 1s |
-| `fast` | 1× | 100ms | 1m |
-| `default` | 2× | 1s | 1m |
-| `slow` | 10× | 15s | 1m |
-| `slowest` | 100× | 15s | 30m |
-
-实际行为矩阵（RustFS）：
-
-| `idle_mode` | 速度档 | 前台并发读 = 0 | 前台并发读 > 0 |
-|---|---|---|---|
-| `false` | 任意 | 完全不休眠，全速 | 完全不休眠，全速（前台退避也被总闸关闭） |
-| `true` | `fastest` | 预设 sleep = 0，等效全速 | 每步 sleep = 前台读下限（10ms×读数，封顶 250ms） |
-| `true` | 其余档 | 每步 sleep = 预设值（1ms~15s 封顶） | 每步 sleep = max(预设值, 前台读下限) |
-
-周期间隔的解析优先级为 env `RUSTFS_SCANNER_CYCLE` > 持久化 `scanner.cycle` > `scanner.start_delay` > 启动期默认覆盖（当前恒无）> 速度档派生（crates/scanner/src/runtime_config.rs）。另有 `scanner.yield_every_n_objects`（默认 128）的协作式让出，与节流 sleep 相互独立。
-
-### MinIO 当前语义（master 逐源码核对）
-
-MinIO 的对应开关是 `scanner:idle_speed` / `MINIO_SCANNER_IDLE_SPEED`（internal/config/scanner/scanner.go）：取值为空串或 `on`（默认）时 `IdleMode=0`，取值 `off` 时 `IdleMode=1`。启动/配置加载时一次性写入 `scannerIdleMode`（cmd/config-current.go），扫描侧闭包 `weSleep = scannerIdleMode.Load() == 0`（cmd/xl-storage-disk-id-check.go）：**`on`（默认）= 目录级与对象级节流 sleep 始终插入（按速度档 factor，minSleep 100µs）；`off` = 两条节流路径完全不 sleep，全速扫描**。当前上游没有任何按 S3 请求/磁盘活动动态调整节流的逻辑——这是静态开关。
-
-命名具有误导性，是历史残留：2024-01 之前 `weSleep` 由磁盘活动驱动（"Entire queue is full, so we sleep"，即有并发 S3/heal 活动才 sleep），minio/minio#18734（commit 7705605b）把该活动门替换为上述静态配置（初版取值 `throttled`/`full`，后改为 `on`/`off`），上游残留注释 "default is throttled when idle"、"Sleep always or based on incoming S3 requests" 均是替换前的语义描述，与现行代码不符。
-
-### 对照与迁移警告
-
-| 维度 | RustFS | MinIO（master） |
-|---|---|---|
-| 开关名 | `scanner.idle_mode` / `RUSTFS_SCANNER_IDLE_MODE` | `scanner:idle_speed` / `MINIO_SCANNER_IDLE_SPEED` |
-| 取值 | 布尔 `true`/`false` | `on`/`off` |
-| 默认 | `true`（节流开启） | `on`（节流开启） |
-| 开 = | 节流总闸开：速度档 sleep + 前台读下限 | 节流总闸开：速度档 sleep |
-| 关 = | 完全不休眠（含前台读下限一并失效） | 完全不休眠 |
-| 活动耦合 | 有：前台并发读抬高 sleep 下限（10ms×读数，封顶 250ms） | 无（2024-01 起为静态开关） |
-| 速度档表 | 两边完全一致（上表） | 同左 |
-
-迁移警告：
-
-- **环境变量名不可照搬**：RustFS 只读取 `RUSTFS_*` 前缀，不解析 `MINIO_SCANNER_*` 任何别名（crates/scanner、crates/utils 的 env 读取无别名链，测试还专门断言 `MINIO_SCANNER_SPEED`/`MINIO_SCANNER_CYCLE` 不泄漏生效）。照搬 `MINIO_SCANNER_IDLE_SPEED=off` 到 RustFS 会静默无效，必须改写成 `RUSTFS_SCANNER_IDLE_MODE=false`。
-- **取值词表不同**：`on/off` vs `true/false`，不能原样复制。
-- **方向澄清（修正父 issue 的预设）**：按当前上游源码，MinIO `idle_speed` 与 RustFS `idle_mode` 在"开=节流、关=全速"方向上是一致的，并非反向；父 issue 中"MinIO on=集群空闲才节流、off=始终按 delay 节流"的矩阵描述的是 2024-01 之前的活动耦合行为与反向解读，与 master 不符。真正需要写进迁移手册的差异是：MinIO 的 `idle_speed` 名称暗示"空闲时才慢"但实际是静态总闸；RustFS 的 `idle_mode=true` 在总闸之上还叠加了 MinIO 没有的前台读保护下限。
-- **`false` 是大锤**：RustFS `idle_mode=false` 会连前台读退避一起关闭，scanner 与前台读完全抢盘；仅在 benchmark 或可独占 IO 的窗口使用。
-
-**决策（HS-14）：保持现状。** RustFS 语义更直观（`idle_mode` = 节流总闸，`true` 即自适应限速），且比 MinIO 多一层前台读保护；不新增 `RUSTFS_SCANNER_IDLE_SPEED` 兼容别名（无社区强诉求不做，避免双入口漂移）。本节即对照表与迁移警告的正式落点。
-
-## 2. bitrot_cycle 默认差异
-
-| 项 | RustFS | MinIO |
-|---|---|---|
-| 键 | `heal.bitrot_cycle` / `RUSTFS_SCANNER_BITROT_CYCLE_SECS`（scanner.bitrot_cycle 为兼容旧键） | `heal:bitrotscan` / `MINIO_HEAL_BITROTSCAN` |
-| 默认 | 30 天（crates/config/src/constants/heal.rs 的 `DEFAULT_HEAL_BITROT_CYCLE_SECS`）：按墙钟周期把扫描切深扫（deep bitrot） | `off`（internal/config/heal/heal.go 默认 `EnableOff`）：不做周期性深扫，仅普通扫描 + 管理端手动深扫 |
-| 对齐方式 | 迁移 MinIO 行为：`heal.bitrot_cycle=off` 或 `RUSTFS_SCANNER_BITROT_CYCLE_SECS=disabled` | 反向：`heal:bitrotscan=<秒>` |
-
-RustFS 的 30 天默认是刻意的耐用性默认（周期性全量 bitrot 校验），代价是每 30 天一轮深扫 IO；单机场景另有清洁空闲退避封顶约 42 分钟的墙钟保护（见 scanner-runtime-controls.md）。这是行为差异而非缺陷，文档化即可。
-
-## 3. alert_excess_folders 默认差异
-
-RustFS 默认 65538（容纳 Proxmox Backup Server 每目录 65536 chunk 的布局），MinIO 默认 50000。差异原因、另两个 excess 阈值（versions=100 相同、version_size TiB vs TB）、事件名映射与冷却语义已完整记录在 [Scanner Excess Alerts](scanner-excess-alerts_zh.md)，此处不重复。
-
-## 4. HS-18：stale multipart 与 tmp/.trash 清理三段核对
-
-MinIO 把"清理已删除数据"拆成三段：stale upload 先 rename 进 `.minio.sys/tmp/.trash/<uuid>` 隔离（rename 快、原子）；trash 由独立例程排空；tmp 下非 trash 的旧目录单独回收。逐段核对 RustFS：
-
-| 段 | MinIO | RustFS | 判定 |
-|---|---|---|---|
-| stale multipart → 隔离 | `cleanupStaleUploadsOnDisk`（cmd/erasure-multipart.go）逐盘列出 multipart 目录，按 uploadID 目录名里的 UnixNano 判龄，超过 `stale_uploads_expiry`（默认 24h）即 `renameAll` 进 `.minio.sys/tmp/.trash/<uuid>`，空 sha 目录、tmp 旧目录同法 | `cleanup_stale_multipart_uploads_in_set`（crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs）发现候选后取 ns 写锁 + 重查（`lock_stale_multipart_cleanup`），`delete_all_with_quorum` 扇出逐盘递归删除，而 LocalDisk 的递归删除内部就是 `move_to_trash`（crates/ecstore/src/disk/local.rs）把目录 rename 进 `.rustfs.sys/tmp/.trash/<uuid>` | 行为等价（都是先隔离后清理）；RustFS 额外有写锁 + quorum 重查 + 锁丢失 fence（crates/ecstore/src/set_disk/ops/multipart.rs 的 `StaleMultipartCleanupGuard`），防并发 CompleteMultipartUpload 竞争，安全性强于 MinIO 的无锁 rename |
-| trash 排空 | 每 `delete_cleanup_interval`（默认 5m，internal/config/api/api.go）逐盘删 `.trash` 内条目，逐条以 `deleteCleanupSleeper`（factor 5 / 25ms，cmd/globals.go）节流 | 每盘独立 `cleanup_deleted_objects_loop`，`DELETED_OBJECTS_CLEANUP_INTERVAL` = 5m（crates/ecstore/src/disk/local.rs），先排空 `.trash` 再回收 tmp 旧目录；排空为顺序 `remove_dir_all`/`remove_file`，**无逐条 sleep 节流** | 基本等价；唯一差异是 RustFS 排空不节流，trash 积压大时单轮 IO 更突发（5m 周期天然限频），文档化，如实测出现清理风暴再补节流 |
-| tmp 非 trash 旧目录 | 并在 `cleanupStaleUploadsOnDisk` 内：非 `.trash` 的 tmp 目录超过 `stale_uploads_expiry`（24h）rename 进 trash（随 6h 任务） | `cleanup_stale_tmp_objects`（crates/ecstore/src/disk/local.rs）随 5m 循环执行：非 `.trash` 目录超过 `STALE_TMP_OBJECT_EXPIRY` = 24h 即 rename 进 trash；另有启动时 tmp → tmp-old 整体换名 + 后台删除的崩溃安全路径 | 行为等价（阈值同为 24h）；RustFS 检查频率 5m vs MinIO 6h，回收更及时 |
-
-周期与环境变量默认值对照（两边一致）：
-
-| 项 | RustFS | MinIO |
-|---|---|---|
-| stale upload 过期阈值 | `RUSTFS_API_STALE_UPLOADS_EXPIRY`，默认 24h | `MINIO_API_STALE_UPLOADS_EXPIRY`，默认 24h |
-| stale multipart 清理周期 | `RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL`，默认 6h | `MINIO_API_STALE_UPLOADS_CLEANUP_INTERVAL`，默认 6h |
-| trash 排空周期 | 5m（常量，暂无开关） | `MINIO_API_DELETE_CLEANUP_INTERVAL`，默认 5m |
-
-关于 rustfs/src/delete_tail_activity.rs：它**不覆盖三段中的任何一段**。该模块是 delete 尾部活动的进程内指标计数（inflight gauge + 耗时 histogram），供 allocator 回收压力判断（rustfs/src/allocator_reclaim.rs）使用；生产代码目前只在对象复用路径使用 `Replication`/`Notify` 两个 stage 计数，`Tail`/`Cleanup` 枚举值暂无调用点。
-
-崩溃残留窗口结论：
-
-- trash 内部残留（排空中途崩溃）：`.trash/<uuid>` 是自包含目录，下一轮 5m tick 重扫 `.trash` 自然收敛，与 MinIO 相同。
-- 跨盘扇出中途崩溃（部分盘已 rename 进 trash、其余未动）：若剩余盘数仍满足写 quorum，下一轮 6h 任务重新发现候选并重删，自然收敛；若已清理盘数超过 parity（剩余低于写 quorum），`check_multipart_upload_path_exists` 因 `FileNotFound` 不在 `OBJECT_OP_IGNORED_ERRS`（crates/ecstore/src/disk/error_reduce.rs）而判 quorum 失败，候选被跳过，残留 uploadID 目录不会被该任务收敛（不可见于 S3 API，仅占盘空间）。该窗口极窄（逐盘 rename 为毫秒级，需恰在扇出中途且已过 parity 盘时进程死亡）。MinIO 同场景会收敛（逐盘独立处理、无 quorum 闸门）。**分级：有崩溃残留窗口（极窄）→ 登记后续修复**；修复需为清理守卫提供把"已不存在"计为达成终态的专用 quorum 变体（不能改共享的 `check_multipart_upload_path_exists` 语义，它同时服务 CompleteMultipartUpload），超出本批"几行小修"边界，不在本 PR 扩 scope。
-
-## 5. HS-16：单机（ErasureSD）默认扫描周期决策
-
-启动期曾有预留钩子 `single_disk_default_cycle_secs`，可按维护特征（lifecycle/replication/巡检失败）为单机覆盖默认周期，但从未接线、恒返回 `None`，已删除（本批 PR）。决策：**单机默认周期保持速度档派生（`default` 档 = 60s），不做特殊覆盖**。理由：其一，无任何实测依据表明单机冷启动 ILM 延迟需要更短周期，凭空缩短只会放大空闲扫描频次；其二，单机已有清洁空闲退避（连续干净周期间隔翻倍，默认 bitrot 窗口下封顶约 42 分钟，见 scanner-runtime-controls.md），空闲时的周期压力已被消化；其三，若确有诉求，用户可用 `RUSTFS_SCANNER_CYCLE` / `scanner.cycle` 显式配置，无需内置特殊路径。需要更激进短周期的场景应先拿实测数据再议。
-
-## 6. 决策摘要
-
-- HS-14：保持 `RUSTFS_SCANNER_IDLE_MODE` 现语义（true=节流总闸+前台读下限，false=全速），文档化对照表与迁移警告，不做兼容别名。
-- HS-16：删除恒 `None` 的单机默认周期钩子，单机周期保持速度档派生 + 清洁空闲退避。
-- HS-18：三段清理行为等价（trash 排空无逐条节流、tmp 回收频率 5m vs 6h 两处小差异文档化）；跨盘扇出的极窄崩溃残留窗口登记后续；周期默认值 24h/6h/5m 与 MinIO 对齐。

docs/operations/hotpath-warp-ab-runbook.md

@@ -1,164 +1,178 @@
-# Hotpath warp A/B runbook
+# Hotpath warp runbook (A/B gate and ABBA evidence)
 
-Relative-budget A/B gate for the hotpath series (rustfs/backlog#935 HP-14). It
-runs the same warp workloads against a **baseline** binary and a **candidate**
-binary, across the drive-sync on/off matrix, then applies a relative budget:
-a metric regressing past the fail budget fails the gate, past the warn budget
-warns. This is how the macOS profiling conclusions of the HP series get
-confirmed or corrected on Linux — structural wins (call counts, read
-amplification) should hold; absolute numbers are whatever the rig measures.
+**Use this when:** you need S3-face (PutObject / GetObject / mixed) performance evidence for a code change: the nightly relative-budget gate, a quick local A/B, or a formal ABBA run whose numbers will be quoted in a PR.
+**Source of truth:** `scripts/run_hotpath_warp_ab.sh` (quick A/B rig), `scripts/run_hotpath_warp_abba.sh` (ABBA runner; also what CI executes), `scripts/hotpath_warp_ab_gate.sh` (relative-budget gate), `scripts/run_object_batch_bench_enhanced.sh` (warp driver, medians, `baseline_compare.csv`), `.github/workflows/performance-ab.yml` (CI gate).
 
-Pieces:
+This runbook compares two binaries. To sweep one `RUSTFS_*` runtime knob at a time against a fixed binary, use [object-io-tuning-ab-matrix.md](object-io-tuning-ab-matrix.md) instead.
 
-- `scripts/run_hotpath_warp_ab.sh` — orchestrator (baseline vs candidate,
-  workload × drive-sync matrix).
-- `scripts/hotpath_warp_ab_gate.sh` — the budget gate over the
-  `baseline_compare.csv` deltas the load driver emits.
-- `scripts/run_object_batch_bench_enhanced.sh` — the warp driver + median +
-  `baseline_compare.csv` (reused, not reimplemented).
-- `.github/workflows/performance-ab.yml` — nightly on `main` (post-merge
-  detection) plus opt-in pre-merge via the `perf-ab` label.
+Two entry points share one workload matrix, one gate, and one deploy-hook shape:
 
-Metric directions: `reqps` (put obj/s) and `throughput` (get MiB/s) are
-higher-is-better; `latency` / p99 (mixed) is lower-is-better. warp is assumed
-pre-installed, as elsewhere in `scripts/`.
+| Entry | Script | Legs per cell | Use |
+| --- | --- | --- | --- |
+| Quick A/B | `scripts/run_hotpath_warp_ab.sh` | baseline → candidate | local smoke, fast triage of a suspected regression |
+| ABBA | `scripts/run_hotpath_warp_abba.sh` | A1 baseline → B1 candidate → B2 candidate → A2 baseline | formal evidence; the CI gate |
+
+This rig is the only entry point for S3-face performance coverage. There is deliberately no in-process criterion benchmark for those operations: a criterion harness that stands up an embedded server measures the harness, not the S3 path. Micro-benchmarks stay at function level (EC encode, `xl.meta` parse, `rename_data`). Knob-level sweeps against a fixed binary are a different question; see [object-io-tuning-ab-matrix.md](object-io-tuning-ab-matrix.md).
+
+## Shared prerequisites
+
+- Linux host. A laptop is acceptable for a quick A/B smoke only; formal evidence needs a dedicated runner or a cluster.
+- `warp` on `PATH`, or `--warp-bin <path>`.
+- Two Linux release binaries, baseline and candidate (build each with `cargo build --release -p rustfs --bins` at its commit; cross-compile with `cargo zigbuild --release --target x86_64-unknown-linux-gnu -p rustfs --bins` for a cluster).
+- Disposable disks or data root in local mode; an isolated benchmark bucket and credentials in cluster mode. Never bench against production data.
+- Readiness polling (`--health-timeout`, default 180 s in both scripts) must outlast the server's own startup budget, `DEFAULT_STARTUP_READINESS_MAX_WAIT_SECS` in `crates/config/src/constants/health.rs`; a shorter poll misreports a slow cold start as a failure.
+- Metric directions: `reqps` (put obj/s) and `throughput` (get MiB/s) are higher-is-better; `latency`/p99 (mixed) is lower-is-better.
 
 ## Workload matrix
 
-Six workloads × the drive-sync on/off matrix × baseline/candidate = 24 cells:
+Every workload runs with `RUSTFS_DRIVE_SYNC_ENABLE=true` and `=false`. Quick A/B: 6 × 2 × 2 = 24 cells; ABBA: 6 × 2 × 4 = 48 cells.
 
-| Workload | mode | size | why |
+| Workload | mode | size | Why |
 | --- | --- | --- | --- |
-| `put-4kib` / `get-4kib` | put / get | 4KiB | the #4221 fsync regression size (~-10% @4KiB) — previously invisible |
-| `put-4mib` / `get-4mib` | put / get | 4MiB | bulk obj/s and MiB/s |
-| `get-10mib` | get | 10MiB | the historical large-GET EOF size |
-| `mixed-256k` | mixed | 256KiB | p99 latency |
+| `put-4kib` / `get-4kib` | put / get | 4 KiB | small-object fsync-sensitive path |
+| `put-4mib` / `get-4mib` | put / get | 4 MiB | bulk obj/s and MiB/s |
+| `get-10mib` | get | 10 MiB | large-GET streaming path |
+| `mixed-256k` | mixed | 256 KiB | p99 latency |
 
-Sizes are passed to the load driver via `--sizes` (one size per cell); the
-driver's `DEFAULT_SIZES` covers 1KiB..10MiB, so any of those can be added by
-editing `WORKLOADS` in `scripts/run_hotpath_warp_ab.sh`. A 1KiB cell is left
-out for now to keep the nightly matrix comfortably under budget; re-enable it
-(one line in `WORKLOADS`) once perf-6 recalibrates the warp params.
+Sizes are passed to the driver as `--sizes` (one per cell); any size in the driver's `DEFAULT_SIZES` (`scripts/run_object_batch_bench_enhanced.sh`) can be added by editing the `WORKLOADS` array in the chosen script.
 
-CI runs a **short** warp matrix (`--duration`/`--rounds`/`--cooldown` tuned in
-`.github/workflows/performance-ab.yml`) so all 24 cells fit the budget without
-dropping cells. These params are deliberately noisy-but-fast for the Phase-0
-"keep the pipeline alive" goal; perf-6 recalibrates them.
+## Gate
 
-## Baseline binary cache (CI)
+`scripts/hotpath_warp_ab_gate.sh` compares each metric against baseline: a regression beyond `--fail-pct` (default `FAIL_PCT=10`) fails, beyond `--warn-pct` (default `WARN_PCT=5`) warns. `--allow-regression --exemption-reason "<why>"` records a FAIL as an exempted WARN and exits 0; use it only for a deliberate correctness trade (for example paying write cost to restore power-loss durability).
 
-The nightly no longer builds both binaries from source. Every push to `main`
-runs a `build-baseline-cache` job that builds the release binary once and stores
-it in the actions cache under `rustfs-baseline-<sha>` (perf-3). The A/B job
-restores the binary for `origin/main` by that key and passes it as
-`--baseline-bin`; on the nightly, where the candidate commit equals the baseline
-commit, the same cached binary serves both phases (`--skip-build`) and the run
-does zero source builds — the common path finishes well under 50 minutes. A
-cache miss (binary evicted, or not built for that SHA yet) transparently falls
-back to the source double-build via `--baseline-ref origin/main`.
+## Deploy-hook contract (external / cluster mode)
 
-Each `gate.md` ends with a **Provenance** section recording the baseline and
-candidate commit SHAs and whether each binary came from the cache or a source
-build, plus the runner, warp version, and matrix params. `perf-5`/`perf-12`
-reuse this contract for their archived baselines.
+In external mode (`--endpoint <host:port>`) the rig never starts or restarts RustFS. Before each phase or leg it runs `--deploy-hook <cmd>` with the context below in the environment, then waits for `http://<endpoint><health-path>` (default `/health`). A non-zero hook exit aborts the run.
 
-## Local mode (quick / CI smoke)
+| Variable | Quick A/B | ABBA | Value |
+| --- | --- | --- | --- |
+| `HOTPATH_AB_PHASE` / `HOTPATH_ABBA_PHASE` | yes | yes | `baseline` or `candidate` |
+| `HOTPATH_AB_BINARY` / `HOTPATH_ABBA_BINARY` | yes | yes | selected binary path (A/B: may be empty if the hook builds its own) |
+| `HOTPATH_AB_DRIVE_SYNC` / `HOTPATH_ABBA_DRIVE_SYNC` | yes | yes | `true` or `false` for this cell |
+| `HOTPATH_ABBA_LEG` | — | yes | `A1`, `B1`, `B2`, `A2` |
+| `HOTPATH_ABBA_WORKLOAD`, `HOTPATH_ABBA_MODE`, `HOTPATH_ABBA_SIZE`, `HOTPATH_ABBA_CELL_ID` | — | yes | cell identity |
+| `HOTPATH_ABBA_DATASET_NAMESPACE`, `HOTPATH_ABBA_BUCKET` | — | yes | run namespace and per-leg benchmark bucket the hook must provision or reset |
+| `HOTPATH_ABBA_DEPLOY_EVIDENCE_FILE` | — | yes | path the hook must write a non-empty evidence file to |
 
-Builds both binaries and runs a throwaway single-node server on local disks.
+The ABBA runner refuses external mode without a hook unless `--allow-unmanaged-external` is passed, and output from that mode is not formal evidence.
+
+Ansible-shaped hook (replace `/path/to/ansible` and the inventory group; the `config` tag must thread `RUSTFS_DRIVE_SYNC_ENABLE`, or the finer `RUSTFS_DURABILITY_MODE`, into the deployed unit):
 
 ```bash
+--deploy-hook '
+  set -euo pipefail
+  cd /path/to/ansible
+  cp "${HOTPATH_ABBA_BINARY:?}" roles/rustfs/files/rustfs
+  export RUSTFS_DRIVE_SYNC_ENABLE="${HOTPATH_ABBA_DRIVE_SYNC:?}"
+  ansible-playbook -f 4 -l bench rustfs-manage.yml --tags stop
+  ansible-playbook -f 4 -l bench rustfs-manage.yml --tags config
+  ansible-playbook -f 4 -l bench rustfs-manage.yml --tags binary-copy
+  ansible-playbook -f 4 -l bench rustfs-manage.yml --tags start
+'
+```
+
+For the quick A/B rig use the `HOTPATH_AB_*` names.
+
+## Quick A/B (`run_hotpath_warp_ab.sh`)
+
+```bash
+# build both binaries (baseline from --baseline-ref, default origin/main) and run a throwaway single-node server on local disks
 scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main
-# or with prebuilt binaries:
-scripts/run_hotpath_warp_ab.sh --skip-build \
-  --baseline-bin ./rustfs-main --candidate-bin ./target/release/rustfs
+
+# prebuilt binaries
+scripts/run_hotpath_warp_ab.sh --skip-build --baseline-bin ./rustfs-main --candidate-bin ./target/release/rustfs
+
+# print the plan only
+scripts/run_hotpath_warp_ab.sh --dry-run --skip-build --baseline-bin /tmp/base --candidate-bin /tmp/cand
+
+# external cluster
+scripts/run_hotpath_warp_ab.sh --endpoint "$CLUSTER_ENDPOINT" --deploy-hook '<see above>' \
+  --baseline-bin /path/to/rustfs-main --candidate-bin ./target/x86_64-unknown-linux-gnu/release/rustfs
 ```
 
-Preview the full plan without running anything:
+Outputs under `target/hotpath-ab/<ts>/`: `gate.md` (ends with a **Provenance** section: baseline/candidate SHAs, binary source, runner, warp version, matrix params; extend with `--provenance-note`) and `server-logs/<phase>-sync-<sync>.{log,env}`. On a health-check failure the rig prints the last 50 server log lines; in local mode it fails fast if the server process exits before becoming healthy.
+
+## ABBA evidence run (`run_hotpath_warp_abba.sh`)
+
+`B1` and `B2` are compared with `A1` for the candidate delta; `A2` is compared with `A1` for baseline drift. Required flags: `--baseline-bin`, `--candidate-bin`, `--baseline-revision`, `--candidate-revision`. The script enforces `--rounds >= 3`; prefer `--rounds 5` or more for formal evidence when budget allows.
 
 ```bash
-scripts/run_hotpath_warp_ab.sh --dry-run --skip-build \
-  --baseline-bin /tmp/base --candidate-bin /tmp/cand
+# local Linux runner (throwaway data root; a reused run namespace is rejected)
+scripts/run_hotpath_warp_abba.sh \
+  --baseline-bin /tmp/rustfs-baseline --candidate-bin /tmp/rustfs-candidate \
+  --baseline-revision "$(git rev-parse origin/main)" --candidate-revision "$(git rev-parse HEAD)" \
+  --address 127.0.0.1:9000 --data-root /var/tmp/rustfs-hotpath-abba --disks 4 \
+  --duration 120s --rounds 3 --cooldown 30 --concurrency 16 \
+  --out-dir target/hotpath-abba/linux-local
+
+# production-like cluster
+scripts/run_hotpath_warp_abba.sh \
+  --baseline-bin /srv/rustfs-binaries/rustfs-baseline --candidate-bin /srv/rustfs-binaries/rustfs-candidate \
+  --baseline-revision <sha> --candidate-revision <sha> \
+  --endpoint rustfs-bench.example.internal:9000 --deploy-hook '<see above>' \
+  --duration 180s --rounds 5 --cooldown 45 --concurrency 32 \
+  --out-dir target/hotpath-abba/cluster-pr-XXXX
 ```
 
-## External mode (real cluster, ansible-deployed)
+Add `--dry-run` to print the schedule without starting servers or warp. Output layout:
 
-For the production-representative run, warp targets an already-running cluster
-and a `--deploy-hook` swaps in each phase's binary and durability config
-between the baseline and candidate phases. The hook receives context via the
-environment:
+```text
+<out-dir>/
+  manifest.env
+  abba_schedule.csv
+  candidate_gate.md
+  baseline_drift_gate.md
+  summary.md
+  <workload>/<sync>/<leg>/median_summary.csv
+  <workload>/<sync>/<leg>/baseline_compare.csv
+```
 
-- `HOTPATH_AB_PHASE` — `baseline` or `candidate`
-- `HOTPATH_AB_BINARY` — binary path (or empty; the hook may build its own)
-- `HOTPATH_AB_DRIVE_SYNC` — `true` or `false` for this matrix cell
+Attach to the PR or issue: `summary.md`, `candidate_gate.md`, `baseline_drift_gate.md`, `abba_schedule.csv`, every `median_summary.csv` and `baseline_compare.csv` for a failed or borderline workload, and the host telemetry used to explain saturation. Preserve the output directory unmodified.
 
-This maps directly onto the team's ansible harness. Build the candidate with
-the cross toolchain, stage both binaries, then let the hook drive
-`rustfs-manage.yml`:
+### Interpretation
+
+| Candidate gate | A2 drift gate | Interpretation |
+| --- | --- | --- |
+| PASS | PASS | Candidate acceptable for the measured matrix. |
+| WARN | PASS | Small measurable signal; inspect telemetry and decide whether it is expected. |
+| FAIL | PASS | Candidate likely regressed the workload; investigate before merge. |
+| FAIL | FAIL on the same workload | Environment drift is high; rerun on a quieter runner or raise duration and rounds. |
+| PASS | FAIL | Rig unstable; do not quote the numbers as proof of improvement. |
+
+Rules that override the table: a candidate result is actionable only when the `A2` drift for the same workload passes or is materially smaller than the `B1`/`B2` delta; never report a win or loss for a workload whose drift gate failed without a rerun. When `B1` and `B2` disagree, the cell is inconclusive even if the gate passes. Report only measured facts: deltas, drift, saturation, failed workloads.
+
+### CPU and memory evidence
+
+Warp output says whether throughput or latency changed; host telemetry says why. Collect it for the whole run and stop the collectors after the script exits.
+
+| Tool | Command | Answers |
+| --- | --- | --- |
+| `pidstat` | `pidstat -durh 5 > <out-dir>/telemetry/pidstat.txt &` | per-process CPU, memory, disk |
+| `mpstat` | `mpstat 5 > <out-dir>/telemetry/mpstat.txt &` | CPU saturation and steal |
+| `iostat` | `iostat -xz 5 > <out-dir>/telemetry/iostat.txt &` | device queue depth and latency |
+| `perf` | `perf record -F 99 -g -- sleep 180` around one representative cell, then `perf report --stdio` | CPU attribution after the gate shows an effect |
+
+samply against a running RustFS process goes through the bounded helper, one attach window per leg or focused cell:
 
 ```bash
-# 1. Build the candidate (cross-compile for the cluster target).
-cargo zigbuild --release --target x86_64-unknown-linux-gnu -p rustfs --bins
-
-# 2. Run the A/B against the cluster; the hook deploys the phase's binary and
-#    applies the drive-sync config, then restarts, before each phase.
-scripts/run_hotpath_warp_ab.sh \
-  --endpoint "$CLUSTER_ENDPOINT" \
-  --deploy-hook '
-    set -euo pipefail
-    cd /home/xiaomage/xiaomage/ansible
-    # Select the phase binary and the drive-sync value for this cell.
-    cp "${HOTPATH_AB_BINARY:?}" ./roles/rustfs/files/rustfs
-    export RUSTFS_DRIVE_SYNC_ENABLE="$HOTPATH_AB_DRIVE_SYNC"
-    ansible-playbook -f 4 -l testing rustfs-manage.yml --tags stop
-    ansible-playbook -f 4 -l testing rustfs-manage.yml --tags config
-    ansible-playbook -f 4 -l testing rustfs-manage.yml --tags binary-copy
-    ansible-playbook -f 4 -l testing rustfs-manage.yml --tags start
-  ' \
-  --baseline-bin /path/to/rustfs-main \
-  --candidate-bin ./target/x86_64-unknown-linux-gnu/release/rustfs
+scripts/run_samply_attach_window.sh --pid "$RUSTFS_PID" --duration-secs 180 \
+  --output <out-dir>/telemetry/samply-A1-get-4mib.json.gz
 ```
 
-The `config` tag is responsible for threading `RUSTFS_DRIVE_SYNC_ENABLE` (or
-the finer `RUSTFS_DURABILITY_MODE`) into the deployed unit — the hook exports
-it so the config template can pick it up. The rig itself never restarts the
-cluster; lifecycle stays with ansible.
+After each window confirm the `.json.gz` profile and its `.syms.json` sidecar are non-empty, no `samply` process is still attached to the PID, and any temporary `perf_event_paranoid` change is restored; reject the cell otherwise.
 
-## Budget and exemptions
+Instrumented builds (features in `rustfs/Cargo.toml`): `--features hotpath-alloc` for allocation attribution, `--features hotpath-cpu` for CPU hotpath sections. Compare instrumented binaries only with other builds of the same mode; never use them for throughput acceptance, because the instrumentation changes what is measured.
 
-Default budget: a metric regressing more than **10%** vs baseline fails,
-more than **5%** warns. Tune with `--fail-pct` / `--warn-pct`.
+## CI gate (`performance-ab.yml`)
 
-Some regressions are the correct trade — #4221 deliberately paid a large write
-cost to restore power-loss durability. For those, run with
-`--allow-regression` (or add the `perf-deliberate-tradeoff` label in CI): the
-FAIL is recorded and rendered as an exempted WARN, and the gate exits 0.
+| Aspect | Value |
+| --- | --- |
+| Triggers | `schedule` (nightly cron `31 6 * * *` UTC against `main`) and `workflow_dispatch` (inputs `duration`, default `12s`; `allow_regression`, boolean). No `pull_request` trigger and no label gating. |
+| Jobs | `warp-ab` (runner `sm-standard-2`, `timeout-minutes: 180`); `alert-on-failure` (opens the scheduled-failure issue; scheduled runs only). |
+| Baseline commit | scheduled: head of the last successful scheduled run (falls back to the candidate itself when there is none); dispatch: `origin/main`. The baseline must be an ancestor of the candidate. |
+| Binary cache | `actions/cache` inside `warp-ab`, key `rustfs-baseline-<baseline_sha>`. Miss: source build (same-commit runs build once and reuse the binary for both phases); a run whose gate passes saves the candidate binary under `rustfs-baseline-<candidate_sha>` for the next night. |
+| Command | `scripts/run_hotpath_warp_abba.sh --duration <input> --rounds 3 --cooldown 5 --health-timeout 180 --baseline-revision <sha> --candidate-revision <sha> --baseline-bin ... --candidate-bin ...` |
+| Exemption | `allow_regression=true` on dispatch adds `--allow-regression --exemption-reason "workflow dispatch override"`. |
+| Artifacts | `hotpath-warp-ab-<run_number>` containing `target/hotpath-abba/` (14-day retention); the step summary renders `candidate_gate.md`, or the server-log tails when the rig failed before the gate. The `Enforce gate` step fails the job on a non-zero rig exit. |
 
-## Diagnosing a failed run
-
-Each phase's server log and its startup environment are written under the run's
-output dir (`target/hotpath-ab/<ts>/server-logs/<phase>-sync-<sync>.{log,env}`)
-and uploaded in the `hotpath-warp-ab-<run>` artifact, so a failure is
-diagnosable after the fact. On a health-check failure the rig also dumps the
-last 50 log lines into the job log and the CI job writes the failing phase (or
-the gate table) into the GitHub step summary.
-
-Readiness polling waits up to `--health-timeout` seconds (default **180**),
-which must outlast the server's own startup-readiness budget
-(`RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS`, default 120s) — a shorter poll on a
-slow shared runner misreports a slow cold start as a failure. In local mode the
-rig also fails fast if the server process exits before becoming healthy instead
-of polling out the full budget.
-
-## Scope note
-
-The gate logic is unit-validated across pass/warn/fail/exempt outcomes; the
-orchestrator and workflow are shellcheck- and `--dry-run`-validated. The first
-real warp measurement belongs on a Linux runner or the ansible cluster — there
-is no warp/multi-disk rig in the repo's local checkout.
-
-This warp A/B gate is the **only** entry point for S3-face (PutObject /
-GetObject / ListObjects) performance coverage. There is deliberately no
-in-process criterion benchmark for those operations: a criterion harness that
-stands up an embedded server measures the harness, not the S3 path, so it would
-report a number without guarding anything. Micro-benchmarks stay at the
-function level (EC encode, `xl.meta` parse, `rename_data`; perf-8).
+The nightly detects a regression within a day of landing; it does not block a merge. For pre-merge evidence run the ABBA procedure above and attach the outputs to the PR.

docs/operations/hotpath-warp-abba-runbook.md

@@ -1,296 +0,0 @@
-# Hotpath warp ABBA validation runbook
-
-This runbook describes how to collect formal Linux or production-cluster
-evidence for hotpath performance changes. Use it when a short local A/B smoke
-run is too noisy to decide whether a regression is real.
-
-The ABBA runner executes each workload and drive-sync cell as:
-
-```text
-A1 baseline -> B1 candidate -> B2 candidate -> A2 baseline
-```
-
-`B1` and `B2` are compared with `A1` to measure the candidate delta. `A2` is
-also compared with `A1` to measure baseline drift. Treat a candidate regression
-as actionable only when the `A2` drift is passing or materially smaller than
-the `B1` and `B2` delta for the same workload.
-
-## Scope
-
-Use this runbook for hotpath profiling and performance validation of RustFS
-object I/O changes, especially when CPU, memory allocation, lock/channel wait
-time, request throughput, or tail latency is the review question.
-
-The script validates the same workload matrix as the hotpath warp A/B gate:
-
-| Workload | mode | size |
-| --- | --- | --- |
-| `put-4kib` | put | 4KiB |
-| `put-4mib` | put | 4MiB |
-| `get-4kib` | get | 4KiB |
-| `get-4mib` | get | 4MiB |
-| `get-10mib` | get | 10MiB |
-| `mixed-256k` | mixed | 256KiB |
-
-Each workload runs with `RUSTFS_DRIVE_SYNC_ENABLE=true` and
-`RUSTFS_DRIVE_SYNC_ENABLE=false`, so a full ABBA pass produces 48 measurement
-cells: 6 workloads x 2 drive-sync modes x 4 ABBA legs.
-
-## Prerequisites
-
-Run the formal pass on Linux, not on a laptop smoke environment.
-
-Required tools on the bench host:
-
-- `bash`, `curl`, `git`, and core GNU userland.
-- `warp` on `PATH`, or pass `--warp-bin`.
-- Two RustFS Linux binaries: one baseline and one candidate.
-- Enough isolated disks or directories for the local runner, or an externally
-  managed RustFS cluster for production-like validation.
-- Stable host telemetry collection such as `pidstat`, `mpstat`, `iostat`,
-  `sar`, `perf`, `heaptrack`, or the platform's equivalent observability stack.
-
-Cluster-mode requirements:
-
-- A deploy hook that can replace the RustFS binary on every node.
-- The hook must apply `RUSTFS_DRIVE_SYNC_ENABLE` for the current ABBA leg.
-- The hook must restart RustFS and return only after the rollout command has
-  been accepted. The ABBA script performs the HTTP readiness wait.
-- The benchmark client should run outside the RustFS nodes when possible.
-- Do not run against a production data set unless the workload bucket and test
-  credentials are isolated and approved for destructive benchmark traffic.
-
-## Build the binaries
-
-Build the baseline from the comparison commit, usually `origin/main` or the
-previous accepted release:
-
-```bash
-git fetch origin main
-git switch --detach origin/main
-cargo build --release -p rustfs --bins
-cp target/release/rustfs /tmp/rustfs-baseline
-```
-
-Build the candidate from the PR commit:
-
-```bash
-git switch <candidate-branch>
-cargo build --release -p rustfs --bins
-cp target/release/rustfs /tmp/rustfs-candidate
-```
-
-For cross-compiled cluster binaries, keep both outputs on the bench host and
-make the deploy hook copy the selected binary to the cluster. The ABBA runner
-passes the selected binary path through `HOTPATH_ABBA_BINARY`.
-
-## Local Linux runner
-
-Use local mode for a dedicated Linux runner with disposable data paths. This is
-not a substitute for a production-like cluster, but it is useful before spending
-cluster time.
-
-```bash
-scripts/run_hotpath_warp_abba.sh \
-  --baseline-bin /tmp/rustfs-baseline \
-  --candidate-bin /tmp/rustfs-candidate \
-  --address 127.0.0.1:9000 \
-  --data-root /var/tmp/rustfs-hotpath-abba \
-  --disks 4 \
-  --duration 120s \
-  --rounds 3 \
-  --cooldown 30 \
-  --concurrency 16 \
-  --out-dir target/hotpath-abba/linux-local
-```
-
-The script starts and stops RustFS for each ABBA leg. The data root is
-throwaway and should not contain important data.
-
-## Production-like cluster runner
-
-Use external mode when RustFS lifecycle is managed by ansible, systemd, a
-cluster scheduler, or a dedicated deployment harness. In this mode the ABBA
-script does not start RustFS directly; it calls `--deploy-hook` before each leg
-and then waits for `http://<endpoint><health-path>`.
-
-The deploy hook receives:
-
-| Environment variable | Value |
-| --- | --- |
-| `HOTPATH_ABBA_LEG` | `A1`, `B1`, `B2`, or `A2` |
-| `HOTPATH_ABBA_PHASE` | `baseline` or `candidate` |
-| `HOTPATH_ABBA_BINARY` | selected baseline or candidate binary path |
-| `HOTPATH_ABBA_DRIVE_SYNC` | `true` or `false` |
-
-Example ansible-shaped command:
-
-```bash
-scripts/run_hotpath_warp_abba.sh \
-  --baseline-bin /srv/rustfs-binaries/rustfs-baseline \
-  --candidate-bin /srv/rustfs-binaries/rustfs-candidate \
-  --endpoint rustfs-bench.example.internal:9000 \
-  --deploy-hook '
-    set -euo pipefail
-    cd /srv/rustfs-ansible
-    cp "${HOTPATH_ABBA_BINARY:?}" roles/rustfs/files/rustfs
-    export RUSTFS_DRIVE_SYNC_ENABLE="${HOTPATH_ABBA_DRIVE_SYNC:?}"
-    ansible-playbook -f 4 -l bench rustfs-manage.yml --tags stop
-    ansible-playbook -f 4 -l bench rustfs-manage.yml --tags config
-    ansible-playbook -f 4 -l bench rustfs-manage.yml --tags binary-copy
-    ansible-playbook -f 4 -l bench rustfs-manage.yml --tags start
-  ' \
-  --duration 180s \
-  --rounds 5 \
-  --cooldown 45 \
-  --concurrency 32 \
-  --out-dir target/hotpath-abba/cluster-pr-XXXX
-```
-
-For formal evidence, prefer `--rounds 5` or higher when the cluster budget
-allows it. The script enforces `--rounds >= 3`.
-
-## CPU and memory evidence
-
-ABBA warp output answers whether the candidate changed throughput or latency.
-Collect host telemetry at the same time to explain why.
-
-Recommended minimum:
-
-```bash
-mkdir -p target/hotpath-abba/cluster-pr-XXXX/telemetry
-
-pidstat -durh 5 > target/hotpath-abba/cluster-pr-XXXX/telemetry/pidstat.txt &
-PIDSTAT_PID=$!
-
-mpstat 5 > target/hotpath-abba/cluster-pr-XXXX/telemetry/mpstat.txt &
-MPSTAT_PID=$!
-
-iostat -xz 5 > target/hotpath-abba/cluster-pr-XXXX/telemetry/iostat.txt &
-IOSTAT_PID=$!
-```
-
-Stop the collectors after the ABBA script exits:
-
-```bash
-kill "$PIDSTAT_PID" "$MPSTAT_PID" "$IOSTAT_PID"
-```
-
-For deeper CPU attribution, run `perf record` around one representative
-workload after the ABBA gate identifies a candidate regression or improvement:
-
-```bash
-perf record -F 99 -g -- sleep 180
-perf report --stdio > target/hotpath-abba/cluster-pr-XXXX/telemetry/perf-report.txt
-```
-
-When using samply against an already-running RustFS service, attach through the
-bounded helper instead of calling `samply record -p` directly:
-
-```bash
-scripts/run_samply_attach_window.sh \
-  --pid "$RUSTFS_PID" \
-  --duration-secs 180 \
-  --output target/hotpath-abba/cluster-pr-XXXX/telemetry/samply-A1-get-4mib.json.gz
-```
-
-Run one attach window per ABBA leg or focused verification cell. After every
-window, confirm that the `.json.gz` profile and `.syms.json` sidecar are
-non-empty, that no `samply` process is still attached to the RustFS PID, and
-that any temporary `perf_event_paranoid` change has been restored before the
-next cell starts.
-
-For allocation profiling, build the candidate with:
-
-```bash
-cargo build --release -p rustfs --bins --features hotpath-alloc
-```
-
-Then run the same ABBA command with that binary. Compare allocation-heavy
-function sections only within the same build mode. Do not compare
-`hotpath-alloc` binaries directly with default release binaries for throughput
-acceptance, because allocation instrumentation intentionally changes what is
-measured.
-
-For CPU hotpath sections emitted by hotpath, build with:
-
-```bash
-cargo build --release -p rustfs --bins --features hotpath-cpu
-```
-
-Use the CPU-enabled report to explain hotspots after the default or plain
-`hotpath` ABBA gate shows a real effect.
-
-## Output layout
-
-The ABBA runner writes:
-
-```text
-<out-dir>/
-  manifest.env
-  abba_schedule.csv
-  candidate_gate.md
-  baseline_drift_gate.md
-  summary.md
-  <workload>/<sync>/<leg>/median_summary.csv
-  <workload>/<sync>/<leg>/baseline_compare.csv
-```
-
-Attach or link at least these files in the issue or PR:
-
-- `summary.md`
-- `candidate_gate.md`
-- `baseline_drift_gate.md`
-- `abba_schedule.csv`
-- every `median_summary.csv` and `baseline_compare.csv` for a failed or
-  borderline workload
-- host telemetry files used to explain CPU, memory, or disk saturation
-
-## Interpretation
-
-Use this decision table:
-
-| Candidate gate | A2 drift gate | Interpretation |
-| --- | --- | --- |
-| PASS | PASS | Candidate is acceptable for the measured matrix. |
-| WARN | PASS | Candidate has a small measurable signal; inspect telemetry and decide if it is expected. |
-| FAIL | PASS | Candidate likely regressed the affected workload; investigate before merge. |
-| FAIL | FAIL on the same workload | Environment drift is high; rerun on a quieter runner or increase duration and rounds. |
-| PASS | FAIL | Candidate did not exceed the budget, but the rig was unstable; avoid using the numbers as proof of improvement. |
-
-When `B1` and `B2` disagree, treat the result as inconclusive even if the gate
-passes. Increase duration, rounds, cooldown, or runner isolation before drawing
-a conclusion.
-
-## AI execution checklist
-
-When delegating the run to an AI agent or an automation runner, provide these
-inputs explicitly:
-
-- repository checkout and candidate branch or commit;
-- baseline commit or binary path;
-- candidate binary path;
-- runner type: local Linux or external cluster;
-- endpoint, access key, secret key source, and region;
-- deploy hook path or exact command for cluster mode;
-- output directory;
-- required duration, rounds, cooldown, concurrency, and fail/warn budgets;
-- where to upload artifacts after the run.
-
-The AI agent should execute this sequence:
-
-1. Confirm `uname -a`, RustFS commits, binary SHA256 sums, `warp --version`,
-   CPU model, memory size, disk layout, and whether the run is local or cluster.
-2. Run `scripts/run_hotpath_warp_abba.sh --dry-run` with the final arguments.
-3. Run the real ABBA command with `--rounds >= 3`.
-4. For samply CPU attribution, use `scripts/run_samply_attach_window.sh` for
-   each bounded attach window and reject the cell if the profile is empty or a
-   stale `samply` process remains.
-5. Preserve the full output directory without editing generated CSV files.
-6. Read `summary.md`, `candidate_gate.md`, and `baseline_drift_gate.md`.
-7. Summarize only measured facts: candidate deltas, baseline drift, CPU or
-   memory saturation, and any failed workloads.
-8. Post the summary and artifact location to the tracking issue or PR.
-
-Do not report a performance win or loss when the baseline drift gate failed on
-the same workload and no rerun was collected.

docs/operations/internode-grpc-benchmark-runbook.md

@@ -1,146 +1,94 @@
-# Internode gRPC Optimization — A/B Benchmark Runbook
+# Internode gRPC A/B benchmark runbook
 
-Reproducible procedure to collect **before/after** artifacts for each internode gRPC
-optimization stage (grpc-optimization P0–P3). Every stage is env-gated, so "before" and
-"after" are the *same binary* with different env — no rebuild between runs.
+**Use this when:** collecting before/after evidence for an env-gated internode gRPC transport stage (P0 transport tuning, P1 channel isolation, P2 msgpack-only codec, P3 prewarm/offline bypass) on a real multi-node cluster.
+**Source of truth:** `scripts/run_internode_grpc_ab_bench.sh` (stage/phase driver), `crates/config/src/constants/internode.rs` (`ENV_INTERNODE_*` / `DEFAULT_INTERNODE_*`), `crates/io-metrics/src/internode_metrics.rs` (metric names), [internode-msgpack-json-convergence-runbook.md](internode-msgpack-json-convergence-runbook.md) (the P2 gate).
 
-> Live runs need a multi-node cluster (Docker or ≥2 rustfs endpoints), a load tool
-> (`warp` or `s3bench`), and a Prometheus scrape of `/metrics`. They are not runnable in a
-> single-process sandbox. Capture artifacts on a real cluster.
+Every stage is env-gated, so "before" and "after" run the *same binary* with different server env; no rebuild between runs. Live runs need a multi-node cluster (Docker compose or two or more endpoints), a load tool (`warp` or `s3bench`), and a metrics sink; they are not runnable in a single-process sandbox.
 
-## One-click driver
+## Prerequisites
 
-`scripts/run_internode_grpc_ab_bench.sh --stage <p0|p1|p2|p3> --phase <before|after|request-only|canary|rollback> [-- <bench args>]`
-wraps the env matrix below: it writes the stage/phase **server** env to
-`<out-dir>/server-env.sh`, then runs the right underlying bench into
-`target/bench/internode-transport/<stage>-<phase>/`.
+| Requirement | Detail |
+| --- | --- |
+| RPC secret | Internode RPC fails closed: remote endpoints with default credentials and no `RUSTFS_RPC_SECRET` abort startup with `store init aborted: endpoints include remote nodes but ...` (`crates/ecstore/src/store/init.rs`). Set a non-default `RUSTFS_RPC_SECRET`, identical on every node. |
+| systemd start timeout | `deploy/build/rustfs.service` is `Type=notify` and ships `TimeoutStartSec=120s`; READY fires only after quorum. If freshly purged disks need longer, raise it in a drop-in rather than lowering it. |
+| Metrics export | RustFS has no Prometheus pull endpoint (`/admin/v3/metrics` is NDJSON, see `rustfs/src/admin/handlers/metrics.rs`); it pushes OTLP. Run an otel-collector (OTLP receiver → Prometheus exporter) and set `RUSTFS_OBS_ENDPOINT`, `RUSTFS_OBS_METRICS_EXPORT_ENABLED=true`, `RUSTFS_OBS_METER_INTERVAL=5`. For lock p99 also set `RUSTFS_OBJECT_LOCK_DIAG_ENABLE=true` (default off). |
+| Server env | `RUSTFS_INTERNODE_*` are **server** env. For p0/p1/p2, source the emitted `server-env.sh` on every node and restart before the run; the driver cannot mutate a running server. |
+
+## Driver
 
 ```bash
-# P1 A/B (restart the cluster with each phase's server-env.sh between the two runs):
-scripts/run_internode_grpc_ab_bench.sh --stage p1 --phase before -- --access-key AK --secret-key SK --metrics-url http://node1:9000/metrics
-scripts/run_internode_grpc_ab_bench.sh --stage p1 --phase after  -- --access-key AK --secret-key SK --metrics-url http://node1:9000/metrics
-# P3 failover A/B (docker four-node):
-scripts/run_internode_grpc_ab_bench.sh --stage p3 --phase after
-# P2 rollout gates (env preview for request-only rehearsal, canary, and rollback):
-scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase request-only --dry-run
-scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase canary --dry-run
-scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase rollback --dry-run
+scripts/run_internode_grpc_ab_bench.sh --stage <p0|p1|p2|p3> --phase <before|after|request-only|canary|rollback> [--dry-run] [-- <bench args>]
 ```
 
-`RUSTFS_INTERNODE_*` are **server** env: for the load-driven stages (p0/p1/p2) source the emitted `server-env.sh` on every node and restart rustfs *before* the run — the driver cannot mutate an already-running server. The P2 `canary` phase is the exception: source it only on the selected canary node after the release-window counters and fleet-support checks pass, and keep the rest of the fleet on `before` or `request-only` while observing fallback/decode-error counters. Use `--dry-run` to preview the env and command.
+The driver writes the stage/phase server env to `<out-dir>/server-env.sh` and runs the underlying bench into `target/bench/internode-transport/<stage>-<phase>/`. `p0/p1/p3` accept `before|after`; `p2` also accepts `request-only|canary|rollback`. `--dry-run` prints the env and command only.
 
-## Harness
+```bash
+# P1 A/B: restart the cluster with each phase's server-env.sh between the two runs
+scripts/run_internode_grpc_ab_bench.sh --stage p1 --phase before -- --access-key AK --secret-key SK --metrics-url http://node1:9000/metrics
+scripts/run_internode_grpc_ab_bench.sh --stage p1 --phase after  -- --access-key AK --secret-key SK --metrics-url http://node1:9000/metrics
+# P3 failover A/B (docker four-node)
+scripts/run_internode_grpc_ab_bench.sh --stage p3 --phase after
+# P2 env previews
+scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase canary --dry-run
+```
 
-- Throughput / latency: `scripts/run_internode_transport_baseline.sh` (drives
-  `run_object_batch_bench.sh`; writes `target/bench/internode-transport-<ts>/`). Pass
-  `--metrics-url <prometheus>` to also capture internode metric deltas.
-- Failover / offline: `scripts/run_four_node_cluster_failover_bench.sh` (spins up a 4-node
-  compose cluster, kills `FAILOVER_NODE`, benchmarks; writes
-  `target/bench/four-node-failover-<ts>/`).
-
-The one-click driver writes each run to `target/bench/internode-transport/<stage>-<phase>/`
-(e.g. `p0-before/`, `p0-after/`, `p1-before/`, `p1-after/`, `p3-before/`, `p3-after/`), each
-containing the emitted `server-env.sh` plus the underlying bench artifacts. `target/` is
-gitignored — attach the paired directories to the PR / issue.
-
-## Metrics to capture (Prometheus)
-
-| Metric | Stage signal |
-|---|---|
-| `rustfs_system_network_internode_operation_duration_ms{operation,backend}` | control-plane RTT (P0), lock/bulk latency |
-| `rustfs_system_network_internode_operation_payload_bytes` | payload size distribution (P0/P1 sizing) |
-| `rustfs_system_network_internode_operation_large_payloads_total` | large unary RPCs sharing the channel (P1 target) |
-| `rustfs_system_network_internode_dial_avg_time_nanos`, `..._dial_errors_total` | connect cost (P3 prewarm) |
-| `rustfs_system_network_internode_msgpack_json_decode_total{direction,message,codec}` | must be **>0** for each expected P2 series before zero fallback/error readings are meaningful |
-| `rustfs_system_network_internode_msgpack_json_fallback_total{direction,message}` | must be **0** before enabling both msgpack-only gates (P2) |
-| `rustfs_system_network_internode_msgpack_json_decode_error_total{direction,message,codec}` | must be **0** before enabling both msgpack-only gates (P2) |
-| `rustfs_cluster_servers_offline_total` | offline detection correctness (P3 bypass) |
-| lock p99 (lock metrics) | P1 head-of-line-blocking win |
+Underlying benches: `scripts/run_internode_transport_baseline.sh` (throughput/latency via `scripts/run_object_batch_bench.sh`; `--metrics-url <prometheus>` also captures internode metric deltas) and `scripts/run_four_node_cluster_failover_bench.sh` (4-node compose cluster, kills `FAILOVER_NODE`). `target/` is gitignored; attach the paired directories to the PR or issue.
 
 ## Per-stage env matrix
 
-Run **before** with the stage's env at its baseline column, **after** with the enabled
-column, everything else at defaults. Roll a restart between runs.
+Run **before** at the baseline column and **after** at the enabled column, everything else at defaults, with a restart between. Defaults are the `DEFAULT_INTERNODE_*` constants in `crates/config/src/constants/internode.rs`.
 
-| Stage | Env | before (baseline) | after (enabled) |
-|---|---|---|---|
-| P0 nodelay | `RUSTFS_INTERNODE_RPC_TCP_NODELAY` | `false` | `true` (default) |
-| P0 stream window | `RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE` | `0` | unset (1 MiB) |
-| P0 conn window | `RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE` | `0` | unset (2 MiB) |
-| P0 msg limit | `RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE` | `4194304` | unset (100 MiB) |
+| Stage | Env | before | after |
+| --- | --- | --- | --- |
+| P0 nodelay | `RUSTFS_INTERNODE_RPC_TCP_NODELAY` | `false` | unset (`DEFAULT_INTERNODE_RPC_TCP_NODELAY`) |
+| P0 stream window | `RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE` | `0` | unset (`DEFAULT_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE`) |
+| P0 conn window | `RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE` | `0` | unset (`DEFAULT_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE`) |
+| P0 msg limit | `RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE` | `4194304` (tonic default) | unset (RustFS default, see `rustfs/src/server/http.rs`) |
 | P1 isolation | `RUSTFS_INTERNODE_CHANNEL_ISOLATION` | `false` (default) | `true` |
-| P1 bulk pool | `RUSTFS_INTERNODE_BULK_CHANNELS` | `1` | `2`–`4` |
-| P2 msgpack-only request | `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY` | `false` (default) | `true` (only after fallback counter = 0 across a window) |
-| P2 fleet confirmation | `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` | `false` (default) | `true` (only after mixed-version, fallback-zero, soak, and rollback gates pass) |
+| P1 bulk pool | `RUSTFS_INTERNODE_BULK_CHANNELS` | `1` | unset (`DEFAULT_INTERNODE_BULK_CHANNELS`) or higher |
+| P2 msgpack-only | `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY` + `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` | both `false` (default) | per the convergence runbook |
 | P3 prewarm | `RUSTFS_INTERNODE_PREWARM` | `false` (default) | `true` |
 | P3 offline bypass | `RUSTFS_INTERNODE_OFFLINE_BYPASS` | `false` (default) | `true` |
-| P3 reprobe / threshold | `RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS` / `RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD` | defaults | `5` / `3` |
+| P3 reprobe / threshold | `RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS` / `RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD` | defaults | defaults (`DEFAULT_INTERNODE_OFFLINE_REPROBE_SECS`, `DEFAULT_INTERNODE_OFFLINE_FAILURE_THRESHOLD`) |
 
-## Procedure per stage
+## Metrics to capture
 
-1. **Baseline**: start the cluster with the stage's env at the *before* column. Run the
-   relevant bench; save to `.../baseline/` (or `.../after-P{n-1}/` when chaining stages).
-2. **After**: restart with the *after* column; re-run the identical bench; save to
-   `.../after-P{n}/`.
-3. Diff the object-bench summaries and the metric deltas.
+All names are defined in `crates/io-metrics/src/internode_metrics.rs`.
 
-- **P0** — `run_internode_transport_baseline.sh` with `--sizes 4KiB,1MiB,16MiB,128MiB` and
-  `--concurrencies 1,16,64`. Expect: small-RPC `duration_ms` (DiskInfo/Ping) down (nodelay),
-  large-metadata (ReadMultiple/BatchReadVersion) throughput up (windows). Functional: a
-  `>4 MiB` multi-version `xl.meta` no longer fails `out_of_range`.
-- **P1** — mixed workload (large `ReadAll` + high-frequency `Refresh`). Acceptance gate from
-  the design doc: **lock p99 down ≥ 20%** with `RUSTFS_INTERNODE_CHANNEL_ISOLATION=true`.
-- **P2** — observe `msgpack_json_decode_total`, `msgpack_json_fallback_total`, and `msgpack_json_decode_error_total` across a release window; every expected `codec="msgpack"` decode series must be **>0**, while fallback and decode-error series must stay **0** before flipping both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true` (see the msgpack convergence runbook). Codec allocation via a `dhat`/`heaptrack` micro-run.
-- **P3** — cold-start: first cross-node op latency should drop ~one connect RTT with prewarm.
-  Failover: `run_four_node_cluster_failover_bench.sh`, kill a node with
-  `RUSTFS_INTERNODE_OFFLINE_BYPASS=true`; expect faster failover and a correct
-  `rustfs_cluster_servers_offline_total` (1 while the node is down, back to 0 after recovery).
+| Metric | Stage signal |
+| --- | --- |
+| `rustfs_system_network_internode_operation_duration_ms{operation,backend}` | control-plane RTT (P0), lock/bulk latency (P1), first-op latency (P3) |
+| `rustfs_system_network_internode_operation_payload_bytes` | payload size distribution (P0/P1 sizing) |
+| `rustfs_system_network_internode_operation_large_payloads_total` | large unary RPCs sharing a channel (P1 target) |
+| `rustfs_system_network_internode_dial_avg_time_nanos`, `rustfs_system_network_internode_dial_errors_total` | connect cost and failures (P3) |
+| `rustfs_system_network_internode_msgpack_json_decode_total{direction,message,codec}`, `..._msgpack_json_fallback_total`, `..._msgpack_json_decode_error_total` | P2 gate inputs |
+| `rustfs_cluster_servers_offline_total` | offline detection correctness (P3 bypass) |
+| lock p99 (lock metrics, needs `RUSTFS_OBJECT_LOCK_DIAG_ENABLE=true`) | P1 head-of-line-blocking win |
 
-## Acceptance gates & artifact layout
+## Acceptance gates
 
-Each stage's paired run must satisfy an explicit gate before its numbers are accepted. Record
-the gate verdict (pass/fail + measured delta) in the paired directory's `summary` and attach it.
+Record the verdict (pass/fail plus measured delta) in the paired directory's summary.
 
-| Stage | Bench | Acceptance gate | Primary metric(s) |
-|---|---|---|---|
-| **P0** | `run_internode_transport_baseline.sh` | small-RPC `duration_ms` (DiskInfo/Ping) **down**; large-metadata (ReadMultiple/BatchReadVersion) throughput **up**; a `>4 MiB` multi-version `xl.meta` no longer fails `out_of_range` (functional). | `..._operation_duration_ms{operation}`, `..._operation_payload_bytes`, object-bench throughput |
-| **P1** | `run_internode_transport_baseline.sh` (mixed: large `ReadAll` + high-frequency `Refresh`) | **lock p99 down ≥ 20%** with `RUSTFS_INTERNODE_CHANNEL_ISOLATION=true` vs baseline. | lock p99 (lock metrics), `..._operation_large_payloads_total` |
-| **P3 cold-start** | `run_internode_transport_baseline.sh` (fresh cluster, first cross-node op) | first cross-node op latency **drops ~one connect RTT** with `RUSTFS_INTERNODE_PREWARM=true`. | `..._dial_avg_time_nanos`, first-op `..._operation_duration_ms` |
-| **P3 offline** | dedicated *sustained-offline + survivor cross-node access* experiment (the standard four-node failover bench is **not** sensitive to the bypass — quorum holds, `recovery_seconds=0`) | with `RUSTFS_INTERNODE_OFFLINE_BYPASS=true`, survivor cross-node op latency to the downed peer **fast-fails** instead of hanging the dial timeout; `rustfs_cluster_servers_offline_total` = **1** while down, back to **0** after recovery. | `rustfs_cluster_servers_offline_total`, survivor cross-node `..._operation_duration_ms`, `..._dial_errors_total` |
+| Stage | Bench | Gate | Primary metrics |
+| --- | --- | --- | --- |
+| P0 | `run_internode_transport_baseline.sh --sizes 4KiB,1MiB,16MiB,128MiB --concurrencies 1,16,64` | small-RPC `duration_ms` (DiskInfo/Ping) down; large-metadata (ReadMultiple/BatchReadVersion) throughput up; a `>4 MiB` multi-version `xl.meta` no longer fails `out_of_range` | `..._operation_duration_ms{operation}`, `..._operation_payload_bytes`, object-bench throughput |
+| P1 | `run_internode_transport_baseline.sh` with a mixed workload (large `ReadAll` plus high-frequency `Refresh`) | lock p99 down by at least 20% with `RUSTFS_INTERNODE_CHANNEL_ISOLATION=true` | lock p99, `..._operation_large_payloads_total` |
+| P2 | none (not a throughput gate) | operational gate defined once in [internode-msgpack-json-convergence-runbook.md](internode-msgpack-json-convergence-runbook.md): expected `codec="msgpack"` decode series non-zero, fallback and decode-error series zero across a full observation window before both flags are enabled. Optionally a `dhat`/`heaptrack` micro-run for codec allocation. | the three `msgpack_json_*` counters |
+| P3 cold-start | `run_internode_transport_baseline.sh` on a fresh cluster, first cross-node op | first cross-node op latency drops by about one connect RTT with prewarm | `..._dial_avg_time_nanos`, first-op `..._operation_duration_ms` |
+| P3 offline | sustained-offline plus survivor cross-node access (below) | with bypass on, survivor ops to the downed peer fast-fail instead of hanging for the dial timeout; `rustfs_cluster_servers_offline_total` is `1` while down and `0` after recovery | `rustfs_cluster_servers_offline_total`, survivor `..._operation_duration_ms`, `..._dial_errors_total` |
 
-> **P2 is not a throughput gate.** Its acceptance is operational: every expected `msgpack_json_decode_total{codec="msgpack"}` series must have non-zero traffic, while `msgpack_json_fallback_total` and `msgpack_json_decode_error_total` must read **0** across a full release window before both msgpack-only env gates are enabled (see the msgpack convergence runbook). Do not benchmark P2 as before/after throughput.
+P3 offline method: the standard four-node failover bench is not sensitive to the bypass (quorum holds, `recovery_seconds=0`). Instead: all nodes up → stop one node and keep it down → warm up until offline detection trips → drive warp against the survivors only (`--host` excludes the dead node) → compare survivor op p99 and the offline gauge with `RUSTFS_INTERNODE_OFFLINE_BYPASS` off and on.
 
-Artifact layout per stage (attach both halves + the diff):
+Artifact layout:
 
-```
+```text
 target/bench/internode-transport/
-  p0-before/  p0-after/     # server-env.sh + object-bench summaries + metric deltas
-  p1-before/  p1-after/     # + lock p99 delta (the ≥20% gate)
-  p2-before/  p2-request-only/  p2-canary/  p2-after/  p2-rollback/  # env gate artifacts + fallback/decode-error observations
-  p3-before/  p3-after/     # cold-start + sustained-offline experiment + offline gauge trace
+  p0-before/  p0-after/                                            # server-env.sh + object-bench summaries + metric deltas
+  p1-before/  p1-after/                                            # + lock p99 delta
+  p2-before/  p2-request-only/  p2-canary/  p2-after/  p2-rollback/ # env previews + counter observations
+  p3-before/  p3-after/                                            # cold-start + sustained-offline + offline gauge trace
 ```
 
-## Bench-host prerequisites (ansible bare-metal)
-
-Captured while running the first real A/B on a 4-node ansible cluster; needed before any live run:
-
-- **RPC secret is mandatory on current `main`.** Internode RPC fails closed: default creds
-  (`RUSTFS_SECRET_KEY=rustfsadmin`) with no `RUSTFS_RPC_SECRET` → the node aborts startup immediately
-  with `store init aborted: endpoints include remote nodes but RPC authentication secret is not
-  configured` (builds before the preflight instead showed `No valid auth token` and never reached
-  `storage_quorum`). Set a **non-default** `RUSTFS_RPC_SECRET`, identical on every node.
-- **systemd start timeout.** The install unit is `Type=notify`; READY only fires after quorum, which on
-  freshly-purged disks exceeds a 30 s `TimeoutStartSec` → crash loop. Use a drop-in `TimeoutStartSec=infinity`.
-- **Server-side metrics need OTLP.** RustFS has no Prometheus pull endpoint (`/admin/v3/metrics` is NDJSON,
-  not exposition); it only pushes via OTLP. To capture lock/offline/internode metrics, run an
-  otel-collector (OTLP receiver → Prometheus exporter) and set `RUSTFS_OBS_ENDPOINT`,
-  `RUSTFS_OBS_METRICS_EXPORT_ENABLED=true`, `RUSTFS_OBS_METER_INTERVAL=5`. For lock p99 also set
-  `RUSTFS_OBJECT_LOCK_DIAG_ENABLE=true` (default off).
-- **P3 offline method.** Standard failover is quorum-insensitive to the bypass. Use a *sustained-offline +
-  survivor cross-node access* run instead: all nodes up → stop one node (sustained) → warm up to trip
-  offline detection → drive warp on the survivors only (`--host` excludes the dead node) → compare
-  survivor op p99 and `rustfs_cluster_servers_offline_total` for `RUSTFS_INTERNODE_OFFLINE_BYPASS` off/on.
-
 ## Rollback
 
-Every stage rolls back by setting the env back to its baseline column and restarting. P2 rolls back by unsetting either msgpack-only gate, or by setting both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=false` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false`; no wire format is broken because the JSON fields, `_bin` fields, and proto field numbers remain additive. Do not remove or reuse JSON proto fields as part of this benchmark stage.
+Every stage rolls back by restoring the baseline column and restarting. P2 rollback and its wire-format guarantees are in the [convergence runbook's rollback matrix](internode-msgpack-json-convergence-runbook.md#rollback-matrix); do not remove or reuse JSON proto fields as part of any benchmark stage.

docs/operations/internode-msgpack-json-convergence-runbook.md

@@ -1,75 +1,37 @@
-# Internode msgpack/JSON Convergence Runbook
+# Internode msgpack/JSON convergence runbook
 
-Operational runbook for retiring the redundant JSON compatibility fields on internode
-gRPC metadata RPCs (grpc-optimization **P2-1**). This is a **cross-version** change: it
-proceeds strictly by observation-gated stages, never in one step.
+**Use this when:** operating or changing the staged retirement of the JSON compatibility fields on internode gRPC metadata RPCs, deciding whether a fleet may enable `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY`, or adding a new `*_bin` proto field.
+**Source of truth:** `crates/config/src/constants/internode.rs` (`ENV_INTERNODE_RPC_MSGPACK_ONLY`, `ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` and their compile-time default-off asserts), `crates/protos/src/node.proto` (`*_bin` fields), `decode_msgpack_or_json` in `rustfs/src/storage/rpc/node_service/disk.rs` (server side) and `crates/ecstore/src/cluster/rpc/remote_disk.rs` (client side), `crates/io-metrics/src/internode_metrics.rs` (counters).
+
+This is a cross-version change and proceeds by observation-gated stages, never in one step.
 
 ## Background
 
-Internode RPCs dual-encode each metadata value as **both**:
+Internode RPCs dual-encode each metadata value as a msgpack binary field (`*_bin`, for example `file_info_bin`) and a JSON compatibility string (for example `file_info`). Decoders prefer `_bin` and fall back to JSON only when `_bin` is empty. The dual-write costs bandwidth and CPU; before the JSON fields can be dropped, the fallback branch must be proven unused across the fleet, otherwise a mixed-version rolling upgrade could read an emptied field.
 
-- a msgpack binary field (`*_bin`, e.g. `file_info_bin`), and
-- a JSON compatibility string (e.g. `file_info`).
+## Observation counters
 
-Decoders prefer the `_bin` payload and fall back to the JSON string only when `_bin` is
-empty (`decode_msgpack_or_json`). The dual-write costs bandwidth and CPU. Before the JSON
-fields can be dropped, the fallback branch must be proven **unused** in production —
-otherwise a rolling upgrade with mixed node versions could read an emptied field.
+| Counter | Labels | Increments when |
+| --- | --- | --- |
+| `rustfs_system_network_internode_msgpack_json_decode_total` | `direction`, `message`, `codec` | a msgpack or JSON decode succeeded |
+| `rustfs_system_network_internode_msgpack_json_fallback_total` | `direction`, `message` | a decode fell back to JSON because `_bin` was empty |
+| `rustfs_system_network_internode_msgpack_json_decode_error_total` | `direction`, `message`, `codec` | either codec failed to decode (`codec` = the failed codec) |
 
-## The observation metric (already shipped)
+`direction="request"` is a server decoding a peer's request (`node_service/disk.rs`); `direction="response"` is a client decoding a peer's response (`cluster/rpc/remote_disk.rs`), including the list-level `ReadMultiple` / `BatchReadVersion` fallbacks. `message` is the value name (`FileInfo`, `RawFileInfo`, `ReadMultipleResp`, ...).
 
-```
-rustfs_system_network_internode_msgpack_json_fallback_total{direction, message}
-rustfs_system_network_internode_msgpack_json_decode_total{direction, message, codec}
-rustfs_system_network_internode_msgpack_json_decode_error_total{direction, message, codec}
-```
-
-The decode counter increments after a successful msgpack or JSON compatibility decode. The fallback counter increments whenever a decode falls back to the JSON field because the msgpack payload was absent. The decode-error counter increments whenever either codec fails to decode.
-
-- `direction="request"` — a server decoding a peer's request (`node_service/disk.rs`).
-- `direction="response"` — a client decoding a peer's response (`cluster/rpc/remote_disk.rs`),
-  including the list-level `ReadMultiple` / `BatchReadVersion` fallbacks.
-- `message` — the value name, e.g. `FileInfo`, `RawFileInfo`, `ReadMultipleResp`.
-- `codec` — the failed codec for decode errors: `msgpack` for corrupt non-empty `_bin`, or `json` for corrupt legacy fallback JSON.
-
-## Stage 0 — Observe (current stage)
-
-Ship the current release (which contains these counters) and let it run for **at least one
-full release window** across the whole fleet. The fallback and decode-error counters must stay at **zero**.
-
-First confirm each expected message/direction has real traffic in the observation window:
+Gate query template, run over the whole observation window (adjust `[30d]`):
 
 ```promql
-sum by (direction, message, codec) (
-  increase(rustfs_system_network_internode_msgpack_json_decode_total[30d])
-)
+sum by (direction, message, codec) (increase(<counter>[30d]))
 ```
 
-For every convergence-ready message/direction below, the `codec="msgpack"` series must be non-zero before a zero fallback result is meaningful. Missing series, zero traffic, counter reset, or scrape gaps make the gate inconclusive rather than passed.
+| Counter | Required reading | Meaning of a violation |
+| --- | --- | --- |
+| `..._decode_total` | every convergence-ready `{direction, message}` has a non-zero `codec="msgpack"` series | no traffic, a counter reset, or scrape gaps make the gate inconclusive, not passed |
+| `..._fallback_total` | `0` for every series | some peer still sends an empty `_bin` (old node, or a sender that does not fill `_bin`); investigate the labels |
+| `..._decode_error_total` | `0` for every series | `codec="msgpack"`: corrupt or incompatible `_bin` bytes; `codec="json"`: corrupt legacy fallback. Either blocks convergence and rollback confidence |
 
-Confirm zero across the observation window (adjust `[30d]` to the window length):
-
-```promql
-sum by (direction, message) (
-  increase(rustfs_system_network_internode_msgpack_json_fallback_total[30d])
-)
-```
-
-Every series must be `0`. A non-zero value means some peer is still emitting an empty
-`_bin` (an old node, or a message whose sender does not fill `_bin`) — investigate the
-`{direction, message}` label before proceeding.
-
-Decode errors must also stay at zero across the observation window:
-
-```promql
-sum by (direction, message, codec) (
-  increase(rustfs_system_network_internode_msgpack_json_decode_error_total[30d])
-)
-```
-
-A non-zero `codec="msgpack"` series means a peer sent corrupt or incompatible `_bin` bytes; it must fail closed and block convergence. A non-zero `codec="json"` series means the legacy fallback field was corrupt or semantically incompatible; it also blocks convergence and rollback confidence.
-
-Standing alert (keep enabled through all stages):
+Standing alerts (keep enabled through every stage):
 
 ```yaml
 - alert: InternodeMsgpackJsonFallback
@@ -90,96 +52,70 @@ Standing alert (keep enabled through all stages):
 
 ## Field → peer-decoder audit
 
-The send-side change (Stage 1) may only empty a JSON field whose **peer decodes `_bin`
-first**. The following mapping is verified against the current code.
+Stage 1 may only empty a JSON field whose peer decodes `_bin` first. Any `*_bin` field not listed here must be mapped to a confirmed `_bin`-first peer decoder before it joins the convergence set.
 
-### Convergence-ready (peer decodes `_bin` first)
+Convergence-ready:
 
 | Direction | Message / field | Peer decoder |
-|---|---|---|
-| request | `WriteMetadata.file_info` | `FileInfo` (node_service/disk.rs) |
+| --- | --- | --- |
+| request | `WriteMetadata.file_info` | `FileInfo` (`node_service/disk.rs`) |
 | request | `UpdateMetadata.file_info` | `FileInfo` |
 | request | `UpdateMetadata.opts` | `UpdateMetadataOpts` |
 | request | `RenameData.file_info` | `FileInfo` |
 | request | `ReadMultiple.read_multiple_req` | `ReadMultipleReq` |
 | request | `BatchReadVersion.batch_read_version_req` | `BatchReadVersionReq` |
 | request | `Read*.opts` | `ReadOptions` |
-| response | `ReadVersion.file_info` | `FileInfo` (cluster/rpc/remote_disk.rs) |
+| response | `ReadVersion.file_info` | `FileInfo` (`cluster/rpc/remote_disk.rs`) |
 | response | `ReadXL.raw_file_info` | `RawFileInfo` |
 | response | `RenameData.rename_data_resp` | `RenameDataResp` |
-| response | `ReadMultiple` resp list | per-item + list fallback |
-| response | `BatchReadVersion` resp list | per-item + list fallback |
+| response | `ReadMultiple` response list | per-item plus list fallback |
+| response | `BatchReadVersion` response list | per-item plus list fallback |
 
-### `_bin` support added THIS release — converge after their own window
-
-The `DeleteVersion`/`DeleteVersions` protos had **no `_bin` fields**. They gained additive
-`*_bin` fields plus bin-first server decoders in this release, and the client now dual-writes
-them. They are **kept out** of the msgpack-only set (always dual-write) until their own
-fallback counter has read zero across a window with the new decoders fully deployed.
+Delete messages, kept on dual-write until their own window reads zero with `_bin`-first decoders deployed fleet-wide (the client always dual-writes these regardless of the flags):
 
 | Direction | Message / field | Status |
-|---|---|---|
-| request | `DeleteVersion.file_info` (`FileInfo`) | `_bin` added; dual-write; converge after window |
-| request | `DeleteVersion.opts` (`DeleteOptions`) | `_bin` added; dual-write; converge after window |
-| request | `DeleteVersions.versions` (`FileInfoVersions`) | `_bin` added; dual-write; converge after window |
-| request | `DeleteVersions.opts` (`DeleteOptions`) | `_bin` added; dual-write; converge after window |
+| --- | --- | --- |
+| request | `DeleteVersion.file_info` (`FileInfo`) | `_bin` present; dual-write; converge after its own window |
+| request | `DeleteVersion.opts` (`DeleteOptions`) | same |
+| request | `DeleteVersions.versions` (`FileInfoVersions`) | same |
+| request | `DeleteVersions.opts` (`DeleteOptions`) | same |
 
-> Any `*_bin` proto field not in the tables above must be mapped to a confirmed `_bin`-first
-> peer decoder before it is added to the convergence set.
-
-### Still JSON-only (no `_bin` field)
+Still JSON-only:
 
 | Direction | Message / field | Note |
-|---|---|---|
-| response | `DeleteVersion.raw_file_info` | proto has no `_bin`; needs an additive proto field before it can converge. |
+| --- | --- | --- |
+| response | `DeleteVersion.raw_file_info` | proto has no `_bin`; needs an additive proto field before it can converge |
 
-## Stage 1 — Stop writing JSON (env-gated, after Stage 0 reads zero)
+## Stage 0 — Observe
 
-The send-side lever is **implemented** and requires two default-off env flags:
+Run the fleet with both flags at their default (`false`) for at least one full observation window and evaluate the three gate rows above. All three must hold before Stage 1.
 
-- `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true`
-- `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true`
+## Stage 1 — Stop writing JSON (env-gated)
 
-The first flag only requests msgpack-only. The second flag is the explicit proof gate that the fleet has passed the release-window fallback, capability, and rollback checks. If only `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` is set, RustFS keeps dual-writing JSON compatibility fields so old JSON-only peers remain compatible. When both flags are enabled, the convergence-ready fields above send only `_bin` and leave the JSON string empty; the `_bin` payload is always sent and decoders keep the JSON read fallback unchanged. The delete fields are excluded (dual-write) per the section above.
+Flag semantics are defined on `ENV_INTERNODE_RPC_MSGPACK_ONLY` and `ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` in `crates/config/src/constants/internode.rs`; both default off and only their conjunction stops the JSON write for convergence-ready fields. Decoders keep the JSON fallback in every state.
 
-Only enable it **after** Stage 0 has read zero for a full window across the fleet:
+1. Rehearse `request-only`: `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true`, `..._FLEET_CONFIRMED=false`. Behaviour is unchanged (still dual-write) and safe with old peers; any fallback or decode-error increment still blocks.
+2. Canary: set both flags `true` on one node, restart that node only, and watch the fallback and decode-error counters for a soak period with real internode traffic.
+3. Fleet: if the counters stay zero, enable both flags fleet-wide with a rolling restart.
+4. Rollback: set either flag to `false` (or unset it) and restart; no wire format changed, so rollback is immediate.
 
-1. Ship with the flag **off** (no behavior change).
-2. Rehearse the `request-only` gate with `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false`. This must keep writing JSON compatibility fields and is safe with unsupported/old peers; any fallback or decode-error increment still blocks convergence.
-3. Enable it on one canary node with both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=true`, then restart that node only and watch the fallback and decode-error counters for a soak period while real internode traffic is present. If they stay zero, enable fleet-wide.
-4. **Rollback:** set either `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=false` or `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false` (or unset either flag) and restart. No
-   wire-format was broken in this stage, so rollback is immediate and safe.
+The benchmark driver pins these operator states as dry-run phases (`before`, `request-only`, `canary`, `after`, `rollback`); see [internode-grpc-benchmark-runbook.md](internode-grpc-benchmark-runbook.md).
 
-The benchmark helper pins these operator states as dry-run phases:
+## Stage 2 — Remove the proto JSON fields
 
-```bash
-scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase before --dry-run
-scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase request-only --dry-run
-scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase canary --dry-run
-scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase after --dry-run
-scripts/run_internode_grpc_ab_bench.sh --stage p2 --phase rollback --dry-run
-```
+Only after Stage 1 has been stable fleet-wide for a full window with the counters still zero.
 
-## Stage 2 — Remove the proto JSON fields (next release, N+1)
-
-Only after Stage 1 has been stable with the flag on for a full window and the counter is
-still zero.
-
-1. Mark the retired text fields `reserved` in `crates/protos/src/node.proto` (never reuse
-   the field numbers) and delete the JSON read-fallback branches; codec becomes msgpack-only.
-2. This is a hard wire-format change — it requires the mixed-version upgrade rehearsal
-   (four-node scripts) to pass, and it cannot be rolled back by env alone.
+1. Mark the retired text fields `reserved` in `crates/protos/src/node.proto` (never reuse field numbers) and delete the JSON read-fallback branches.
+2. This is a hard wire-format change: it requires the mixed-version upgrade rehearsal (four-node scripts) to pass and cannot be rolled back by env alone.
 
 ## Rollback matrix
 
-| Stage | Wire-format broken? | Rollback |
-|---|---|---|
-| 0 Observe | no | n/a (metric only) |
-| 1 msgpack-only send | no | unset either msgpack-only env flag + restart |
-| 2 remove fields | yes | redeploy prior release; field numbers stay `reserved` |
+| Stage | Wire format broken? | Rollback |
+| --- | --- | --- |
+| 0 Observe | no | n/a (metrics only) |
+| 1 msgpack-only send | no | unset either flag and restart |
+| 2 remove fields | yes | redeploy the prior release; field numbers stay `reserved` |
 
 ## Related
 
-- Codec + counter implementation: commit `feat(internode): P2 msgpack/JSON codec observability + encode buffer presizing`.
-- Decoders: `decode_msgpack_or_json` in `crates/ecstore/src/cluster/rpc/remote_disk.rs` (client)
-  and `rustfs/src/storage/rpc/node_service/disk.rs` (server).
+- Transport and codec observability landed in `feat(internode): optimize gRPC transport (#4337)`.

docs/operations/keycloak-oidc-integration.md

@@ -1,305 +0,0 @@
-# Keycloak OIDC Integration Runbook
-
-This runbook describes how to connect the RustFS Console to Keycloak by using OpenID Connect Authorization Code Flow. The examples use the default RustFS provider id, `default`.
-
-## 1. Integration Model
-
-RustFS supports standard OpenID Connect for Console login:
-
-- RustFS sends an authorization-code request with PKCE S256.
-- Keycloak redirects back with `code` and `state`.
-- RustFS exchanges the code at the token endpoint.
-- RustFS requires an `id_token` and verifies signature, issuer, audience, expiry, and nonce.
-- RustFS maps ID token claim values to local RustFS IAM policies.
-
-RustFS does not call Keycloak Authorization Services for object or admin authorization. Authorization is handled by RustFS policies after claims are mapped.
-
-## 2. Example Values
-
-Replace these values for your environment:
-
-| Value | Example | Notes |
-| --- | --- | --- |
-| Keycloak base URL | `https://keycloak.example.com` | Public Keycloak URL. |
-| Realm | `rustfs` | Keycloak realm name. |
-| Keycloak issuer | `https://keycloak.example.com/realms/rustfs` | RustFS `config_url`. |
-| Discovery URL | `https://keycloak.example.com/realms/rustfs/.well-known/openid-configuration` | Used to validate metadata. |
-| Public RustFS browser origin | `https://rustfs.example.com` | The scheme and authority users open in the browser. |
-| Provider id | `default` | This runbook uses the default provider. |
-| RustFS callback URL | `https://rustfs.example.com/rustfs/admin/v3/oidc/callback/default` | Register this exact URL in Keycloak. |
-| Keycloak client id | `rustfs-console` | OIDC client used by RustFS. |
-| Keycloak client secret | `<KEYCLOAK_CLIENT_SECRET>` | Confidential client secret. |
-| RustFS scopes | `openid,profile,email` | Add custom scopes if they emit authorization claims. |
-| RustFS groups claim | `groups` | Recommended flat array claim. |
-| RustFS roles claim | `roles` | Optional flat array claim. |
-
-## 3. Keycloak Configuration
-
-### 3.1 Create or Select the Realm
-
-1. Open the Keycloak Admin Console.
-2. Create or select the `rustfs` realm.
-3. Verify discovery:
-
-```bash
-curl -fsS "https://keycloak.example.com/realms/rustfs/.well-known/openid-configuration" \
-  | jq '.issuer,.authorization_endpoint,.token_endpoint,.jwks_uri'
-```
-
-The `issuer` should be:
-
-```text
-https://keycloak.example.com/realms/rustfs
-```
-
-### 3.2 Create the RustFS Client
-
-In the Keycloak Admin Console:
-
-1. Open `Clients` and create a client.
-2. Set `Client type` to `OpenID Connect`.
-3. Set `Client ID` to `rustfs-console`.
-4. Enable `Client authentication`.
-5. Enable `Standard flow`.
-6. Disable unused flows such as `Implicit flow`, `Direct access grants`, and `Service accounts roles`.
-7. Set `Valid redirect URIs` to:
-
-```text
-https://rustfs.example.com/rustfs/admin/v3/oidc/callback/default
-```
-
-8. Set `Web origins` to:
-
-```text
-https://rustfs.example.com
-```
-
-9. Set `Proof Key for Code Exchange Code Challenge Method` to `S256`.
-10. Save and copy the client secret from `Credentials`.
-
-RustFS submits the client secret in the token request body. Do not use a client policy that disables `client_secret_post`.
-
-### 3.3 Map Groups or Roles to RustFS Policies
-
-RustFS policy names are the final authorization source. Common built-in policies are:
-
-| Policy | Purpose |
-| --- | --- |
-| `consoleAdmin` | Full Console, admin, KMS, and S3 access. |
-| `readwrite` | S3 read/write access. |
-| `readonly` | S3 read-only access. |
-| `writeonly` | S3 write-only access. |
-| `diagnostics` | Diagnostic admin access. |
-
-Recommended production setup:
-
-1. Create Keycloak groups such as `consoleAdmin` and `readonly`.
-2. Add users to the groups.
-3. Add a group membership mapper for the `rustfs-console` client.
-4. Emit a flat top-level ID token claim named `groups`.
-5. Keep group values equal to RustFS policy names.
-
-### 3.4 Group Claim Mapper
-
-Create a `Group Membership` mapper in the dedicated client scope:
-
-| Mapper field | Value |
-| --- | --- |
-| Name | `rustfs-groups` |
-| Token Claim Name | `groups` |
-| Full group path | `Off` |
-| Add to ID token | `On` |
-| Add to access token | `On` |
-| Add to userinfo | `On` |
-| Multivalued | `On` |
-
-Keep `Full group path` disabled. RustFS policy names cannot contain `/`, so `/consoleAdmin` will not map to the `consoleAdmin` policy.
-
-### 3.5 Optional Role Claim Mapper
-
-If the deployment uses Keycloak roles:
-
-1. Assign realm or client roles such as `consoleAdmin`.
-2. Add a `User Realm Role` or `User Client Role` mapper.
-3. Emit a flat top-level claim named `roles`.
-4. Set `RUSTFS_IDENTITY_OPENID_ROLES_CLAIM=roles`.
-
-RustFS does not parse Keycloak's default nested `realm_access.roles` claim. Emit a flat `roles` array when role mapping is required.
-
-## 4. RustFS Configuration
-
-### 4.1 Environment Variables
-
-Configure the provider and the public browser origin:
-
-```bash
-export RUSTFS_BROWSER_REDIRECT_URL="https://rustfs.example.com"
-
-export RUSTFS_IDENTITY_OPENID_ENABLE=on
-export RUSTFS_IDENTITY_OPENID_CONFIG_URL="https://keycloak.example.com/realms/rustfs"
-export RUSTFS_IDENTITY_OPENID_CLIENT_ID="rustfs-console"
-export RUSTFS_IDENTITY_OPENID_CLIENT_SECRET="<KEYCLOAK_CLIENT_SECRET>"
-export RUSTFS_IDENTITY_OPENID_SCOPES="openid,profile,email"
-export RUSTFS_IDENTITY_OPENID_REDIRECT_URI="https://rustfs.example.com/rustfs/admin/v3/oidc/callback/default"
-export RUSTFS_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC=off
-export RUSTFS_IDENTITY_OPENID_DISPLAY_NAME="Keycloak"
-export RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM="groups"
-export RUSTFS_IDENTITY_OPENID_ROLES_CLAIM="roles"
-export RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM="email"
-export RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM="preferred_username"
-```
-
-If RustFS reaches Keycloak through an internal URL while tokens use a public issuer, configure both values:
-
-```bash
-export RUSTFS_IDENTITY_OPENID_CONFIG_URL="http://keycloak.keycloak.svc.cluster.local:8080/realms/rustfs/.well-known/openid-configuration"
-export RUSTFS_IDENTITY_OPENID_ISSUER="https://keycloak.example.com/realms/rustfs"
-export RUSTFS_OUTBOUND_ALLOW_ORIGINS="http://keycloak.keycloak.svc.cluster.local:8080"
-```
-
-Discovery and issuer-relative JWKS requests use the internal `CONFIG_URL` base. ID token issuer validation still uses `ISSUER`.
-The outbound allowlist entry is the exact internal origin only; do not include the realm or discovery path. RustFS reads this process setting at startup, so restart every RustFS node after changing it.
-Use HTTPS with a trusted CA for the internal URL whenever possible. Discovery and JWKS define the token-signing trust root; use HTTP only on a network where DNS and traffic cannot be tampered with, because a compromised response can authorize forged tokens.
-
-For short-lived connectivity testing only, you may temporarily add:
-
-```bash
-export RUSTFS_IDENTITY_OPENID_ROLE_POLICY="consoleAdmin"
-```
-
-Do not use the temporary `role_policy` shortcut as a permanent production authorization model.
-
-Restart RustFS after changing OIDC settings.
-
-### 4.2 Admin Config
-
-If the deployment uses compatible admin configuration commands:
-
-```bash
-mc admin config set rustfs identity_openid \
-  enable=on \
-  config_url="https://keycloak.example.com/realms/rustfs" \
-  client_id="rustfs-console" \
-  client_secret="<KEYCLOAK_CLIENT_SECRET>" \
-  scopes="openid,profile,email" \
-  redirect_uri="https://rustfs.example.com/rustfs/admin/v3/oidc/callback/default" \
-  redirect_uri_dynamic=off \
-  display_name="Keycloak" \
-  groups_claim="groups" \
-  roles_claim="roles" \
-  email_claim="email" \
-  username_claim="preferred_username"
-
-mc admin service restart rustfs
-```
-
-`RUSTFS_BROWSER_REDIRECT_URL` is a process environment variable, not an `identity_openid` provider key. Configure it in the RustFS service environment even when the provider is stored through admin config.
-
-### 4.3 Named Provider
-
-To use a provider id such as `keycloak`, register this callback URL in Keycloak:
-
-```text
-https://rustfs.example.com/rustfs/admin/v3/oidc/callback/keycloak
-```
-
-Then suffix the provider-specific environment variables:
-
-```bash
-export RUSTFS_IDENTITY_OPENID_ENABLE_keycloak=on
-export RUSTFS_IDENTITY_OPENID_CONFIG_URL_keycloak="https://keycloak.example.com/realms/rustfs"
-export RUSTFS_IDENTITY_OPENID_CLIENT_ID_keycloak="rustfs-console"
-export RUSTFS_IDENTITY_OPENID_CLIENT_SECRET_keycloak="<KEYCLOAK_CLIENT_SECRET>"
-export RUSTFS_IDENTITY_OPENID_SCOPES_keycloak="openid,profile,email"
-export RUSTFS_IDENTITY_OPENID_REDIRECT_URI_keycloak="https://rustfs.example.com/rustfs/admin/v3/oidc/callback/keycloak"
-export RUSTFS_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC_keycloak=off
-export RUSTFS_IDENTITY_OPENID_DISPLAY_NAME_keycloak="Keycloak"
-export RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM_keycloak="groups"
-```
-
-`RUSTFS_BROWSER_REDIRECT_URL` remains global and is not suffixed per provider.
-
-### 4.4 Redirect URL Priority
-
-RustFS builds browser-facing URLs with this priority:
-
-1. Provider `redirect_uri`, when configured, is used for the OIDC callback URL sent to Keycloak.
-2. `RUSTFS_BROWSER_REDIRECT_URL`, when configured, is used as the public origin for OIDC callback generation when no provider `redirect_uri` exists, and for Console success redirects and logout fallback redirects.
-3. Request headers are used only when provider dynamic redirects are enabled and no browser redirect URL is configured.
-
-For reverse-proxy or load-balancer deployments, set `RUSTFS_BROWSER_REDIRECT_URL` to avoid depending on `Host` and `X-Forwarded-Proto` for Console redirects. OIDC authorize and callback requests must still reach the same RustFS node because in-flight OIDC `state` is local to the node.
-
-## 5. Validation
-
-### 5.1 Validate Discovery
-
-```bash
-curl -fsS "https://keycloak.example.com/realms/rustfs/.well-known/openid-configuration" | jq '{
-  issuer,
-  authorization_endpoint,
-  token_endpoint,
-  jwks_uri,
-  code_challenge_methods_supported,
-  token_endpoint_auth_methods_supported
-}'
-```
-
-Check that:
-
-- `issuer` equals `RUSTFS_IDENTITY_OPENID_ISSUER` when set; otherwise it matches the issuer derived from `RUSTFS_IDENTITY_OPENID_CONFIG_URL`
-- `authorization_endpoint`, `token_endpoint`, and `jwks_uri` are present
-- `code_challenge_methods_supported` includes `S256`
-- the token endpoint accepts client secret authentication compatible with request-body submission
-
-### 5.2 Validate ID Token Claims
-
-After a test login, decode the ID token and confirm:
-
-- `iss` matches the Keycloak issuer
-- `aud` includes the RustFS client id
-- `email` and `preferred_username` are present when configured
-- `groups` or `roles` contains RustFS policy names if fine-grained authorization is enabled
-
-### 5.3 Test Browser Login
-
-Open:
-
-```text
-https://rustfs.example.com/rustfs/admin/v3/oidc/authorize/default
-```
-
-Expected flow:
-
-1. Browser redirects to Keycloak.
-2. The user signs in.
-3. Keycloak redirects to `/rustfs/admin/v3/oidc/callback/default?code=...&state=...`.
-4. RustFS validates the ID token and issues STS credentials.
-5. The browser lands on the RustFS Console and can use the expected permissions.
-
-## 6. Troubleshooting
-
-| Symptom | Common cause | Fix |
-| --- | --- | --- |
-| Keycloak reports `invalid redirect_uri` | Valid Redirect URIs does not match the RustFS callback URL | Use the exact callback URL and provider id. |
-| Callback reports missing `code` or `state` | Proxy dropped the query string | Preserve the full callback URL and query string. |
-| Token exchange fails | Client secret or client authentication policy mismatch | Confirm the client is confidential and accepts request-body secret auth. |
-| RustFS reports no `id_token` | Missing `openid` scope or disabled Standard Flow | Include `openid` and enable Standard Flow. |
-| ID token verification fails | Issuer, client id, audience, or JWKS mismatch | Compare discovery metadata and client settings. |
-| Login succeeds but access is denied | No RustFS policy claim was mapped | Emit `groups` or `roles` as a flat ID token claim matching RustFS policy names. |
-| Groups appear as `/consoleAdmin` | Keycloak `Full group path` is enabled | Disable `Full group path`. |
-| Console redirects to an internal host | Missing `RUSTFS_BROWSER_REDIRECT_URL` or incorrect proxy headers | Set `RUSTFS_BROWSER_REDIRECT_URL` to the public browser origin. |
-| Invalid or expired OIDC state | Callback reached a different RustFS node | Configure load-balancer session affinity for authorize and callback requests. |
-| OIDC provider or login button is missing after upgrading to beta.12+ | The internal Keycloak origin is blocked by the outbound policy | Add the exact `scheme://host:port` origin to `RUSTFS_OUTBOUND_ALLOW_ORIGINS` and restart every RustFS node. |
-
-## 7. Production Checklist
-
-- [ ] Keycloak and RustFS use HTTPS.
-- [ ] Keycloak Valid Redirect URIs uses exact callback URLs.
-- [ ] `RUSTFS_BROWSER_REDIRECT_URL` is set to the public RustFS browser origin.
-- [ ] `RUSTFS_IDENTITY_OPENID_REDIRECT_URI` matches the registered Keycloak callback URL.
-- [ ] PKCE S256 is enabled or required.
-- [ ] Users receive `groups` or `roles` claims that match RustFS policy names.
-- [ ] `role_policy=consoleAdmin` is not used as a permanent production shortcut.
-- [ ] The load balancer preserves query strings.
-- [ ] OIDC authorize and callback requests have session affinity to the same RustFS node.
-- [ ] Internal Keycloak origins are listed exactly in `RUSTFS_OUTBOUND_ALLOW_ORIGINS` on every RustFS node.

docs/operations/kms-admin-contract.md

@@ -1,68 +1,56 @@
-# KMS admin API contract and client handoff
+# KMS admin API contract
 
-This page is the server-side handoff for rustfs/backlog#1639. Response-shape snapshots for the key and metadata handlers live beside the producers under `rustfs/src/admin/handlers/snapshots/` (PR #5626). This matrix records the remaining route-level handoff contract without duplicating those snapshots. The management `KmsStatusResponse` shape remains an identified gap and will be pinned after the status-handler changes in #1636 land.
+**Use this when:** you are wiring a client (CLI, console, automation) to the KMS admin endpoints and need the IAM action, risk class, per-key scope, and key-listing paging rules for each route.
+**Source of truth:** `rustfs/src/admin/route_policy.rs` (action and risk per route, asserted by `rustfs/src/admin/route_registration_test.rs`); `crates/kms/src/backends/mod.rs` (`DEFAULT_LIST_KEYS_PAGE_SIZE`, `MAX_LIST_KEYS_PAGE_SIZE`, `list_keys_page_size`); response shapes pinned by snapshots under `crates/kms/src/snapshots/` and `rustfs/src/admin/handlers/snapshots/`.
 
-The wire prefix is `/rustfs/admin/v3`. Request and response field names for the producer snapshots are pinned in #5626; fields in this matrix are the client handoff reference. `GET /kms/status` and `GET /kms/service-status` intentionally use different response types; `capabilities` on `/kms/status` is additive and optional.
+The wire prefix is `/rustfs/admin/v3`. `GET /kms/status` and `GET /kms/service-status` return different response types; `capabilities` on `/kms/status` is additive and optional. The **Per-key** column says whether the route authorizes against the key it names (see [Per-key KMS authorization](kms-per-key-authorization.md)); `no` means the route matches any KMS resource in the caller's policy.
 
-| Method and endpoint | Action / risk | Per-key | rc | console | Handoff |
-| --- | --- | ---: | --- | --- | --- |
-| `POST /kms/configure` | `kms:Configure` / high | no | supported | supported | none |
-| `POST /kms/reconfigure` | `kms:Configure` / high | no | supported | supported | none |
-| `POST /kms/start` | `kms:ServiceControl` / high | no | supported | supported | none |
-| `POST /kms/stop` | `kms:ServiceControl` / high | no | supported | supported | none |
-| `POST /kms/reload` | `kms:ServiceControl` / high | no | pending | pending | Re-reads the cluster-persisted configuration without resubmitting secrets; response reuses the configure shape. |
-| `GET /kms/config` | `kms:Configure` / sensitive | no | no | supported | Redact operational paths before display. |
-| `POST /kms/clear-cache` | `kms:ClearCache` / high | no | no | supported | Keep the current `{status,message}` response stable. |
-| `POST /kms/keys` | `kms:Configure` / high | no | supported | supported | none |
-| `GET /kms/keys` | `kms:ListKeys` / sensitive | no | supported | supported | none |
-| `GET /kms/keys/{key_id}` | `kms:DescribeKey` / sensitive | yes | supported | supported | none |
-| `DELETE /kms/keys/delete` | `kms:DeleteKey` / critical | yes | supported | supported | Preserve immediate-delete confirmations. |
-| `POST /kms/keys/cancel-deletion` | `kms:DeleteKey` / high | yes | supported | supported | none |
-| `POST /kms/create-key` | `kms:Configure` / high | no | no | no | Legacy `mc` alias; do not add a second client command. |
-| `POST /kms/key/create` | `kms:Configure` / high | no | no | no | Legacy `mc` alias; do not add a second client command. |
-| `GET /kms/describe-key` | `kms:DescribeKey` / sensitive | yes | no | no | Legacy `mc` alias. |
-| `GET /kms/key/status` | `kms:DescribeKey` / sensitive | yes | supported | no | `rc key status` uses this legacy-compatible shape. |
-| `GET /kms/list-keys` | `kms:ListKeys` / sensitive | no | supported | no | `rc key list` uses this legacy-compatible shape. |
-| `POST /kms/generate-data-key` | `kms:GenerateDataKey` / high | yes | do not expose | do not expose | Programmatic primitive; never print plaintext key material. |
-| `GET /kms/status` | `kms:ServiceControl` / sensitive | no | supported | supported | Keep `capabilities` optional for old servers. |
-| `POST /kms/status` | `kms:ServiceControl` / high | no | no | no | Internal compatibility route; not a client command. |
-| `GET /kms/service-status` | `kms:ServiceControl` / sensitive | no | no | supported | Do not conflate this type with `/kms/status`. |
-| `POST /kms/keys/enable` | `kms:EnableKey` / high | yes | pending | pending | Add an explicit lifecycle command/UI action. |
-| `POST /kms/keys/disable` | `kms:DisableKey` / high | yes | pending | pending | Add an explicit lifecycle command/UI action. |
-| `POST /kms/keys/rotate` | `kms:RotateKey` / high | yes | pending | pending | Add an explicit lifecycle command/UI action. |
-| `POST /kms/keys/update-description` | `kms:UpdateKeyDescription` / high | yes | pending | pending | Add a metadata mutation command/UI action. |
-| `POST /kms/keys/tag` | `kms:TagResource` / high | yes | pending | pending | Add a metadata mutation command/UI action. |
-| `POST /kms/keys/untag` | `kms:UntagResource` / high | yes | pending | pending | Add a metadata mutation command/UI action. |
-| `GET /kms/backup` | `kms:Backup` / sensitive | no | pending | pending | Status/readiness only; never expose KEK material. |
-| `POST /kms/backup` | `kms:Backup` / high | no | pending | pending | Preserve `backup_id` and metadata-only response. |
-| `POST /kms/restore/dry-run` | `kms:Restore` / sensitive | no | pending | pending | Dry-run must be the default and show differences. |
-| `POST /kms/restore` | `kms:Restore` / high | no | pending | pending | Require `confirm_backup_id` and `confirm_conflict_policy`; no blanket `--yes`. |
-| `POST /kms/restore/abort` | `kms:Restore` / high | no | pending | pending | Require `confirm_target_key_dir`. |
+## Endpoint matrix
+
+| Method and endpoint | IAM action | Risk | Per-key | Notes |
+| --- | --- | --- | --- | --- |
+| `POST /kms/configure` | `kms:Configure` | high | no | Persists to cluster storage, switches the local node, broadcasts a best-effort peer reload |
+| `POST /kms/reconfigure` | `kms:Configure` | high | no | Same contract as configure |
+| `POST /kms/start` | `kms:ServiceControl` | high | no | |
+| `POST /kms/stop` | `kms:ServiceControl` | high | no | |
+| `POST /kms/reload` | `kms:ServiceControl` | high | no | Re-reads the persisted configuration without resubmitting secrets; reuses the configure response shape |
+| `GET /kms/status` | `kms:ServiceControl` | sensitive | no | Backend type plus capability matrix |
+| `POST /kms/status` | `kms:ServiceControl` | high | no | Compatibility route; not a client command |
+| `GET /kms/service-status` | `kms:ServiceControl` | sensitive | no | Carries `cluster_config` fingerprints and the `consistent` flag |
+| `GET /kms/config` | `kms:Configure` | sensitive | no | Contains operational paths; redact before display |
+| `POST /kms/clear-cache` | `kms:ClearCache` | high | no | `KmsClearCacheResponse` (`{status,message}`) |
+| `POST /kms/keys` | `kms:Configure` | high | no | Key creation shares the configure action |
+| `GET /kms/keys` | `kms:ListKeys` | sensitive | no | See the key listing contract below |
+| `GET /kms/keys/{key_id}` | `kms:DescribeKey` | sensitive | yes | `?impact=true` opts into the configuration-reference report |
+| `DELETE /kms/keys/delete` | `kms:DeleteKey` | critical | yes | JSON body; `force_immediate` also requires `confirm_key_id` and the server-side `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION` gate |
+| `POST /kms/keys/cancel-deletion` | `kms:DeleteKey` | high | yes | |
+| `POST /kms/keys/enable` | `kms:EnableKey` | high | yes | |
+| `POST /kms/keys/disable` | `kms:DisableKey` | high | yes | |
+| `POST /kms/keys/rotate` | `kms:RotateKey` | high | yes | Subject to the rotation constraints in [KMS backend security properties](kms-backend-security.md#master-key-rotation-retention-destruction-and-upgrade-ordering) |
+| `POST /kms/keys/rekey` | `kms:Rekey` | high | no | Bulk DEK rekey sweep; cluster-scoped, see [`kms-bulk-rekey-contract.md`](../architecture/kms-bulk-rekey-contract.md) |
+| `GET /kms/keys/rekey/status` | `kms:Rekey` | sensitive | no | |
+| `POST /kms/keys/rekey/cancel` | `kms:Rekey` | high | no | |
+| `POST /kms/keys/update-description` | `kms:UpdateKeyDescription` | high | yes | |
+| `POST /kms/keys/tag` | `kms:TagResource` | high | yes | |
+| `POST /kms/keys/untag` | `kms:UntagResource` | high | yes | |
+| `POST /kms/generate-data-key` | `kms:GenerateDataKey` | high | yes | Response carries a base64 plaintext data key; never surface it in a UI or CLI |
+| `GET /kms/backup` | `kms:Backup` | sensitive | no | Status and readiness only; no KEK material |
+| `POST /kms/backup` | `kms:Backup` | high | no | Returns `backup_id` and metadata only |
+| `POST /kms/restore/dry-run` | `kms:Restore` | sensitive | no | Preflight; writes nothing |
+| `POST /kms/restore` | `kms:Restore` | high | no | Requires `confirm_backup_id` and `confirm_conflict_policy` |
+| `POST /kms/restore/abort` | `kms:Restore` | high | no | Requires `confirm_target_key_dir` |
+| `POST /kms/create-key`, `POST /kms/key/create` | `kms:Configure` | high | no | Legacy `mc` aliases of `POST /kms/keys` |
+| `GET /kms/describe-key`, `GET /kms/key/status` | `kms:DescribeKey` | sensitive | yes | Legacy aliases of `GET /kms/keys/{key_id}` |
+| `GET /kms/list-keys` | `kms:ListKeys` | sensitive | no | Legacy alias of `GET /kms/keys`; same listing contract |
 
 ## Key listing contract
 
 Both listing routes (`GET /kms/keys` and the legacy `GET /kms/list-keys`) share one contract.
 
-`limit` is optional. When it is absent the server applies its own default page size of 100. When it is present it must parse as a non-negative integer: `limit=abc`, `limit=-1` and a value-less `limit` are refused with `400`, not silently read as "use the default". `limit=0` is a well-formed request for an empty page. Any page size above 1000 is served as 1000 — the response is `truncated` with a usable `next_marker`, so a client that pages until `truncated` is false still reaches every key. Clients must not assume a page is the size they asked for.
+`limit` is optional. When it is absent the server applies `DEFAULT_LIST_KEYS_PAGE_SIZE` (100). When it is present it must parse as a non-negative integer: `limit=abc`, `limit=-1` and a value-less `limit` are refused with `400`, not silently read as "use the default". `limit=0` is a well-formed request for an empty page. Any page size above `MAX_LIST_KEYS_PAGE_SIZE` (1000) is served as 1000 — the response is `truncated` with a usable `next_marker`, so a client that pages until `truncated` is false still reaches every key. Clients must not assume a page is the size they asked for.
 
 `marker` is opaque to the client: treat it as a cursor to hand back unchanged, never as a value to construct. On the Local, Vault KV2, Vault Transit and Static backends it happens to be an exclusive lower bound on the key identifier, which is what makes paging survive keys being created or destroyed mid-listing; on the AWS backend it is AWS's own pagination token, and sending a key id there is rejected. An empty `marker` means the same thing as no marker at all. Filters are applied after the page is cut, so a filtered page can be short — even empty — while more keys remain. Page until `truncated` is false, never until a page comes back short.
 
 `unreadable_key_ids` is present only when the server listed a key whose record it could not describe — a record written by a newer build, or damaged material. The identifiers are reported rather than omitted, so a listing never quietly understates the key set; a client displaying an inventory should surface them as damaged rather than dropping them, and paging always advances past a damaged key. A failure that says nothing about a specific key (timeout, `5xx`, permission denied) still fails the whole listing instead of appearing here.
 
 One case is deliberately an error rather than a report: a listing that covered the entire key set — no `marker`, and not `truncated` — in which nothing was readable. An empty `keys` array there would be indistinguishable, to any client written before this field existed, from a deployment that has no keys, and the usual response to that is to provision a new one. Such a listing returns `500` instead, naming the first failure; the individual identifiers are in the server log. A truncated page, or one resumed from a marker, always reports rather than failing, so a damaged key can never strand the keys behind it.
-
-## Server-side snapshot coverage
-
-The merged #5626 producer snapshots cover the nine modern/legacy key response types and the metadata response type served by `kms_keys.rs` and `kms_key_metadata.rs`: create, describe, list, generate-data-key, delete, cancel-deletion, update-description, tag, and untag. The four dynamic responses served verbatim by `kms_dynamic.rs` are covered in `crates/kms/src/snapshots/`: configure, start, stop, and the `service-status` response. `POST /kms/reload` serves the same `ConfigureKmsResponse` type the configure snapshot pins; it adds no new wire shape.
-
-`POST /kms/clear-cache` now has a named `KmsClearCacheResponse` and a producer snapshot beside the others; its serialized bytes are unchanged from the inline JSON it replaced.
-
-The remaining wire-shape gaps are intentionally documented rather than duplicated here: the management `KmsStatusResponse` (`GET|POST /kms/status`, pending #1636), `KmsConfigResponse`, all three lifecycle responses, and the backup/restore response family. Adding producer snapshots for those gaps is a separate server test task; it must not be inferred from the client matrix.
-
-## Client handoff gaps
-
-The `rc` client currently has status, key list/status/create/delete/cancel-deletion, configure/reconfigure/start/restart/stop, and diagnostic/roundtrip entry points. It has no lifecycle enable/disable/rotate, key metadata, backup/restore, or reload commands; `POST /kms/reload` is the recovery path when a restarted server reports not-configured while a persisted configuration exists, so it is a client delivery item alongside the lifecycle gaps. The console currently calls service-status, configure/reconfigure/start/stop/config, clear-cache, status, and the modern key CRUD routes. It has no lifecycle, metadata, or backup/restore UI. These pending cells are delivery items for `rustfs/cli` and `rustfs/console`; they are not implemented in this repository. A read-only issue search on 2026-08-02 found no matching KMS issue in either client repository, so the client handoff still needs issue creation there.
-
-`POST /kms/generate-data-key` is deliberately marked “do not expose” for both clients: its response contains a base64 plaintext data key. `GET /kms/config` and backup status/restore responses contain operational paths and identifiers, not key material, but still require UI/CLI redaction and confirmation handling.
-
-The producer response snapshots in #5626 and this matrix do not imply that rustfs/backlog#1639 is complete. The pending client cells must be closed in their respective repositories before the parent delivery item can be marked complete.

docs/operations/kms-backend-security.md

@@ -1,56 +1,38 @@
 # KMS backend security properties
 
-RustFS ships several KMS backends. They differ not only in deployment effort but in **where master key material lives and who can read it**. Pick a backend based on the confidentiality boundary you need, not on the name alone.
+**Use this when:** choosing a KMS backend, scheduling or debugging master key rotation, planning a rolling upgrade of a cluster with KMS enabled, or auditing where master key material lives and who can read it.
+**Source of truth:** `crates/kms/src/config.rs` (`KmsBackend`, `ENV_KMS_*` constants), `crates/kms/src/backends/{local,vault,vault_transit,aws}.rs`, `crates/kms/src/encryption/dek.rs` (`DataKeyEnvelope`), `rustfs/src/admin/route_policy.rs` (KMS route actions).
 
-For how the Vault backends authenticate (static token, AppRole, Kubernetes, Vault Agent token file) and how credential refresh and the fail-closed window behave, see the [Vault KMS authentication runbook](vault-kms-authentication.md). For what may be claimed about the cryptographic implementations themselves, see [Cryptographic compliance positioning](kms-cryptographic-compliance.md). For which RustFS identities may manage or use a given key, see [Per-key KMS authorization](kms-per-key-authorization.md). If you are migrating from MinIO, read [Migrating from MinIO: encrypted objects do not carry over](#migrating-from-minio-encrypted-objects-do-not-carry-over) first.
+RustFS ships several KMS backends. They differ not only in deployment effort but in **where master key material lives and who can read it**. Pick a backend based on the confidentiality boundary you need, not on the name alone. Related: [Vault KMS authentication runbook](vault-kms-authentication.md) (credential sources, refresh, fail-closed window), [Cryptographic compliance positioning](kms-cryptographic-compliance.md), [Per-key KMS authorization](kms-per-key-authorization.md), [KMS admin API contract](kms-admin-contract.md), [KMS observability runbook](kms-observability-runbook.md).
 
 ## Backend comparison
 
 | Backend | Config tag | Master key material location | At-rest protection of key material | Durability | Rotation | Intended use |
 | --- | --- | --- | --- | --- | --- | --- |
-| Local | `Local` | Files under `key_dir`, encrypted with the configured local master key | Local master key (AES-GCM) + file permissions | Crash-durable commits on local filesystems only; see [Local backend durability and deployment support matrix](#local-backend-durability-and-deployment-support-matrix) | Rejected by design (single material, development backend) | Development, testing and demos only; not supported for production |
+| Local | `Local` | Files under `key_dir`, encrypted with the configured local master key | Local master key (AES-GCM) + file permissions | Crash-durable commits on local filesystems only; see [Local backend durability and deployment support matrix](#local-backend-durability-and-deployment-support-matrix) | Rejected by design (single material) | Development, testing and demos only; not supported for production |
 | Static | `Static` | Provided out-of-band via environment/file; never persisted by RustFS | Operator-managed secret distribution | No state persisted by RustFS | Rejected (read-only backend) | Development and testing with an externally supplied key; not supported for production |
 | Vault KV2 | `VaultKV2` (legacy alias `Vault`) | Stored **directly** in Vault KV v2 (Base64-encoded plaintext) | Vault ACLs + KV v2 at-rest encryption + TLS only | Delegated to Vault storage | Versioned retention (immutable per-version records + current pointer) | Deployments that accept Vault KV ACLs as the sole confidentiality boundary |
 | Vault Transit | `VaultTransit` | Key-encryption keys never leave Vault; only Transit ciphertext is visible outside | Vault Transit engine (cryptographic isolation) | Delegated to Vault storage | Via Vault Transit key versioning | Deployments that need key material to be unreadable through storage APIs |
-| AWS KMS | `AWS` (alias `AwsKms`) | Key material never leaves AWS KMS; RustFS mirrors no key state | AWS KMS (cryptographic isolation) + IAM | Delegated to AWS | On-demand `RotateKeyOnDemand`; prior backing keys stay usable for decryption | Deployments already rooted in AWS IAM that want AWS as the cryptographic root — read [AWS KMS: deviations from the shared backend contract](#aws-kms-deviations-from-the-shared-backend-contract) first |
+| AWS KMS | `AWS` (alias `AwsKms`) | Key material never leaves AWS KMS; RustFS mirrors no key state | AWS KMS (cryptographic isolation) + IAM | Delegated to AWS | On-demand `RotateKeyOnDemand`; prior backing keys stay usable for decryption | Deployments rooted in AWS IAM — read [AWS KMS: deviations from the shared backend contract](#aws-kms-deviations-from-the-shared-backend-contract) first |
 
 ## Migrating from MinIO: encrypted objects do not carry over
 
-> **Warning: RustFS does not currently support reading objects that MinIO encrypted.**
-> This applies to SSE-S3, SSE-KMS, and SSE-C, in every released binary and container image, and it holds regardless of which KMS backend you configure. Configuring the `Static` backend with the same key material MinIO used does **not** make those objects readable — MinIO wraps data keys in a different envelope format that no RustFS backend produces or accepts (`crates/kms/src/config.rs:304-308`). Plan for this **before** moving data. Tracked in rustfs/backlog#1638.
+> **Warning: default RustFS builds fail closed on objects that MinIO encrypted.** This applies to SSE-S3, SSE-KMS, and SSE-C, whichever KMS backend you configure; configuring `Static` with MinIO's key material does not make them readable. Such objects list and HEAD normally (their `xl.meta` parses), and only the payload read fails — with S3 `InvalidObjectState`, never plaintext. Read a sample of encrypted objects, not just their listings, before decommissioning the MinIO deployment.
 
-The read does fail closed — ciphertext is never served as plaintext. MinIO's internal encryption headers mark the object as encrypted (`crates/utils/src/http/header_compat.rs:50-67`), so the read path demands encryption material and refuses when none resolves (`crates/ecstore/src/object_api/readers.rs:559-568`). Two properties still make the problem easy to discover late:
-
-- **The error does not say what happened.** It surfaces as a 500 `InternalError`, which reads as a RustFS fault rather than "another implementation encrypted this object".
-- **Surrounding metadata migrates fine.** The object's `xl.meta` parses, so encrypted objects list and HEAD normally and report plausible sizes. The failure appears only when something reads the payload.
-
-Read a sample of encrypted objects, not just their listings, before decommissioning the MinIO deployment.
-
-Current options for a migration whose source contains encrypted objects:
-
-- Decrypt on the MinIO side first, migrate plaintext, then let RustFS re-encrypt with its own KMS.
-- Copy through the S3 API rather than moving drives — MinIO decrypts on read, and RustFS encrypts on write. This re-encrypts rather than preserving ciphertext and costs a full data transfer.
-- Leave encrypted objects on MinIO and migrate only unencrypted data.
-
-Inventory the source before choosing: bucket default-encryption settings mean objects can be encrypted without any client having sent SSE headers.
-
-The same limitation applies in reverse — objects RustFS encrypts are not readable by MinIO. For the code-level breakdown of which seams block each SSE mode, see [MinIO file-format interoperability, Part C](../architecture/minio-file-format-compat.md#part-c--server-side-encryption-sse).
-
-The migration warning is not a wire-protocol promise: the **AWS KMS wire protocol** and **MinIO KES wire protocol** are explicit non-targets for this document. The AWS backend uses the AWS SDK client path (`crates/kms/src/backends/aws.rs:830`), and KES remains outside the MinIO on-disk interop scope. Track those ecosystem evaluations and the MinIO/RustFS SSE compatibility matrix in the [#1562 Production Ready exit gate](https://github.com/rustfs/backlog/issues/1562); #1638 alone does not satisfy that gate.
+The read path exists behind the `rio-v2` feature as a migration-only build, the reverse direction (RustFS-written SSE objects read by MinIO) is unsupported, and the migration options are enumerated in [MinIO file-format interoperability, Part C](../architecture/minio-file-format-compat.md#part-c--server-side-encryption-sse). The AWS KMS and MinIO KES wire protocols are non-targets of that document and of this one.
 
 ## Vault KV2: what the backend does and does not do
 
 The Vault KV2 backend uses Vault purely as a **secure storage** service:
 
 - Master key material is generated by RustFS and written to KV v2 as a Base64-encoded value (`encrypted_key_material` is an encoding, not a ciphertext).
-- The backend never calls the Vault Transit engine. The `mount_path` configuration field and the `RUSTFS_KMS_VAULT_MOUNT_PATH` environment variable are deprecated leftovers: they are accepted for compatibility and ignored.
+- The backend never calls the Vault Transit engine. The `mount_path` configuration field and the `RUSTFS_KMS_VAULT_MOUNT_PATH` environment variable are deprecated leftovers: accepted for compatibility and ignored.
 - Data-encryption keys (DEKs) handed to the object-encryption path are still wrapped with AES-256-GCM under the master key; the statement above concerns the master key's storage in Vault, not the DEK envelope.
-- No runtime interface reports this boundary. `GET /rustfs/admin/v3/kms/status` names the active backend (`backend_type: vault-kv2`) and returns a `capabilities` matrix, but that matrix enumerates only the operations the backend supports — nothing in it describes where master key material lives or who can read it. Determining which confidentiality boundary is in force means reading `backend_type` and applying the comparison table above; this document is the only statement of the boundary an operator can consult.
-- The `at_rest_protection: storage-only` field carried by a KMS backup manifest is a different thing: it declares the protection state of key material inside a backup bundle, not a property the running backend reports about itself.
-- Key rotation retains every historical master key version as an immutable record under `{prefix}/{key_id}/versions/{N}` and only then moves the current-version pointer; see [Master key rotation](#master-key-rotation-retention-destruction-and-upgrade-ordering) for the retention preconditions and the cluster-upgrade ordering constraint.
+- No runtime interface reports this boundary. `GET /rustfs/admin/v3/kms/status` names the active backend (`backend_type: vault-kv2`) and returns a `capabilities` matrix, but that matrix enumerates only supported operations. Determining the confidentiality boundary in force means reading `backend_type` and applying the comparison table above.
+- The `at_rest_protection: storage-only` field carried by a KMS backup manifest declares the protection state of key material inside a backup bundle, not a property the running backend reports about itself.
+- Key rotation retains every historical master key version as an immutable record under `{prefix}/{key_id}/versions/{N}` and only then moves the current-version pointer; see [Master key rotation](#master-key-rotation-retention-destruction-and-upgrade-ordering).
 
-> **Warning: KV read access is equivalent to holding the master keys.**
-> Any Vault identity (token, AppRole, or policy) that can `read` the RustFS key path in KV v2 can recover the plaintext master key material and decrypt every object protected by those keys. Treat KV read grants on that path with the same care as handing out the keys themselves. If this is not acceptable, use the Vault Transit backend instead.
+> **Warning: KV read access is equivalent to holding the master keys.** Any Vault identity (token, AppRole, or policy) that can `read` the RustFS key path in KV v2 can recover the plaintext master key material and decrypt every object protected by those keys. If this is not acceptable, use the Vault Transit backend instead.
 
 ## Minimal Vault policy for the KV2 backend
 
@@ -67,63 +49,49 @@ path "secret/metadata/rustfs/kms/keys/*" {
 }
 ```
 
-Notes:
-
-- The trailing wildcards also cover the per-version material records that rotation creates under `.../keys/{key_id}/versions/{N}`; no extra policy paths are needed.
-- `delete` on the metadata path is required for permanent key deletion (`force_immediate`); drop it if you never hard-delete keys. RustFS refuses `force_immediate` unless the server sets `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true`, so leaving that gate off keeps the capability unreachable no matter what the Vault policy allows.
-- Do not attach `sudo`, wildcard mounts, or Transit paths to this policy; the KV2 backend does not use them.
-- Auditing KV reads on the key prefix is strongly recommended: every read event is a potential master-key disclosure.
+- The trailing wildcards also cover the per-version material records under `.../keys/{key_id}/versions/{N}`; no extra policy paths are needed.
+- `delete` on the metadata path is required only for permanent key deletion (`force_immediate`); drop it if you never hard-delete keys. RustFS refuses `force_immediate` unless the server sets `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true`, so leaving that gate off keeps the capability unreachable whatever the Vault policy allows.
+- Do not attach `sudo`, wildcard mounts, or Transit paths; the KV2 backend does not use them.
+- Audit KV reads on the key prefix: every read event is a potential master-key disclosure.
 
 ## Master key rotation: retention, destruction, and upgrade ordering
 
-Rotation support differs per backend. Local and Static advertise no `rotate` capability — `capabilities.rotate` is false in the `kms/status` response — and reject rotation with `UnsupportedCapability`; their single key material is never overwritten. Vault Transit delegates rotation to the Transit engine's own key versioning (ciphertext is version-prefixed, e.g. `vault:v1:...`). Vault KV2 rotates by retaining every historical version, as described below. Rotation is reachable through the admin API as `POST /rustfs/admin/v3/kms/keys/rotate`, which the route policy classifies as high risk and gates behind `kms:RotateKey`; it is not exposed through the S3 surface. The upgrade ordering constraint below therefore applies to an operator action, not only to a call from inside the process.
+Rotation is reachable through `POST /rustfs/admin/v3/kms/keys/rotate` (`kms:RotateKey`, high risk); it is not exposed on the S3 surface. Local and Static advertise no `rotate` capability (`capabilities.rotate` is false in the `kms/status` response) and reject rotation with `UnsupportedCapability`. Vault Transit delegates rotation to the Transit engine's own versioning (ciphertext is version-prefixed, e.g. `vault:v1:...`). Vault KV2 rotates by retaining every historical version, as described below.
 
 ### Rotation drivers and scheduling, per backend
 
-The rotate endpoint is one API over three very different mechanisms, and which component actually performs the rotation decides how periodic rotation must be scheduled — on two backends it cannot be scheduled at all.
+The rotate endpoint is one API over three different mechanisms, and which component performs the rotation decides how periodic rotation must be scheduled.
 
-| Backend | Can rotate | Who performs the rotation | How to schedule periodic rotation |
-| --- | --- | --- | --- |
-| Local | No | Nobody — the backend advertises no `rotate` capability and the rotate endpoint is refused with `UnsupportedCapability` | Cannot be scheduled. Migrating to a rotating backend is the only path to rotation |
-| Static | No | Nobody — same refusal as Local; the material is supplied out-of-band and read-only | Cannot be scheduled. Migrate to a rotating backend |
-| Vault KV2 | Yes | **RustFS** owns the whole rotation protocol: freeze the outgoing material as an immutable version record, persist the new version's material, then move the current pointer with a check-and-set write | An **external scheduler** (cron, Kubernetes CronJob, your automation platform) calling `POST /rustfs/admin/v3/kms/keys/rotate`. RustFS deliberately ships no built-in rotation timer — see below |
-| Vault Transit | Yes | **Vault's Transit engine** — RustFS only forwards the call to Transit's rotate endpoint and records the version bump in its own metadata | Vault's native `auto_rotate_period` on the Transit key. Do **not** additionally point an external scheduler at the RustFS rotate endpoint — see below |
-| AWS KMS | Yes | **AWS** — the RustFS rotate endpoint maps to `RotateKeyOnDemand` | AWS's native automatic rotation, configured on the AWS side. Do **not** drive periodic rotation through the RustFS endpoint — see below |
+| Backend | Can rotate | Who performs the rotation | How to schedule periodic rotation | Wrap ceiling |
+| --- | --- | --- | --- | --- |
+| Local | No | Nobody — the rotate endpoint is refused with `UnsupportedCapability` | Cannot be scheduled; migrating to a rotating backend is the only path | Unmitigable |
+| Static | No | Nobody — same refusal; the material is supplied out-of-band and read-only | Cannot be scheduled; migrate | Unmitigable |
+| Vault KV2 | Yes | **RustFS** owns the protocol: freeze the outgoing material as an immutable version record, persist the new material, move the current pointer with a check-and-set write | Exactly **one external scheduler** (cron, Kubernetes CronJob) calling the rotate endpoint with credentials scoped to `kms:RotateKey` | Reset by each rotation |
+| Vault Transit | Yes | **Vault's Transit engine**; RustFS forwards the call and records the version bump | Vault's native `auto_rotate_period` on the Transit key. Do **not** also drive the RustFS endpoint: two owners of the version cadence means neither configured period holds. The reported key version advances only through RustFS, so on an auto-rotating key treat it as a floor | Not applicable (wraps inside Vault) |
+| AWS KMS | Yes | **AWS** — the endpoint maps to `RotateKeyOnDemand` | AWS's native automatic rotation. Do **not** drive periodic rotation through the RustFS endpoint: AWS caps lifetime on-demand rotations, so a scheduler exhausts the quota and then fails forever. Keep the endpoint for incidents. RustFS neither enables nor observes AWS automatic rotation and records no rotation timestamp, so its rotation-age signals measure key age on this backend | Not applicable (wraps inside AWS) |
 
-**Local and Static: the wrap ceiling is unmitigable.** These backends wrap every DEK with AES-256-GCM under their single master key using a random 96-bit nonce, and NIST SP 800-38D caps AES-GCM at 2^32 invocations per key when nonces are chosen at random. Each encrypted object write wraps a DEK, so the invocation count tracks the number of encrypted-object writes over the deployment's lifetime. On a rotating backend that count restarts whenever new master key material takes over; on Local and Static it can never restart, because there is no rotation to restart it. The only mitigation is migrating to a backend that rotates. The same 2^32 bound applies to the KV2 backend's wrapping — RustFS wraps DEKs locally there too — but there each rotation mints fresh master key material and resets the count, which is one more reason to actually schedule KV2 rotation rather than merely support it.
+**Wrap ceiling.** Where RustFS wraps DEKs locally (Local, Static, Vault KV2) every DEK is wrapped with AES-256-GCM under the master key using a random 96-bit nonce, and NIST SP 800-38D caps AES-GCM at 2^32 invocations per key under random nonces. Each encrypted-object write is one wrap, so the count tracks lifetime encrypted writes. Rotation installs fresh material and restarts the count; on Local and Static there is no rotation, so the ceiling can only be escaped by migrating.
 
-**Vault KV2: bring your own scheduler, deliberately.** RustFS performs the rotation but does not decide when: there is no built-in rotation worker, by design rather than omission. A timer inside the server cannot verify the [cluster-upgrade precondition](#upgrade-before-first-rotation-hard-constraint) before firing, and rotation is not idempotent — without leader election, N nodes running the same schedule would perform N rotations per period, advancing the key version N times. Run exactly one external scheduler, point it at the admin rotate endpoint with credentials scoped to `kms:RotateKey`, and use the [rotation readiness fields](#rotation-readiness-reported-never-acted-on) plus the `KmsKeyRotationOverdue` alert in the [KMS observability runbook](kms-observability-runbook.md#kmskeyrotationoverdue) to verify the schedule is actually keeping up.
-
-**Vault Transit: exactly one owner of the version cadence.** Configure `auto_rotate_period` on the Transit key and let Vault own the schedule. Layering an external scheduler that calls the RustFS rotate endpoint on top of `auto_rotate_period` creates two competing owners of the key's version cadence, and the effective rotation period stops being the one either owner was configured with. The data path is indifferent to who rotates — Transit ciphertext self-describes the version that wrapped it, so envelopes never pin a version RustFS tracked — but the key version RustFS reports only advances when rotation goes through RustFS, so on an auto-rotating key treat the reported version as a floor, not the truth.
-
-**AWS KMS: native automatic rotation for cadence, `RotateKeyOnDemand` for incidents.** The RustFS rotate endpoint maps to AWS `RotateKeyOnDemand`, and AWS enforces a lifetime limit on the number of on-demand rotations a key may receive (see the AWS KMS documentation) — a periodic scheduler driving the RustFS endpoint will exhaust that quota and then fail forever. Configure AWS's automatic rotation for periodic cadence and keep the RustFS endpoint for what on-demand rotation is for: incident response and one-off rotations. Note that RustFS neither enables nor observes AWS automatic rotation, and it records no rotation timestamp for AWS keys, so the readiness fields and the rotation-age gauge measure key age on this backend — verify the actual cadence in AWS, not through RustFS.
+**Why there is no built-in rotation timer (KV2).** A timer inside the server cannot verify the [upgrade-before-first-rotation constraint](#upgrade-before-first-rotation-hard-constraint), and rotation is not idempotent: without leader election, N nodes on the same schedule would advance the key version N times per period. Verify that your scheduler keeps up with the [rotation readiness fields](#rotation-readiness-reported-never-acted-on) and the `KmsKeyRotationOverdue` alert in the [KMS observability runbook](kms-observability-runbook.md#kmskeyrotationoverdue).
 
 **Pre-rotation checklist** (before the first rotation of any key, and before enabling any schedule):
 
-1. Every node in the cluster runs a build that understands the `master_key_version` envelope field — the [hard upgrade-ordering constraint](#upgrade-before-first-rotation-hard-constraint) below. A timer cannot check this; you must.
+1. Every node runs a build that understands the `master_key_version` envelope field — the [hard constraint](#upgrade-before-first-rotation-hard-constraint) below. A timer cannot check this; you must.
 2. No rolling upgrade is in progress — see [Do not do these during a mixed-version window](#do-not-do-these-during-a-mixed-version-window).
-3. The [retention and destruction preconditions](#retention-and-destruction-preconditions) are understood: every version record a stored DEK envelope references must remain readable forever, and no retention tooling prunes the version subtree.
-4. For KV2, exactly one scheduler exists, so no two callers race the same rotation period.
+3. The [retention and destruction preconditions](#retention-and-destruction-preconditions) are understood: every version record a stored DEK envelope references must remain readable, and no retention tooling prunes the version subtree.
+4. For KV2, exactly one scheduler exists.
 5. `RUSTFS_KMS_ROTATION_MAX_AGE_SECS` is set to the rotation period your policy requires, so the per-key `rotation_due` verdict and the rotation-age alert verify the schedule instead of assuming it.
 
 ### Rotation readiness: reported, never acted on
 
-RustFS does not rotate keys on a schedule. There is no built-in rotation worker, deliberately: rotation is a policy decision with a per-backend cost and a hard upgrade-ordering constraint (see below), and a server that rotated on its own would make that decision on an operator's behalf at a moment it did not choose. What the server does instead is tell you which keys have outlived a period you configure.
+RustFS reports which keys have outlived a period you configure; nothing consults the verdict before encrypting or decrypting, and it has no effect on readiness or liveness.
 
-Set `RUSTFS_KMS_ROTATION_MAX_AGE_SECS` to that period in whole seconds. Unset — the default — leaves the verdict unreported rather than assuming a policy: how often keys must be rotated is a compliance decision, and a built-in default would report keys as overdue against a rule nobody wrote. An unparsable value is treated the same way, with a warning, instead of silently falling back to a number the operator did not choose. Values below one hour are raised to one hour, because a threshold of seconds reports every key as overdue moments after it was rotated and teaches operators to ignore the signal.
+| Setting | Meaning | Unset or unparsable | Floor |
+| --- | --- | --- | --- |
+| `RUSTFS_KMS_ROTATION_MAX_AGE_SECS` | Rotation period in whole seconds; keys older than this report `rotation_due` with reason `age` or `never_rotated` | No age verdict is reported (a warning is logged for an unparsable value) — how often keys must rotate is a compliance decision, not a built-in default | 1 hour |
+| `RUSTFS_KMS_ROTATION_MAX_WRAPS` | Data keys one key's material may wrap before `rotation_due` with reason `wraps` | No wrap verdict is reported | 1,000,000 (wraps are accounted in reserved blocks of that size) |
 
-A second, independent threshold covers the cryptographic bound rather than the policy one. `RUSTFS_KMS_ROTATION_MAX_WRAPS` is the number of data keys one key's material may wrap before the verdict reports `rotation_due` with reason `wraps`. It follows the same discipline — unset or unparsable leaves the verdict unreported, and values below one million are raised to one million because wraps are accounted in reserved blocks of that size, so a smaller threshold would trip on the first reservation. Only backends where RustFS wraps locally and can rotate report a count (Vault KV2 today); Transit and AWS wrap externally and report none, so the wrap half stays silent there rather than guessing. When both thresholds are crossed the reported reason is `wraps`: the AES-GCM random-nonce ceiling is not negotiable, while the age period is a policy an operator chose.
-
-`GET /rustfs/admin/v3/kms/keys` then carries two additional fields per key:
-
-- `rotation_due` — whether the key has outlived the configured period.
-- `rotation_due_reason` — `age` when the key was rotated but longer ago than the period, `never_rotated` when it has never been rotated and has been in use longer than the period, and `unsupported` when the backend cannot rotate at all. Absent when there is no verdict.
-
-The verdict is advisory in the strongest sense: nothing consults it before encrypting or decrypting, a key reported as due keeps serving traffic unchanged, and it has no effect on readiness or liveness. It is computed in one place, from the backend's declared rotation capability plus the key's own timestamps, so no two backends can disagree about what "overdue" means — and a backend that cannot rotate is reported as `unsupported` rather than being told to do something it cannot.
-
-`GET /rustfs/admin/v3/kms/keys/{key_id}` does **not** carry these fields. Its response type records a creation date but no rotation timestamp, so a verdict computed there could not tell a key rotated last week from one never rotated at all, and reporting `never_rotated` for a key that was in fact rotated would be worse than reporting nothing. Read the verdict from the listing.
-
-Driving the rotation itself remains external: call `POST /rustfs/admin/v3/kms/keys/rotate` from your own scheduler, having first satisfied the upgrade-ordering constraint below — and only on the backend where that is the right scheduling model; see [Rotation drivers and scheduling, per backend](#rotation-drivers-and-scheduling-per-backend).
+`GET /rustfs/admin/v3/kms/keys` carries `rotation_due` and `rotation_due_reason` per key: `age` (rotated, but longer ago than the period), `never_rotated` (in use longer than the period, never rotated), `wraps` (wrap budget exceeded; wins over `age` when both hold, because the AES-GCM ceiling is not negotiable), or `unsupported` (the backend cannot rotate). Only backends where RustFS wraps locally and can rotate count wraps (Vault KV2); Transit and AWS report no count. `GET /rustfs/admin/v3/kms/keys/{key_id}` does **not** carry these fields — its response records a creation date but no rotation timestamp, so read the verdict from the listing.
 
 ### Vault KV2 versioned retention model
 
@@ -133,59 +101,53 @@ Decryption loads exactly the version recorded in the envelope and fails closed w
 
 ### Retention and destruction preconditions
 
-- Every version record that any stored DEK envelope references must remain readable. The bulk rekey sweep (`POST /rustfs/admin/v3/kms/keys/rekey`, gated on `kms:Rekey`) rewraps stored envelopes onto the current version; until a sweep has completed with zero failures after the last rotation, assume **every** version of a rotated key is referenced: destroying a version record permanently orphans all objects whose DEKs it wrapped. A completed sweep is evidence, not authority — the deletion gate stays the decision point. Replication strips encryption metadata in transit, so a sweep never propagates to a replica site: each site runs its own.
-- Version records are ordinary KV v2 secrets under the key subtree. Never run `kv metadata delete` or `kv destroy` against `{prefix}/{key_id}/versions/*`, and do not apply `delete-version-after` or retention tooling to that subtree. RustFS-managed retention does not rely on KV2's own secret versioning (each version record has a single KV revision), so KV `max-versions` settings do not protect or endanger history — but metadata deletion always removes a record entirely.
-- Permanent key deletion through RustFS (`force_immediate` after `PendingDeletion`) purges the key's version records together with the key record; that is the only supported way to remove them. It is refused by default: the server must set `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true`, and the request must be a `DELETE` with a JSON body that sets `force_immediate` and echoes the key id back as `confirm_key_id` — the query-parameter form (`?force_immediate=true`) is refused outright, whatever the gate is set to. Leave the gate off unless you are actively destroying keys, and turn it off again afterwards — the pending-deletion window plus `CancelKeyDeletion` is the only recovery path for objects encrypted under the key.
+- Every version record that any stored DEK envelope references must remain readable. The bulk rekey sweep (`POST /rustfs/admin/v3/kms/keys/rekey`, `kms:Rekey`) rewraps stored envelopes onto the current version; until a sweep has completed with zero failures after the last rotation, assume **every** version of a rotated key is referenced. A completed sweep is evidence, not authority — the deletion gate stays the decision point. Replication strips encryption metadata in transit, so each replica site runs its own sweep.
+- Version records are ordinary KV v2 secrets under the key subtree. Never run `kv metadata delete` or `kv destroy` against `{prefix}/{key_id}/versions/*`, and do not apply `delete-version-after` or retention tooling to that subtree. Each version record has a single KV revision, so KV `max-versions` settings neither protect nor endanger history — but metadata deletion removes a record entirely.
+- Permanent key deletion through RustFS (`force_immediate` after `PendingDeletion`) purges the key's version records together with the key record; that is the only supported way to remove them. It requires `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true` on the server and a `DELETE` with a JSON body that sets `force_immediate` and echoes the key id as `confirm_key_id`; the query-parameter form is refused outright. Leave the gate off except while actively destroying keys — the pending-deletion window plus `CancelKeyDeletion` is the only recovery path for objects encrypted under the key.
+- `force_immediate` is refused with `409 Conflict` while any bucket's default encryption configuration names the key or the key is the KMS service default key. A scheduled deletion is not refused for that reason: it destroys nothing and stays cancellable, and the background sweep re-checks the same references before destroying material.
 - For Vault Transit, retention is governed by the Transit key's `min_decryption_version`: never raise it above the oldest version that may still protect live ciphertext.
-- `force_immediate` is additionally refused, with a `409 Conflict`, while any bucket's default encryption configuration still names the key, or while the key is the KMS service default key. A scheduled deletion is not refused for that reason: it destroys nothing and stays cancellable, and the background sweep re-checks the same references before it destroys the material.
 
 ### Reading the `impact` section
 
-`DeleteKey` responses always carry an `impact` section listing the configuration that currently points at the key — the buckets whose default encryption names it, and whether it is the service default key — so the references that will refuse the destruction are visible when the deletion is scheduled rather than only in a server-side log once the window has run out.
+`DeleteKey` responses always carry an `impact` section listing the configuration that points at the key (buckets whose default encryption names it, and whether it is the service default key). `DescribeKey` (`GET /rustfs/admin/v3/kms/keys/{key_id}`) returns the same section only when asked with `impact=true`, because collecting it lists every bucket; a value other than `true`/`false` is rejected with `400`. **An absent section means "not collected", never "nothing references this key".**
 
-`DescribeKey` (`GET /rustfs/admin/v3/kms/keys/{key_id}`) can return the same section, but only when the request asks for it with `impact=true`. It is opt-in there because collecting it lists every bucket and `DescribeKey` is polled; without the parameter the endpoint does exactly the work it did before and returns no `impact` field at all. A value other than `true` or `false` is rejected with `400` rather than treated as `false`, so a typo can never answer a request for the section with a response that merely lacks one. **An absent section means "not collected", never "nothing references this key".**
-
-Read it for what it says and nothing more. `coverage.scanned` names the sources that were read; `coverage.not_scanned` names the ones that were not, which currently includes every object encrypted under the key. `completeness` is `exact` only over the scanned sources, and `unavailable` when a source could not be read at all — an unavailable report is not an empty one, and both an unreadable source and an outstanding reference will stop the sweep from destroying the material.
-
-**An empty `references` list does not mean the key is unused.** No object metadata is consulted, so a key with no configuration references can still protect an arbitrary amount of live data, which stays readable only until the material is gone. There is no field in the response that asserts otherwise, and none should be inferred from one.
+`coverage.scanned` names the sources that were read and `coverage.not_scanned` the ones that were not — which currently includes every object encrypted under the key. `completeness` is `exact` only over the scanned sources and `unavailable` when a source could not be read; both an unreadable source and an outstanding reference stop the sweep from destroying material. **An empty `references` list does not mean the key is unused:** no object metadata is consulted, so a key with no configuration references can still protect live data.
 
 ### Upgrade before first rotation (hard constraint)
 
-Do not rotate any key until **every** RustFS node in the cluster runs a build that understands the `master_key_version` envelope field. Older binaries ignore the field and always decrypt with the current material: harmless while nothing has been rotated, but after a rotation they will fail to decrypt every object wrapped by an earlier key version. Complete the rolling upgrade of the entire cluster first, then rotate.
-
-This is the sharpest instance of a broader class of constraints; the rest are collected in [Mixed-version clusters during a rolling upgrade](#mixed-version-clusters-during-a-rolling-upgrade).
+Do not rotate any key until **every** RustFS node runs a build that understands the `master_key_version` envelope field. Older binaries ignore the field and always decrypt with the current material: harmless while nothing has been rotated, but after a rotation they fail to decrypt every object wrapped by an earlier key version. Complete the rolling upgrade of the entire cluster first, then rotate. The rest of this constraint class is collected in [Mixed-version clusters during a rolling upgrade](#mixed-version-clusters-during-a-rolling-upgrade).
 
 ## Mixed-version clusters during a rolling upgrade
 
-During a rolling upgrade the cluster runs two RustFS builds at once. That window matters more for KMS than for most subsystems, because KMS state is shared three ways: **Vault** holds the key records and Transit metadata, **cluster storage** holds the persisted KMS configuration, and **each node's process memory** holds caches and the live backend instance. Nodes on different builds agree on the first, may disagree on the third, and — for configuration — can disagree for as long as the operator leaves them running, because the reload broadcast that converges configuration is one of the things an older build rejects.
-
-This section states only what is true of the current implementation. It is written for the KV2 and Transit backends; the Local backend is unsupported for multi-node deployments regardless of version (see the [deployment support matrix](#deployment-support-matrix)).
+During a rolling upgrade KMS state is shared three ways: **Vault** holds key records and Transit metadata, **cluster storage** holds the persisted KMS configuration, and **each node's process memory** holds caches and the live backend instance. Nodes on different builds agree on the first, may disagree on the third, and can disagree on configuration for as long as the operator leaves them running, because the reload broadcast that converges configuration is one of the things an older build rejects. This section is written for the KV2 and Transit backends; the Local backend is unsupported for multi-node deployments regardless of version (see the [deployment support matrix](#deployment-support-matrix)).
 
 ### Persisted formats are backward compatible in both directions
 
-Nothing in this list requires a coordinated format cutover. The compatibility is deliberate and is covered by decode tests.
+No coordinated format cutover is required; the compatibility is deliberate and covered by decode tests.
 
-- **DEK envelopes.** `DataKeyEnvelope::master_key_version` is optional and omitted when absent, so envelopes written by non-rotating backends stay byte-identical to the historical seven-field JSON shape. An upgraded node reading a pre-versioning envelope resolves `None` to the key's recorded baseline version, or — for a key that was never rotated, and so has no baseline — to the current version, which is exactly the pre-versioning behavior. Unknown values are skipped while parsing; a bounded field-name sample is emitted at a progressively rate-limited `warn` level, and `rustfs_kms_persisted_unknown_fields_total{record_kind="data-key-envelope"}` counts every observed field.
-- **Local key records.** Each `<key_id>.key` record carries `format_version: 1`; records written before that field existed default to version 1 when read, and the pre-version reader ignores the added v1 marker. A reader accepts a record whose version is at most the version it understands, and rejects a newer version with `UnsupportedFormatVersion` before it attempts to decrypt key material. Unknown fields remain accepted for rollback compatibility. Their values are ignored while parsing; a bounded field-name sample is emitted at a progressively rate-limited `warn` level, and `rustfs_kms_persisted_unknown_fields_total{record_kind="local-key-record"}` counts every observed field. Once a future version greater than 1 has written a key record, do not roll back to a build that predates this marker: such a build cannot reject that future version before interpreting the rest of the record.
-- **KV2 key records.** `baseline_version` is read with a serde default, so records written by older builds deserialize unchanged, and `None` correctly means "never rotated".
-- **Transit metadata records.** Metadata persisted in KV v2 by either build decodes on the other.
-
-The one-way hazard is the rotation constraint above: an older binary reading a *new* envelope silently ignores the version field and decrypts with the current material.
+| Record | Compatibility mechanism | Caveat |
+| --- | --- | --- |
+| DEK envelopes | `DataKeyEnvelope::master_key_version` is optional and omitted when absent, so envelopes from non-rotating backends stay byte-identical to the historical seven-field JSON. An upgraded node resolves a pre-versioning envelope to the key's `baseline_version`, or to the current version for a never-rotated key. Unknown fields are skipped; a bounded field-name sample is logged at a rate-limited `warn` and counted by `rustfs_kms_persisted_unknown_fields_total{record_kind="data-key-envelope"}` | An older binary reading a *new* envelope ignores the version field and decrypts with the current material — the rotation constraint above |
+| Local key records | Each `<key_id>.key` carries `format_version: 1`; records without it default to 1. A reader accepts a version at most the one it understands and rejects newer with `UnsupportedFormatVersion` before decrypting. Unknown fields are accepted and counted by `rustfs_kms_persisted_unknown_fields_total{record_kind="local-key-record"}` | Once a future version greater than 1 has written a record, do not roll back to a build predating the marker |
+| KV2 key records | `baseline_version` is read with a serde default; `None` means "never rotated" | An old build drops the field on write-back (see below) |
+| Transit metadata records | Decode on either build | — |
 
 ### DEK envelope context binding (`RUSTFS_KMS_ENVELOPE_AAD`)
 
-Historically the KV2 and Local backends sealed only the DEK plaintext; the `encryption_context` rode in the envelope unauthenticated and was checked by field comparison alone, so a party able to rewrite the stored envelope could rewrite the context to match whatever it presented. With `RUSTFS_KMS_ENVELOPE_AAD=true`, newly wrapped envelopes bind the canonical context bytes as AES-GCM additional data and carry `context_binding: 1`; rewriting the stored context, or stripping the flag, then fails authentication. (Static, Vault Transit and AWS already bound the context through their own mechanisms and are unaffected.)
+Historically the KV2 and Local backends sealed only the DEK plaintext; the `encryption_context` rode in the envelope unauthenticated and was checked by field comparison alone, so a party able to rewrite the stored envelope could rewrite the context. With `RUSTFS_KMS_ENVELOPE_AAD=true`, newly wrapped envelopes bind the canonical context bytes as AES-GCM additional data and carry `context_binding: 1`; rewriting the stored context, or stripping the flag, then fails authentication. Static, Vault Transit and AWS already bound the context through their own mechanisms and are unaffected.
 
-Rollout constraint: **reading bound envelopes needs no switch, but a node that predates the field cannot open them** — its unwrap runs without the additional data and fails authentication. Enable the switch only after every node in the cluster runs a release that understands `context_binding`; the default stays off for one release for exactly this reason, mirroring the `RUSTFS_ENCRYPTION_FRAME_V2` rollout. Rewrap migrates existing envelopes: with the switch on, a rewrap sweep upgrades unbound envelopes to the bound format (converging to zero writes on re-run), and a bound envelope never regresses to the unbound shape whatever the switch says. An envelope carrying an unrecognized `context_binding` value is refused rather than decrypted without its binding.
+Rollout constraint: reading bound envelopes needs no switch, but **a node that predates the field cannot open them** — its unwrap runs without the additional data and fails authentication. The switch defaults off (`ENV_KMS_ENVELOPE_AAD` in `crates/kms/src/config.rs`); enable it only after every node runs a release that understands `context_binding`, mirroring the `RUSTFS_ENCRYPTION_FRAME_V2` rollout. With the switch on, a rewrap sweep upgrades unbound envelopes to the bound format (converging to zero writes on re-run); a bound envelope never regresses to the unbound shape, and an envelope carrying an unrecognized `context_binding` value is refused rather than decrypted without its binding.
 
 ### Guarantees that hold only once every node is upgraded
 
-These are properties of the upgraded code, so a single node left behind removes them for the whole cluster.
+These are properties of builds from `1.0.0-rc.1` onward; a single older node removes them for the whole cluster.
 
-- **Check-and-set lifecycle writes.** Upgraded builds write every KV2 lifecycle mutation — create, enable, disable, tag metadata, schedule deletion, cancel deletion — as a versioned read followed by a check-and-set write, retrying on conflict by re-reading and re-validating the state gate (rustfs/rustfs#5518). Transit metadata writes got the same treatment (rustfs/rustfs#5520). Builds older than those write blind. A blind write from an old node can overwrite a check-and-set commit from an upgraded node without any conflict being reported, which is precisely the lost update the change was made to eliminate.
-- **`baseline_version` survives a write-back.** The KV2 key record does not deny unknown fields, so an old build reads a new record without error — and drops `baseline_version` when it writes that record back for any reason. A key that loses its baseline resolves pre-versioning envelopes to the current version again, which after a rotation means the wrong master key material. Any lifecycle operation issued to an old node is enough to trigger this.
-- **`wrap_budget_reserved` keeps overestimating.** The KV2 key record's approximate wrap counter (`wrap_budget_reserved`, behind the `rustfs_kms_max_key_wrap_operations` gauge) is dropped the same way when an old build rewrites the record, regressing the count toward zero — the one way this deliberately overestimate-only counter can understate the wraps actually performed. Nothing breaks: the counter is advisory, and the next block reservation from an upgraded node re-establishes a floor. Just do not trust a *low* gauge reading taken during or shortly after a mixed-version window.
-- **Version-record awareness.** Rotation stores each historical version under `{prefix}/{key_id}/versions/{N}` as a create-only record (check-and-set of 0), so two nodes racing the same version number produce exactly one creator; the loser adopts the persisted, never-current material or fails without touching the current pointer. Old builds have no concept of that sub-path: they never read or write it, and their key listing reports the KV2 directory entry (`my-key/`) as though it were a key, because the directory filter only exists in upgraded builds.
+| Guarantee | Upgraded behaviour | What an old node does |
+| --- | --- | --- |
+| Check-and-set lifecycle writes | Every KV2 lifecycle mutation (create, enable, disable, tag, schedule/cancel deletion) and every Transit metadata write is a versioned read followed by a check-and-set write, retried on conflict | Writes blind, so it can overwrite a check-and-set commit without any conflict being reported — the lost update the change eliminated |
+| `baseline_version` survives write-back | Preserved | Reads the record without error and drops `baseline_version` on any write-back; the key then resolves pre-versioning envelopes to the current version, which after a rotation is the wrong material |
+| `wrap_budget_reserved` only overestimates | The KV2 record's wrap counter (behind `rustfs_kms_max_key_wrap_operations`) is reserved in blocks and never understates | Drops the field on write-back, regressing the count toward zero. Nothing breaks — the counter is advisory and the next reservation re-establishes a floor — but do not trust a *low* reading taken during or shortly after a mixed-version window |
+| Version-record awareness | Version records under `{prefix}/{key_id}/versions/{N}` are create-only (check-and-set of 0), so two nodes racing a version number produce exactly one creator | Never reads or writes the sub-path, and its key listing reports the KV2 directory entry (`my-key/`) as though it were a key |
 
 ### Windows in which nodes can legitimately disagree
 
@@ -193,79 +155,65 @@ Even with every node on the same build, some state is process-local. These windo
 
 | What can diverge | Bound | Mechanism |
 | --- | --- | --- |
-| Transit key lifecycle state used by the `encrypt` and `generate_data_key` gates | ≤ 300 s (`METADATA_CACHE_TTL`) | Each node caches Transit metadata in process, TTL- and capacity-bounded, with targeted invalidation when a data-path call reports the key is gone server-side. A disable or schedule-deletion performed on one node is enforced on the others within one TTL at the latest, sooner if they hit that signal. |
-| `describe_key` output | One metadata cache TTL: 300 s by default, otherwise whatever `cache_ttl_seconds` was configured with, clamped to 24 h | The manager-level key metadata cache, built from the configured cache settings. This is a reporting cache; the KV2 state gates do not read it. |
-| KV2 key lifecycle state | None | The KV2 backend re-reads the key record from Vault for every lifecycle and data-key operation, so a committed disable is effective on every upgraded node immediately. |
-| Active KMS configuration | One best-effort reload broadcast; unbounded for any peer that did not apply it | See below. |
-
-Builds older than rustfs/rustfs#5520 held the Transit metadata cache with no TTL and no capacity bound. On such a node the divergence window is not 300 seconds but "until the process restarts": it can keep encrypting under a key that another node disabled, indefinitely.
-
-The `describe_key` bound is the only one on that list an operator sets, so compute it rather than assuming the default: the window is the `cache_ttl_seconds` the KMS configure request was given, 300 s when it was omitted, clamped down to 24 h at use if it is larger (clamped rather than rejected, so an oversized setting still starts). Zero is refused outright while caching is enabled. `kms service-status` and the KMS configuration endpoint report the effective, post-clamp value, so the number the admin API shows is the number the cache honours. Note that this is the Transit row's neighbour and not its equal: `METADATA_CACHE_TTL` above is a separate, deliberately non-tunable 300 s, because that cache does gate cryptographic operations.
-
-One upgrade caveat: builds older than rustfs/rustfs#5569 ignored `cache_ttl_seconds` and ran a hardcoded 300 s, while their configure converters persisted 3600 s as the default value. A cluster configured through the admin API before that fix therefore widens its `describe_key` staleness window from an effective 300 s to the 3600 s already stored in `config/kms_config.json`, with no configuration change of its own. Read the reported value back after upgrading instead of assuming it stayed at 300 s. No cryptographic or authorization path widens with it — encrypt, decrypt and data-key generation go straight to the backend and never read this cache.
+| Transit key lifecycle state used by the `encrypt` and `generate_data_key` gates | ≤ `METADATA_CACHE_TTL` (300 s, not tunable — this cache gates cryptographic operations) | Per-node in-process Transit metadata cache, TTL- and capacity-bounded, with targeted invalidation when a data-path call reports the key gone server-side. Builds older than `1.0.0-rc.1` held this cache with no TTL and no capacity bound: on such a node the window is "until the process restarts" |
+| `describe_key` output | One metadata cache TTL: `cache_ttl_seconds` from the KMS configure request, 300 s when omitted, clamped down to 24 h at use (clamped rather than rejected; zero is refused while caching is enabled) | Manager-level key metadata cache; a reporting cache the KV2 state gates never read. `kms service-status` and the configuration endpoint report the effective post-clamp value. Builds older than `1.0.0-rc.1` ignored `cache_ttl_seconds` and ran 300 s while persisting 3600 s as the default, so a cluster configured before then widens to the stored 3600 s on upgrade — read the reported value back |
+| KV2 key lifecycle state | None | The KV2 backend re-reads the key record from Vault for every lifecycle and data-key operation |
+| Active KMS configuration | One best-effort reload broadcast; unbounded for any peer that did not apply it | See below |
 
 ### Configuration changes converge through a best-effort peer reload
 
-`POST /rustfs/admin/v3/kms/configure` and `POST /rustfs/admin/v3/kms/reconfigure` persist the new configuration to cluster storage at `config/kms_config.json`, switch the KMS service **on the node that handled the request**, and then broadcast a reload signal to every peer. A peer that accepts the signal re-reads the persisted configuration and reconfigures itself, so a runtime change normally reaches the whole cluster without any restart. A peer already running that exact configuration treats the signal as a no-op.
+`POST /rustfs/admin/v3/kms/configure` and `/kms/reconfigure` persist the new configuration to cluster storage at `config/kms_config.json`, switch the KMS service **on the node that handled the request**, and broadcast a reload signal once to every peer. A peer that accepts the signal re-reads the persisted configuration and reconfigures itself; a peer already running that configuration treats it as a no-op. Convergence is best effort and the request never fails on account of a peer:
 
-Convergence is best effort by contract, and the request never fails on account of a peer: the local node has already switched, and KMS configuration has no quorum or authoritative holder to roll back to. What that leaves:
+- There is no background retry. A peer that is unreachable, whose build predates the KMS subsystem, or whose reload fails keeps its previous configuration until a later `reconfigure` reaches it or it restarts. For those peers the split is unbounded.
+- The admin response reports success either way, but its message names every peer that did not converge, and the server logs one `kms_peer_config_reload_failed` warning per peer.
+- While a split lasts, both configurations are live: if the change switched backends, or changed the Vault mount or key prefix, nodes write new key material to different places and a key created through one node is invisible to the others.
 
-- The broadcast is sent **once**, with no background retry. A peer that is unreachable, that rejects the signal because its build predates the KMS subsystem, or whose reload itself fails keeps serving its previous configuration until a later `reconfigure` reaches it, or until it restarts and loads the persisted configuration during startup. For those peers the split window is still unbounded.
-- The admin response reports success either way, but its message names every peer that did not converge, and the server logs one `kms_peer_config_reload_failed` warning per peer. Read the message: an operation that reports success can still have left the cluster split.
-- For as long as a split lasts, both configurations are live. If the change switched backends, or changed the Vault mount or key prefix, different nodes write new key material to different places, and a key created through one node is invisible to the others.
-
-`GET /rustfs/admin/v3/kms/service-status` makes the split observable from a single request: it returns a `cluster_config` object holding one redacted configuration fingerprint per node plus a `consistent` flag. `consistent` is true only when every node answered with the same fingerprint — an unreachable peer, a peer whose build reports no fingerprint, and a node with no configuration at all each read as divergent rather than as agreement. Secrets are substituted out before a configuration is fingerprinted, so two nodes on the same backend holding different credentials still fingerprint alike; the field detects a configuration split, not a credential split.
-
-Treat a `configure` or `reconfigure` whose response names unconverged peers as an unfinished cluster-wide operation: re-issue it once those peers are reachable, or restart them.
+`GET /rustfs/admin/v3/kms/service-status` returns a `cluster_config` object holding one redacted configuration fingerprint per node plus a `consistent` flag, true only when every node answered with the same fingerprint (an unreachable peer, a build reporting no fingerprint, and an unconfigured node each read as divergent). Secrets are substituted out before fingerprinting, so the field detects a configuration split, not a credential split. Treat a `configure` or `reconfigure` whose response names unconverged peers as unfinished: re-issue it once those peers are reachable, or restart them.
 
 ### Recommended rolling upgrade order
 
-Follow the node-at-a-time procedure in the [multi-node restart runbook](rolling-restart.md); this adds the KMS-specific sequencing around it.
+Follow the node-at-a-time procedure in the [multi-node restart runbook](rolling-restart.md); this adds the KMS-specific sequencing.
 
 1. **Freeze KMS administrative traffic** for the duration: no key creation, enable, disable, tagging, schedule-deletion, cancel-deletion, rotation, or reconfiguration. Object read and write traffic continues normally.
 2. **Upgrade one node at a time**, waiting for each to report ready before starting the next.
-3. **Verify no node is left behind** before unfreezing. A single old node is enough to reintroduce blind writes and to strip `baseline_version` on its next lifecycle write.
+3. **Verify no node is left behind** before unfreezing. A single old node reintroduces blind writes and strips `baseline_version` on its next lifecycle write.
 4. **Resume administrative traffic.**
-5. **Only then perform the first rotation of any key.** Once the whole cluster understands `master_key_version`, rotation is safe; before that it is not.
-6. **If the KMS configuration was changed at any point**, confirm `cluster_config.consistent` is true in the `service-status` response, and re-issue the change — or restart the node — for every peer still reporting a different fingerprint. A peer whose build predates the reload signal never converges on its own.
+5. **Only then perform the first rotation of any key.**
+6. **If the KMS configuration was changed at any point**, confirm `cluster_config.consistent` is true in the `service-status` response, and re-issue the change — or restart the node — for every peer still reporting a different fingerprint.
 
 ### Do not do these during a mixed-version window
 
-- **Rotate any key.** This is the hard constraint stated above; a rotation is unrecoverable for objects an old node must read.
-- **Issue any KV2 lifecycle write to an old node.** Its blind write can clobber a concurrent check-and-set commit and will drop `baseline_version` from the record.
-- **Create the same key ID from two nodes.** The create path is create-only on upgraded builds, but an old node's blind write does not honor that: the later writer's material wins and every DEK already wrapped with the earlier material becomes permanently unwrappable.
-- **Assume a disable or schedule-deletion took effect cluster-wide.** Old Transit nodes cache lifecycle state without expiry; confirm per node, or restart the old nodes, before treating a key as no longer in use.
-- **Reconfigure the KMS backend and consider it done.** The reload broadcast is exactly what an old build rejects, so during a mixed-version window the change reaches only the node that served it and the already-upgraded peers. Check the response message and `cluster_config.consistent` before assuming otherwise.
+- **Rotate any key.** Unrecoverable for objects an old node must read.
+- **Issue any KV2 lifecycle write to an old node.** Its blind write can clobber a concurrent check-and-set commit and drops `baseline_version`.
+- **Create the same key ID from two nodes.** The create path is create-only on upgraded builds, but an old node's blind write does not honor that: the later writer's material wins and every DEK wrapped with the earlier material becomes permanently unwrappable.
+- **Assume a disable or schedule-deletion took effect cluster-wide.** Old Transit nodes cache lifecycle state without expiry; confirm per node, or restart the old nodes.
+- **Reconfigure the KMS backend and consider it done.** The reload broadcast is exactly what an old build rejects; check the response message and `cluster_config.consistent`.
 - **Delete or prune version records** under `{prefix}/{key_id}/versions/*` for any reason. This is never safe, mixed-version or not; see [Retention and destruction preconditions](#retention-and-destruction-preconditions).
 
 ## Choosing between Vault KV2 and Vault Transit
 
-Use **Vault Transit** (`VaultTransit`) when key material must be cryptographically isolated from anyone holding storage-level read access: Transit keeps key-encryption keys inside Vault and only ever returns ciphertext, and supports server-side key versioning/rotation.
-
-Use **Vault KV2** only when you accept that the Vault ACL on the key path *is* the confidentiality boundary and you want the operational simplicity of a single KV mount.
+Use **Vault Transit** (`VaultTransit`) when key material must be cryptographically isolated from anyone holding storage-level read access: Transit keeps key-encryption keys inside Vault, only ever returns ciphertext, and supports server-side key versioning and rotation. Use **Vault KV2** only when you accept that the Vault ACL on the key path *is* the confidentiality boundary and want the operational simplicity of a single KV mount.
 
 ## AWS KMS: deviations from the shared backend contract
 
-Select it with `RUSTFS_KMS_BACKEND=aws`. Credentials and region resolution are delegated entirely to the standard `aws-config` provider chain (environment, shared profile, container/IMDS role), so RustFS never stores, persists, or redacts AWS credential material of its own. Only two non-credential settings are read: `RUSTFS_KMS_AWS_REGION` and `RUSTFS_KMS_AWS_ENDPOINT_URL`. A plaintext (`http://`) endpoint override would expose every KMS request including plaintext data keys, so it is refused unless the development opt-in is set.
+Select it with `RUSTFS_KMS_BACKEND=aws`. Credentials and region resolution are delegated entirely to the standard `aws-config` provider chain (environment, shared profile, container/IMDS role), so RustFS never stores, persists, or redacts AWS credential material. Only two non-credential settings are read: `RUSTFS_KMS_AWS_REGION` and `RUSTFS_KMS_AWS_ENDPOINT_URL`. A plaintext (`http://`) endpoint override would expose every KMS request including plaintext data keys, so it is refused unless the development opt-in is set.
 
-AWS owns key state, backing-key rotation, and the deletion window, and this backend mirrors none of it locally. That makes four behaviours differ from every RustFS-managed backend. Verify each against your operational assumptions before switching:
+AWS owns key state, backing-key rotation, and the deletion window, and this backend mirrors none of it locally. Four behaviours therefore differ from every RustFS-managed backend:
 
 | Behaviour | RustFS-managed backends | AWS KMS backend |
 | --- | --- | --- |
 | Decryption with a `Disabled` or `PendingDeletion` key | Kept working, so disabling a key never breaks reads of objects already encrypted under it | **Refused by AWS.** Objects encrypted under a key that is later disabled become unreadable until it is re-enabled |
-| Key deletion | Physical deletion available | **No physical delete.** `ScheduleKeyDeletion` is the only removal path; AWS destroys the material when the 7-30 day window elapses. RustFS never destroys AWS-held material, and `force_immediate` is refused |
-| Cancelling a scheduled deletion | Key returns to `Enabled` | Key is left **`Disabled`**; enable it explicitly to make it usable again |
-| Creating a key under a caller-chosen name | The requested name becomes the key id | **Refused.** AWS assigns identifiers and this backend does not manage aliases, so a named create would produce a key unreachable by that name |
+| Key deletion | Physical deletion available | **No physical delete.** `ScheduleKeyDeletion` is the only removal path; AWS destroys the material when the 7-30 day window elapses. `force_immediate` is refused |
+| Cancelling a scheduled deletion | Key returns to `Enabled` | Key is left **`Disabled`**; enable it explicitly |
+| Creating a key under a caller-chosen name | The requested name becomes the key id | **Refused.** AWS assigns identifiers and this backend does not manage aliases |
 
-Two consequences follow from that last row: **SSE-S3 key auto-creation and the synthetic KMS probe are unavailable on this backend**, because both address a key by a name they choose. Pre-create keys in AWS and reference them by AWS key id or ARN.
+Consequences of the last row: **SSE-S3 key auto-creation and the synthetic KMS probe are unavailable on this backend**, because both address a key by a name they choose. Pre-create keys in AWS and reference them by AWS key id or ARN.
 
-The AWS backend is intentionally exempt from `backends::contract_tests::assert_state_machine_contract`. That shared driver assumes that disabled and pending-deletion keys still decrypt, that cancelling deletion returns a key to `Enabled`, and that creation accepts a caller-assigned key name. AWS rejects decryption for the first case, leaves a cancelled key `Disabled`, and assigns key identifiers itself, so running the driver would encode the wrong behavior. The exemption is pinned by the offline `aws_backend_shared_contract_exemption_is_pinned` test in `crates/kms/src/backends/aws.rs`; if AWS changes any of these semantics, integrate the backend into the shared driver and remove this exemption rather than weakening the shared assertions.
+The AWS backend is exempt from `backends::contract_tests::assert_state_machine_contract`, whose assumptions (disabled keys still decrypt, cancel returns to `Enabled`, caller-assigned names) AWS violates. The exemption is pinned by the offline `aws_backend_shared_contract_exemption_is_pinned` test in `crates/kms/src/backends/aws.rs`; if AWS changes any of these semantics, integrate the backend into the shared driver and remove the exemption rather than weakening the shared assertions.
 
-Key versions are opaque. AWS addresses backing keys internally and picks the right one to decrypt with, so RustFS reports `key_version` as 1 and cannot enumerate versions. Rotation uses `RotateKeyOnDemand`, which retains prior backing keys for decryption; AWS's separate automatic yearly rotation is neither enabled nor reported on by RustFS.
+Key versions are opaque: RustFS reports `key_version` as 1 and cannot enumerate versions. Rotation uses `RotateKeyOnDemand`, which retains prior backing keys for decryption; AWS's automatic yearly rotation is neither enabled nor reported on by RustFS.
 
-The KMS admin API accepts the AWS backend as `"backend_type": "AWS"` (aliases `aws`, `aws-kms`, `aws_kms`, `AwsKms`) on `/v3/kms/configure` and `/v3/kms/reconfigure`. The body carries `region` (**required**), and optionally `endpoint_url`, `default_key_id`, and the shared timeout/retry/cache settings. It accepts no credential fields at all — unknown fields are rejected — because every node resolves credentials through its own provider chain.
-
-`region` is mandatory on this path even though `RUSTFS_KMS_AWS_REGION` is optional at startup: the admin configuration is persisted once and replayed on every node, so a request that left the region to each node's ambient chain would let nodes address different regions, and therefore different keys, while reporting an identical configuration. `default_key_id` must be an AWS key id or ARN that already exists — this backend never creates keys by name.
+The KMS admin API accepts the backend as `"backend_type": "AWS"` (aliases `aws`, `aws-kms`, `aws_kms`, `AwsKms`) on `/v3/kms/configure` and `/v3/kms/reconfigure`. The body carries `region` (**required**), and optionally `endpoint_url`, `default_key_id`, and the shared timeout/retry/cache settings; credential fields are rejected as unknown, because every node resolves credentials through its own provider chain. `region` is mandatory even though `RUSTFS_KMS_AWS_REGION` is optional at startup: the admin configuration is replayed on every node, and a request that left the region to each node's ambient chain would let nodes address different regions — and therefore different keys — under an identical configuration. `default_key_id` must be an AWS key id or ARN that already exists.
 
 ## Vault TLS: custom CA and mutual TLS
 
@@ -273,40 +221,34 @@ Both Vault backends (KV2 and Transit) support a private certificate authority an
 
 | Setting | Environment variable | Admin configure field | Meaning |
 | --- | --- | --- | --- |
-| CA bundle | `RUSTFS_KMS_VAULT_CA_CERT` | `ca_cert_path` | Path to a PEM CA bundle trusted for the Vault connection, in addition to nothing else: when set, only this bundle is trusted |
+| CA bundle | `RUSTFS_KMS_VAULT_CA_CERT` | `ca_cert_path` | Path to a PEM CA bundle; when set, only this bundle is trusted |
 | Client certificate | `RUSTFS_KMS_VAULT_CLIENT_CERT` | `client_cert_path` | Path to a PEM client certificate presented to Vault; requires the client key |
 | Client key | `RUSTFS_KMS_VAULT_CLIENT_KEY` | `client_key_path` | Path to the PEM private key matching the client certificate |
 | Skip verification | `RUSTFS_KMS_VAULT_SKIP_TLS_VERIFY` | `skip_tls_verify` | Disables server certificate verification; gated on the insecure development defaults opt-in |
 
-Paths are read on the node applying the configuration, so the files must exist at the same path on every node. The certificate and key must be configured together; configuration validation rejects one without the other, and the files are read and parsed when the backend starts, so a bad path or malformed PEM fails the configuration instead of a later request. The `kms/status` backend summary reports `has_custom_ca` and `has_client_identity` booleans (never the file contents).
-
-The Vault client library would otherwise fall back to the `VAULT_CACERT`, `VAULT_CAPATH`, `VAULT_CLIENT_CERT` and `VAULT_CLIENT_KEY` process environment variables; RustFS always sets the trust roots and identity explicitly - to the configured values or to empty - so stray Vault environment variables cannot splice TLS material into the connection behind the KMS configuration.
+Paths are read on the node applying the configuration, so the files must exist at the same path on every node. Certificate and key must be configured together; the files are read and parsed when the backend starts, so a bad path or malformed PEM fails the configuration rather than a later request. The `kms/status` backend summary reports `has_custom_ca` and `has_client_identity` booleans, never file contents. RustFS always sets the trust roots and identity explicitly — to the configured values or to empty — so the `VAULT_CACERT`, `VAULT_CAPATH`, `VAULT_CLIENT_CERT` and `VAULT_CLIENT_KEY` process environment variables cannot splice TLS material into the connection behind the KMS configuration.
 
 ## Local backend durability and deployment support matrix
 
-The Local backend stores one JSON record per key (`<key_id>.key`) plus an Argon2id salt file (`.master-key.salt`) inside the configured `key_dir`. This section documents which deployments that layout supports and how the backend recovers from a crash or power loss. For where the key material lives and who can read it, see the [backend comparison](#backend-comparison) above.
+The Local backend stores one JSON record per key (`<key_id>.key`) plus an Argon2id salt file (`.master-key.salt`) inside the configured `key_dir`. For where the key material lives and who can read it, see the [backend comparison](#backend-comparison).
 
 ### Positioning
 
-The facts today:
-
-- `Local` is the current default backend (`kms_backend` defaults to `local`).
-- The RustFS Kubernetes operator places the key directory on a PersistentVolumeClaim, so the keys survive pod rescheduling.
-- The in-code documentation labels the backend "for development and testing only", and configuration validation enforces stricter rules outside explicit development mode: a master key is required and `key_dir` must not live under the process temp directory.
+- `Local` is the default backend (`kms_backend` defaults to `local`) and is a development, testing and demo backend; it is not supported for production. Activating a backend whose capabilities report `production_supported: false` logs a `kms_backend_positioning` warning on every start, restart and reconfigure, and the `kms/status` capability matrix carries the same flag. The positioning is a warning, not a gate.
+- Configuration validation enforces stricter rules outside explicit development mode: a master key is required and `key_dir` must not live under the process temp directory.
+- The RustFS Kubernetes operator places the key directory on a PersistentVolumeClaim, so keys survive pod rescheduling.
 - Production multi-node deployments should use the Vault Transit backend.
 
-The backend's positioning is settled (owner decision, 2026-08): `Local` is a development, testing and demo backend and is not supported for production. The runtime now states this itself — activating a backend whose capabilities report `production_supported: false` logs a `kms_backend_positioning` warning on every start, restart and reconfigure, and the `kms/status` capability matrix carries the same flag for consoles and tooling. Existing deployments are not blocked: the positioning is a warning, not a gate. This section describes what the implementation guarantees for those who accept that positioning.
-
 ### Deployment support matrix
 
 | Deployment | Supported | Notes |
 | --- | --- | --- |
-| Local filesystem (ext4, XFS, APFS, ...) | Yes | The commit protocol relies on POSIX `rename`/`hard_link` atomicity and `fsync` durability, which local filesystems provide |
+| Local filesystem (ext4, XFS, APFS, ...) | Yes | The commit protocol relies on POSIX `rename`/`hard_link` atomicity and `fsync` durability |
 | Kubernetes PVC | Yes | Only when the PersistentVolume is backed by a local or block filesystem; this is how the RustFS operator provisions the key directory |
-| NFS or other shared/network filesystems | No | Network filesystems do not reliably provide the atomicity and fsync semantics the commit protocol depends on; an NFS-backed PersistentVolume is this case, not the PVC case above |
-| Multiple RustFS processes sharing one `key_dir` | No | Concurrent key **creation** is linearized (`hard_link` refuses to clobber an existing key), but every other write — status updates, deletion, cancellation — is a read-modify-write with no cross-process lock, so concurrent writers can silently lose updates |
+| NFS or other shared/network filesystems | No | Network filesystems do not reliably provide the atomicity and fsync semantics the protocol depends on; an NFS-backed PersistentVolume is this case |
+| Multiple RustFS processes sharing one `key_dir` | No | Concurrent key **creation** is linearized (`hard_link` refuses to clobber), but every other write is a read-modify-write with no cross-process lock, so concurrent writers can silently lose updates |
 
-Within a single process, per-key write locks serialize read-modify-write updates, so concurrent API calls against one RustFS instance are safe.
+Within a single process, per-key write locks serialize read-modify-write updates.
 
 ### Crash recovery behavior
 
@@ -317,27 +259,23 @@ Every mutation of the key directory uses a durable commit protocol:
 3. The file is published atomically: `rename` to replace an existing file, `hard_link` to create a new one without clobbering.
 4. The parent directory is fsynced so the new directory entry is durable.
 
-Deletion mirrors the tail of the protocol (`remove_file` followed by a parent directory fsync), so a deleted key cannot resurface after power loss. A crash at any step leaves either the complete old state or the complete new state, plus at most an unpublished temp file.
+Deletion mirrors the tail of the protocol (`remove_file` followed by a parent directory fsync). A crash at any step leaves either the complete old state or the complete new state, plus at most an unpublished temp file. On startup the backend:
 
-On startup the backend then:
-
-- **Removes orphaned commit temp files.** The matcher is strict (`<prefix>.tmp-<uuid>`, never anything ending in `.key`), so published key files — including a key the user named to look like a temp file — are never touched. Publishing is atomic, so a matching leftover can only be an unpublished remnant of an interrupted commit.
+- **Removes orphaned commit temp files.** The matcher is strict (`<prefix>.tmp-<uuid>`, never anything ending in `.key`), so published key files are never touched.
 - **Validates every published `.key` file.** A record that fails to decode fails startup rather than being silently skipped.
-- **Guards the salt file.** If `.master-key.salt` is missing but the directory contains keys marked `encrypted-master-key`, initialization fails closed with a configuration error naming the salt path. A regenerated salt derives a different master key and can never decrypt those keys, so the correct recovery is to **restore the salt file (or the whole directory) from backup**, never to let a fresh salt be generated. The guard is equally strict about a record it cannot read or cannot interpret — for example one written by a newer RustFS that names an at-rest protection this build does not implement: such a directory's protection state is unknown, so no replacement salt is generated for it either. Recovery is to restore the salt file, run a build that understands the record, or move the unrecognized file out of `key_dir` after confirming it is not needed. An empty directory, or a legacy directory predating the salt file, still initializes normally.
+- **Guards the salt file.** If `.master-key.salt` is missing but the directory contains keys marked `encrypted-master-key`, initialization fails closed naming the salt path. A regenerated salt derives a different master key and can never decrypt those keys, so the recovery is to **restore the salt file (or the whole directory) from backup**, never to let a fresh salt be generated. The guard is equally strict about a record it cannot read or interpret (for example one written by a newer RustFS naming an at-rest protection this build does not implement): no replacement salt is generated for such a directory either. An empty directory, or a legacy directory predating the salt file, initializes normally.
 
 ### Filesystem permissions and the boundaries the protocol assumes
 
-The key directory is held at `0o700` and every file published into it — key records, the salt, and the files a restore stages and cuts over — is written owner-only. The requested mode is applied and re-read on the open file *before* the content becomes durable, so the process umask cannot widen it, and an unspecified `file_permissions` resolves to owner-only inside the commit protocol rather than at each call site, so no write path can leave it to the umask.
+- The key directory is held at `0o700` and every file published into it — key records, the salt, restore staging — is written owner-only. The requested mode is applied and re-read on the open file before the content becomes durable, so the process umask cannot widen it.
+- A directory wider than `0o700` is **narrowed on every start** (and re-read to confirm), not refused: kubelet creates `emptyDir` at `0o777`, several PVC provisioners `mkdir -m 0777`, and a `--tmpfs` mount lands at `1777`. Only a directory this process cannot secure is fatal. Narrowing is logged with the previous mode whenever it was reachable beyond the owner.
+- Publishing never writes through a symlink: `hard_link` refuses any existing destination (including a dangling symlink) and `rename` replaces the link itself. Startup removes anything wearing a commit-temp name that is not a directory, symlinks included; the protocol only ever creates temps with `create_new`, so such an entry is either its own leftover or something planted.
 
-A directory wider than `0o700` is **narrowed on every start**, and the result is re-read to confirm it took effect. It is not refused: the mode is far more often the platform's than the operator's — kubelet creates an `emptyDir` `0o777`, several PVC provisioners `mkdir -m 0777`, a `--tmpfs` mount lands at `1777` — and refusing would turn each of those into a server that will not start while leaving the exposure in place on the way out. Narrowing removes it. Only a directory this process cannot secure is fatal, because at that point the mode is both dangerous and outside our control. Narrowing is logged with the previous mode whenever it was reachable beyond the owner.
+Two boundaries are **not** verified, and deployments should not assume them:
 
-Publishing never writes through a symlink. `hard_link` refuses any destination that already exists — including a dangling symlink — so a create cannot adopt an inode it did not write, and `rename` replaces the link itself rather than the file it points at. Startup removes anything wearing a commit-temp name that is not a directory, symlinks included; the protocol only ever creates temps with `create_new`, so such an entry is either its own leftover or something planted.
-
-Two boundaries in this area are **not** verified, and deployments should not assume them:
-
-- **Cross-device operations.** The temp file is always created in the destination's own directory, so `rename` and `hard_link` never cross a filesystem and `EXDEV` is unreachable by construction. That invariant is tested; a real cross-device attempt is not, because it needs a second filesystem. The restore staging directory is always `.restore-staging` inside `key_dir`, so this holds by construction unless that subdirectory is separately bind-mounted onto another filesystem — do not do that.
-- **The key directory being replaced mid-commit.** Every path is re-resolved from the directory name rather than held as a directory file descriptor. If `key_dir` is swapped between the `rename` and the parent `fsync`, the fsync lands on the replacement and the new directory entry is never made durable, while the call still reports success. Reaching this requires write access to the key directory's **parent**, which nothing here checks — the mode enforcement above covers `key_dir` itself and says nothing about what encloses it. Keep the parent owner-writable too. Closing this properly means moving the protocol to `renameat`/`linkat` against a held directory descriptor; it is a real gap, recorded as one.
+- **Cross-device operations.** The temp file is always created in the destination's own directory, so `rename` and `hard_link` never cross a filesystem; that invariant is tested, a real cross-device attempt is not. The restore staging directory is always `.restore-staging` inside `key_dir` — do not bind-mount it onto another filesystem.
+- **The key directory being replaced mid-commit.** Paths are re-resolved from the directory name rather than held as a directory descriptor. If `key_dir` is swapped between the `rename` and the parent `fsync`, the fsync lands on the replacement and the call still reports success. Reaching this requires write access to the key directory's **parent**, which nothing checks — keep the parent owner-writable too. Closing it properly means moving to `renameat`/`linkat` against a held descriptor; it is a recorded gap.
 
 ### Backing up the key directory
 
-Back up `key_dir` as a whole, including the hidden `.master-key.salt` file. A key file on its own is not restorable: decrypting it requires the master key derived from the configured `master_key` **and** the persisted salt. Restoring a partial directory — key files without the salt, or the salt without the key files — leaves the backend unable to decrypt, and the salt guard above will (correctly) refuse to start with encrypted keys and no salt. Losing the salt file with no backup means every key encrypted under it is unrecoverable.
+Back up `key_dir` as a whole, including the hidden `.master-key.salt` file. A key file on its own is not restorable: decrypting it requires the master key derived from the configured `master_key` **and** the persisted salt. Restoring a partial directory leaves the backend unable to decrypt, and the salt guard will (correctly) refuse to start. Losing the salt file with no backup means every key encrypted under it is unrecoverable. The rehearsal procedure is in the [KMS disaster-recovery drill](kms-disaster-recovery-drill.md).