fix(heal): preserve decode repair signals and retry startup quorum (#8067 )

* fix(startup): retry transient bucket metadata quorum failures * fix(ecstore): retain recovered shard damage for read repair * fix(ecstore): reconnect missing read sources before GET * test(heal): prove exact marker replay preserves history * fix(ecstore): fence reconnect retries at the deadline Reject expired reconnect waiters before dispatch and start cooldown on sweep completion, including timeout and cancellation. Cover exact-deadline admission independently of cooldown and preserve the original strict RPC-count regression.
fix(ci): publish version-tagged preview Docker images (#8068 )
2026-11-02 17:13:37 +00:00 · 2026-09-22 14:58:31 +00:00 · 2026-09-22 13:06:23 +00:00 · 2026-09-22 12:16:40 +00:00 · 2026-09-22 12:15:20 +00:00 · 2026-09-22 19:23:41 +08:00
106 changed files with 8803 additions and 787 deletions
@@ -101,6 +101,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### SSE and on-disk storage invariants

 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
+- `GHSA-wqmc-vjgv-jrpw`: SSE-C persisted and listed the plaintext MD5 as the ETag, allowing list-only users to confirm or brute-force low-entropy content and correlate equal plaintext across keys. Lesson: encrypted-object ETags must not expose deterministic plaintext fingerprints; protect them with per-object key material and keep listing, conditional, copy, and multipart semantics consistent.

 ### Object Lock and retention invariants

@@ -126,7 +127,7 @@ rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonical
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
 rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
-rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+rg -n "HashReader|EncryptReader|try_resolve_etag|ETag|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
 rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
 rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
@@ -145,5 +146,5 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
+- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
 - Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
@@ -29,6 +29,7 @@ script-tests: ## Run shell script tests
 	./scripts/test_entrypoint_credentials.sh
 	./scripts/test_internode_grpc_ab_bench.sh
 	./scripts/test_object_batch_bench_enhanced.sh
+	./scripts/test_package_service_scripts.sh
 	./scripts/test_hotpath_warp_ab_gate.sh
 	./scripts/test_hotpath_warp_abba.sh
 	./scripts/test_scanner_validation_harness.sh
@@ -24,8 +24,11 @@ on:
      - '.github/actions/**'
      - '.github/workflows/**'
      - 'scripts/release/create_or_update_release.sh'
+      - 'scripts/release/package_service_scripts.sh'
      - 'scripts/release/package_versions.sh'
+      - 'scripts/test_package_service_scripts.sh'
      - 'scripts/test_package_versions.sh'
+      - 'scripts/test_docker_workflow.py'
      - 'scripts/security/check_performance_ab_workflow.sh'
      - 'scripts/security/check_preview_release_workflow.sh'
      - 'scripts/security/check_tier_artifact_workflow.sh'
@@ -40,8 +43,11 @@ on:
      - '.github/actions/**'
      - '.github/workflows/**'
      - 'scripts/release/create_or_update_release.sh'
+      - 'scripts/release/package_service_scripts.sh'
      - 'scripts/release/package_versions.sh'
+      - 'scripts/test_package_service_scripts.sh'
      - 'scripts/test_package_versions.sh'
+      - 'scripts/test_docker_workflow.py'
      - 'scripts/security/check_performance_ab_workflow.sh'
      - 'scripts/security/check_preview_release_workflow.sh'
      - 'scripts/security/check_tier_artifact_workflow.sh'
@@ -158,6 +164,9 @@ jobs:
      - name: Check package version contract
        run: ./scripts/test_package_versions.sh

+      - name: Check package service lifecycle contract
+        run: ./scripts/test_package_service_scripts.sh
+
  dependency-review:
    name: Dependency Review
    runs-on: ubuntu-latest
@@ -818,9 +818,7 @@ jobs:

          echo ""
          echo "🐳 Docker Images:"
-          if [[ "$BUILD_TYPE" == "preview" ]]; then
-            echo "⏭️  Preview tags do not publish Docker images"
-          elif [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
+          if [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
            echo "⏭️  Docker image build was skipped (binary only build)"
          elif [[ "$BUILD_STATUS" == "success" ]]; then
            echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -47,7 +47,7 @@ on:
        default: true
        type: boolean
      version:
-        description: "Version to build (latest for stable release, or specific version like v1.0.0, v1.0.0-alpha1)"
+        description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
        required: false
        default: "latest"
        type: string
@@ -82,8 +82,7 @@ jobs:
      github.event_name == 'workflow_dispatch' ||
      (github.event.workflow_run.conclusion == 'success' &&
       github.event.workflow_run.event == 'push' &&
-       github.event.workflow_run.head_branch != 'main' &&
-       !contains(github.event.workflow_run.head_branch, '-preview'))
+       github.event.workflow_run.head_branch != 'main')
    runs-on: ubuntu-latest
    timeout-minutes: 30
    outputs:
@@ -197,7 +196,14 @@ jobs:
                  version="${version#v}"
                fi

-                if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
+                if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
+                  build_type="preview"
+                  is_prerelease=true
+                  echo "🔍 Building Docker image for preview: $version (version tags only)"
+                elif [[ "$version" == *"-preview"* ]]; then
+                  echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
+                  exit 1
+                elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
                  build_type="prerelease"
                  is_prerelease=true
                  # Pre-GA policy: prereleases update latest until the first stable tag exists.
@@ -249,11 +255,14 @@ jobs:
                echo "🚀 Building with latest stable release version"
                ;;
              *-preview*)
+                if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
+                  echo "❌ Invalid preview version: $input_version" >&2
+                  exit 1
+                fi
                build_type="preview"
                is_prerelease=true
-                should_build=false
-                should_push=false
-                echo "⏭️  Preview tags do not publish Docker images"
+                version="${input_version#v}"
+                echo "🔍 Building with preview version: $version (version tags only)"
                ;;
              # Prerelease versions (must match first, more specific)
              v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
@@ -542,6 +551,54 @@ jobs:
          if-no-files-found: error
          retention-days: 30

+  # Docker Hub's overview is a separate `full_description` field that
+  # `docker push` never touches, so it drifted from README.md and kept
+  # linking to pages that no longer exist (#7995). Republish it from the
+  # commit the images were built from, only when images were pushed.
+  sync-dockerhub-description:
+    name: Sync Docker Hub Description
+    needs: [ build-check, build-docker ]
+    if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          persist-credentials: false
+          ref: ${{ needs.build-check.outputs.source_ref }}
+          sparse-checkout: README.md
+          sparse-checkout-cone-mode: false
+
+      - name: Sync Docker Hub description
+        uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
+        with:
+          username: ${{ env.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+          repository: ${{ env.REGISTRY_DOCKERHUB }}
+          readme-filepath: ./README.md
+          # README.md links to docs/ and CONTRIBUTING.md by relative path;
+          # rewrite those to github.com URLs so they resolve on Docker Hub.
+          enable-url-completion: true
+
+      # Docker Hub caps `full_description` at 25,000 bytes and the action
+      # truncates to fit with only a warning, which would publish a README
+      # cut off mid-sentence. README.md is already near the cap, so read the
+      # published overview back and fail if it hit the limit.
+      - name: Check published description was not truncated
+        env:
+          REPOSITORY: ${{ env.REGISTRY_DOCKERHUB }}
+        run: |
+          limit=25000
+          published=$(curl -sSf "https://hub.docker.com/v2/repositories/${REPOSITORY}/" \
+            | jq -j '.full_description' | wc -c | tr -d ' ')
+          echo "Published overview: ${published} bytes (Docker Hub limit: ${limit})"
+          # Truncation lands within one UTF-8 character of the cap.
+          if [ "$published" -ge $((limit - 4)) ]; then
+            echo "::error file=README.md::Docker Hub overview was truncated at ${limit} bytes; trim README.md before the next release."
+            exit 1
+          fi
+
  # Docker build summary
  docker-summary:
    name: Docker Build Summary
@@ -563,6 +620,10 @@ jobs:
          echo ""

          case "$BUILD_TYPE" in
+            "preview")
+              echo "🔍 Preview Docker image has been built with ${VERSION} tags"
+              echo "⏭️  Preview images do not update latest or prerelease channels"
+              ;;
            "release")
              echo "🚀 Release Docker image has been built with ${VERSION} tags"
              echo "✅ This image is ready for production use"
@@ -117,10 +117,15 @@ jobs:

          cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
+          # See https://docs.rustfs.com for more information
+          # RUSTFS_ACCESS_KEY=<changeme>
+          # RUSTFS_SECRET_KEY=<changeme>
          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
+          # RUSTFS_ADDRESS=":9000"
+          # RUSTFS_CONSOLE_ADDRESS=":9001"
+          # RUSTFS_CONSOLE_ENABLE=true
+          # RUSTFS_OBS_LOGGER_LEVEL=error
+          # RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
          ENVEOF

          # dpkg versions must start with a digit and cannot contain hyphens;
@@ -146,37 +151,13 @@ jobs:
          /etc/default/rustfs
          CONFFILES

-          cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
-          POSTINST
+          ./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
          chmod 755 "${PKG_DIR}/DEBIAN/postinst"

-          cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
+          ./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
          chmod 755 "${PKG_DIR}/DEBIAN/prerm"

-          cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
+          ./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
          chmod 755 "${PKG_DIR}/DEBIAN/postrm"

          cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
@@ -214,10 +195,15 @@ jobs:
          mkdir -p ./tmp-pkg/etc/default
          cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
+          # See https://docs.rustfs.com for more information
+          # RUSTFS_ACCESS_KEY=<changeme>
+          # RUSTFS_SECRET_KEY=<changeme>
          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
+          # RUSTFS_ADDRESS=":9000"
+          # RUSTFS_CONSOLE_ADDRESS=":9001"
+          # RUSTFS_CONSOLE_ENABLE=true
+          # RUSTFS_OBS_LOGGER_LEVEL=error
+          # RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
          ENVEOF

          fpm -s dir -t rpm \
@@ -231,35 +217,13 @@ jobs:
            --description "High-performance distributed object storage" \
            --url "https://rustfs.com" \
            --license "Apache-2.0" \
-            --after-install <(cat << 'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTINST
-            ) \
-            --before-remove <(cat << 'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
-            ) \
-            --after-remove <(cat << 'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
-            ) \
+            --before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
+            --before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
+            --after-install <(./scripts/release/package_service_scripts.sh after-install) \
+            --after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
+            --before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
+            --after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
+            --rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
            --config-files /etc/default/rustfs \
            "rustfs-nightly-${DEB_DATE}/usr/bin/rustfs=/usr/bin/rustfs" \
            ./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
@@ -328,10 +328,15 @@ jobs:

          cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
+          # See https://docs.rustfs.com for more information
+          # RUSTFS_ACCESS_KEY=<changeme>
+          # RUSTFS_SECRET_KEY=<changeme>
          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
+          # RUSTFS_ADDRESS=":9000"
+          # RUSTFS_CONSOLE_ADDRESS=":9001"
+          # RUSTFS_CONSOLE_ENABLE=true
+          # RUSTFS_OBS_LOGGER_LEVEL=error
+          # RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
          ENVEOF

          cat > "${PKG_DIR}/DEBIAN/control" << EOF
@@ -354,37 +359,13 @@ jobs:
          /etc/default/rustfs
          CONFFILES

-          cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
-          POSTINST
+          ./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
          chmod 755 "${PKG_DIR}/DEBIAN/postinst"

-          cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
+          ./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
          chmod 755 "${PKG_DIR}/DEBIAN/prerm"

-          cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
+          ./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
          chmod 755 "${PKG_DIR}/DEBIAN/postrm"

          cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
@@ -422,10 +403,15 @@ jobs:
          mkdir -p ./tmp-pkg/etc/default
          cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
+          # See https://docs.rustfs.com for more information
+          # RUSTFS_ACCESS_KEY=<changeme>
+          # RUSTFS_SECRET_KEY=<changeme>
          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
+          # RUSTFS_ADDRESS=":9000"
+          # RUSTFS_CONSOLE_ADDRESS=":9001"
+          # RUSTFS_CONSOLE_ENABLE=true
+          # RUSTFS_OBS_LOGGER_LEVEL=error
+          # RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
          ENVEOF

          fpm -s dir -t rpm \
@@ -439,35 +425,13 @@ jobs:
            --description "High-performance distributed object storage" \
            --url "https://rustfs.com" \
            --license "Apache-2.0" \
-            --after-install <(cat <<'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTINST
-            ) \
-            --before-remove <(cat <<'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
-            ) \
-            --after-remove <(cat <<'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
-            ) \
+            --before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
+            --before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
+            --after-install <(./scripts/release/package_service_scripts.sh after-install) \
+            --after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
+            --before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
+            --after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
+            --rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
            --config-files /etc/default/rustfs \
            ./bin/rustfs=/usr/bin/rustfs \
            ./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
@@ -0,0 +1,221 @@
+# RustFS Fault-Tolerance (degradation) Matrix Sweep
+#
+# Exhaustive topology x EC outage sweep, complementary to the scenario suite
+# (rustfs-fault-tolerance-test.yml). For each topology (nodes x drives per
+# node) and each EC level P in 1..N*drives/2 the runner installs that exact
+# cluster shape and drives progressive node outages plus recovery:
+#
+#   M-<topo>EC<P>-full   full-capacity reads/writes must succeed
+#   M-<topo>EC<P>-n<k>   with k nodes online: reads succeed iff the online
+#                        drives cover the read quorum (N-P), writes iff they
+#                        cover the write quorum (N-P+1); probes below quorum
+#                        assert the expected outage
+#   M-<topo>EC<P>-rec    after restoring every node, reads/writes succeed
+#
+# 10 topologies x EC levels = 38 combinations, 320 cases. The full sweep runs
+# 3-5 hours and is dispatched on demand (e.g. after a nightly chain goes
+# green), not nightly. Case ids and expectations come from product source
+# (default_parity_count, erasure set sizing) via rustfs/auto-testing.
+
+name: RustFS Fault-Tolerance Matrix
+
+on:
+  workflow_dispatch:
+    inputs:
+      package_url:
+        description: 'Direct .deb URL. Required unless the nightly default is wanted.'
+        required: false
+        type: string
+      auto_testing_ref:
+        description: 'auto-testing branch, tag, or SHA (defaults to main)'
+        required: false
+        default: main
+        type: string
+      matrix_topo:
+        description: 'Topology filter like 4x2; empty sweeps all ten topologies'
+        required: false
+        default: ''
+        type: string
+      matrix_ec:
+        description: 'EC/parity filter; empty sweeps every EC level'
+        required: false
+        default: ''
+        type: string
+      cleanup_before:
+        description: 'Reset the nodes before the test (DESTROYS existing data/config)'
+        type: boolean
+        default: true
+      cleanup_after:
+        description: 'Reset the nodes after the test (DESTROYS test data/config)'
+        type: boolean
+        default: true
+
+permissions:
+  contents: read
+
+# The suite stops services and revokes drive access on the shared fleet; only one
+# functional suite may touch the environment at a time.
+concurrency:
+  group: rustfs-shared-functional-tests-v2
+  cancel-in-progress: false
+
+defaults:
+  run:
+    shell: bash
+
+env:
+  RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
+  RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
+  RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
+  RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
+  PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+  RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
+
+jobs:
+  fault-tolerance-matrix:
+    runs-on: smoke-testing
+    # The full 38-combination sweep takes 3-5h; 360 is the runner job ceiling.
+    timeout-minutes: 360
+    steps:
+      - name: Initialize functional evidence
+        id: evidence
+        run: |
+          set -euo pipefail
+          umask 077
+          FUNCTIONAL_ARTIFACTS_DIR="${RUNNER_TEMP}/rustfs-fault-tolerance-matrix-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
+          mkdir -- "${FUNCTIONAL_ARTIFACTS_DIR}" "${FUNCTIONAL_ARTIFACTS_DIR}-scratch" "${FUNCTIONAL_ARTIFACTS_DIR}/evidence"
+          {
+            printf 'FUNCTIONAL_ARTIFACTS_DIR=%s\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
+            printf 'LOG_FILE=%s/suite.log\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
+            printf 'REPORT_FILE=%s/report.md\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
+            printf 'RESULTS_FILE=%s/results.json\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
+            printf 'EVIDENCE_DIR=%s/evidence\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
+            printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
+          } >> "${GITHUB_ENV}"
+
+      - name: Checkout auto-testing scripts
+        timeout-minutes: 5
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          repository: rustfs/auto-testing
+          ref: ${{ inputs.auto_testing_ref }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+          path: auto-testing
+          persist-credentials: false
+
+      - name: Record auto-testing revision
+        run: |
+          set -euo pipefail
+          AUTO_TESTING_SHA="$(git -C auto-testing rev-parse HEAD)"
+          printf 'AUTO_TESTING_SHA=%s\n' "${AUTO_TESTING_SHA}" >> "${GITHUB_ENV}"
+          echo "auto-testing revision: ${AUTO_TESTING_SHA}"
+
+      - name: Show environment
+        run: |
+          uname -a
+          jq --version
+          aws --version
+          echo "auto-testing: ${AUTO_TESTING_SHA}"
+          df -h /data | tail -1
+
+      - name: Cleanup environment (before)
+        timeout-minutes: 5
+        if: ${{ inputs.cleanup_before != 'false' }}
+        run: |
+          ./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}"
+
+      - name: Run matrix sweep (topology x EC outage combinations)
+        timeout-minutes: 350
+        id: test
+        # No continue-on-error: an UNEXPECTED verdict (a probe that should
+        # fail by quorum math but passes, or vice versa) is the finding this
+        # sweep exists to surface, and the red run is the signal until matrix
+        # runs get backlog integration like the scenario suite has.
+        env:
+          MATRIX_EC: ${{ inputs.matrix_ec }}
+        run: |
+          set -euo pipefail
+          ARGS=(--matrix -y --package-url "${{ inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}" --results-file "${RESULTS_FILE}")
+          if [ -n "${{ inputs.matrix_topo }}" ]; then
+            # Trailing value form: consumed as `--matrix-topo <topo> [ec]`.
+            ARGS+=(--matrix-topo "${{ inputs.matrix_topo }}")
+            if [ -n "${MATRIX_EC}" ]; then
+              ARGS+=("${MATRIX_EC}")
+            fi
+          fi
+          ./auto-testing/rustfs-fault-tolerance-test.sh "${ARGS[@]}"
+
+      - name: Generate report
+        timeout-minutes: 2
+        if: ${{ always() && steps.evidence.outcome == 'success' }}
+        run: |
+          set -euo pipefail
+          {
+            echo "# RustFS fault-tolerance matrix report"
+            echo ""
+            echo "- Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+            echo "- Package: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
+            echo "- auto-testing: \`${AUTO_TESTING_SHA}\`"
+            echo "- Topology filter: ${{ inputs.matrix_topo || 'all (10 topologies)' }}"
+            echo "- EC filter: ${{ inputs.matrix_ec || 'all levels' }}"
+            echo ""
+            echo "## Per-probe results"
+            echo ""
+            echo '```'
+            grep -E '^FT-(CASE|COMPLETE|SUMMARY|REPORT)' "${LOG_FILE}" || echo "(no FT-CASE lines found)"
+            echo '```'
+            echo ""
+            echo "## Health snapshots"
+            echo ""
+            for f in "${FUNCTIONAL_ARTIFACTS_DIR}"/evidence/*.code; do
+              [ -e "${f}" ] || continue
+              printf '%s -> %s\n' "$(basename "${f}" .code)" "$(cat "${f}")"
+            done
+          } > "${REPORT_FILE}"
+          cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
+
+          # An unfiltered sweep must land exactly 320 cases (38 topology x EC
+          # combinations at 2*(nodes+1) cases each); filtered sweeps are held
+          # to internal consistency. Missing/harness-error rows always fail.
+          EXPECTED_TOTAL=""
+          if [ -z "${{ inputs.matrix_topo }}" ] && [ -z "${MATRIX_EC}" ]; then
+            EXPECTED_TOTAL=320
+          fi
+          jq -e --arg testing_sha "${AUTO_TESTING_SHA}" --argjson expected_total "${EXPECTED_TOTAL:-null}" '
+            .schema_version == 1 and
+            .auto_testing_sha == $testing_sha and
+            .complete == true and
+            .counts.expected == ($expected_total // .counts.expected) and
+            .counts.expected == .counts.seen and
+            .counts.harness_errors == 0 and
+            (.expected_cases | length) == .counts.expected and
+            (.expected_cases | unique | length) == .counts.expected and
+            (.seen_cases | length) == .counts.seen and
+            (.missing_cases | length) == 0 and
+            (.harness_errors | length) == 0 and
+            (.cases | length) == .counts.seen and
+            ([.cases[].case_id] | unique | length) == .counts.seen and
+            ([.cases[].case_id] | sort) == (.expected_cases | sort) and
+            all(.cases[]; (.case_id | startswith("M-")) and (.verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED")) and
+            .counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
+            .counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
+          ' "${RESULTS_FILE}" >/dev/null
+
+      - name: Upload test logs & evidence
+        timeout-minutes: 2
+        if: ${{ always() && steps.evidence.outcome == 'success' }}
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: rustfs-fault-tolerance-matrix-${{ github.run_id }}-${{ github.run_attempt }}
+          path: |
+            ${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/report.md
+            ${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/results.json
+            ${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/suite.log
+            ${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/evidence/
+          if-no-files-found: warn
+
+      - name: Cleanup environment (after)
+        timeout-minutes: 5
+        if: ${{ always() && inputs.cleanup_after != 'false' }}
+        run: |
+          ./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}" || true
@@ -149,7 +149,7 @@ jobs:
          CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
        run: python3 scripts/prepare_functional_package.py prepare

-      - name: Run fault-tolerance scenarios (A, B, C, C2, D)
+      - name: Run fault-tolerance scenarios (A, B, C, C2, D, E)
        timeout-minutes: 45
        id: test
        # Standalone case failures are reported by the backlog manager.
@@ -194,21 +194,21 @@ jobs:
          jq -e --arg testing_sha "${AUTO_TESTING_SHA}" '
            .schema_version == 1 and
            .auto_testing_sha == $testing_sha and
-            .selected_scenarios == ["A", "B", "C", "C2", "D"] and
+            .selected_scenarios == ["A", "B", "C", "C2", "D", "E"] and
            .complete == true and
-            .counts.expected == 38 and
-            .counts.seen == 38 and
+            .counts.expected == 44 and
+            .counts.seen == 44 and
            .counts.harness_errors == 0 and
-            (.expected_cases | length) == 38 and
-            (.expected_cases | unique | length) == 38 and
-            (.seen_cases | length) == 38 and
-            (.seen_cases | unique | length) == 38 and
+            (.expected_cases | length) == 44 and
+            (.expected_cases | unique | length) == 44 and
+            (.seen_cases | length) == 44 and
+            (.seen_cases | unique | length) == 44 and
            (.seen_cases | sort) == (.expected_cases | sort) and
            (.missing_cases | length) == 0 and
            (.harness_errors | length) == 0 and
-            (.cases | length) == 38 and
+            (.cases | length) == 44 and
            ([.cases[].case_id] | sort) == (.expected_cases | sort) and
-            ([.cases[].case_id] | unique | length) == 38 and
+            ([.cases[].case_id] | unique | length) == 44 and
            all(.cases[]; .verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED") and
            .counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
            .counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
@@ -54,7 +54,10 @@ jobs:
      - name: Resolve published candidate
        id: candidate
        env:
-          GH_TOKEN: ${{ github.token }}
+          # The staleness probe in resolve_functional_candidate.py reads the
+          # private rustfs/auto-testing repo; github.token cannot, and a probe
+          # failure silently keeps a stale testing_sha pinned for the chain.
+          GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
          BUILD_RUN_ID: ${{ inputs.build_run_id }}
          BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }}
          CHAIN_OUTPUT: ${{ runner.temp }}/chain-candidate.json
@@ -1984,9 +1984,9 @@ dependencies = [

 [[package]]
 name = "cc"
-version = "1.4.6"
+version = "1.4.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a3eb0f42d6c360dc3f8a821f6bf2fdea7f72bfd36b3076eb0e6d1e9e0752fff4"
+checksum = "54413ede23c2daf518f35156dfde027feb2374004d63bd497f983c8db9c0e313"
 dependencies = [
 "find-msvc-tools",
 "jobserver",
@@ -4410,9 +4410,9 @@ dependencies = [

 [[package]]
 name = "find-msvc-tools"
-version = "0.1.12"
+version = "0.1.13"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3e0f1c7c3a72c66fd80abe965175f7523475c0489a87d3ff9d6e8c87d87a9d2d"
+checksum = "ef25905e51abafe4dcea6c15fec58c57b601cdbd0ee53d22ea1d3016c587d39b"

 [[package]]
 name = "findshlibs"
@@ -10408,6 +10408,8 @@ dependencies = [
 "opentelemetry_sdk",
 "percent-encoding",
 "pyroscope",
+ "rcgen",
+ "reqwest",
 "rustfs-audit",
 "rustfs-common",
 "rustfs-config",
@@ -10421,6 +10423,8 @@ dependencies = [
 "rustfs-storage-api",
 "rustfs-utils",
 "rustix",
+ "rustls",
+ "rustls-pki-types",
 "serde",
 "serde_json",
 "sysinfo",
@@ -10428,6 +10432,7 @@ dependencies = [
 "tempfile",
 "thiserror 2.0.20",
 "tokio",
+ "tokio-rustls",
 "tokio-util",
 "tracing",
 "tracing-appender",
@@ -11258,9 +11263,8 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"

 [[package]]
 name = "s3s"
-version = "0.16.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "87d22507a8ebd0ea64972c76dfedbc477418dc70afd98bfce928263894067d2f"
+version = "0.16.0"
+source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
 dependencies = [
 "arc-swap",
 "arrayvec",
@@ -11278,7 +11282,6 @@ dependencies = [
 "http 1.5.0",
 "http-body 1.1.0",
 "http-body-util",
- "httparse",
 "hyper",
 "itoa",
 "jiff",
@@ -11291,6 +11294,7 @@ dependencies = [
 "pin-project-lite",
 "quick-xml",
 "regex",
+ "s3s-multipart",
 "s3s-rfc2047",
 "s3s-sigv2",
 "s3s-sigv4",
@@ -11315,11 +11319,23 @@ dependencies = [
 "zeroize",
 ]

+[[package]]
+name = "s3s-multipart"
+version = "0.17.0-alpha.1"
+source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
+dependencies = [
+ "arrayvec",
+ "bytes",
+ "futures-core",
+ "httparse",
+ "memchr",
+ "thiserror 2.0.20",
+]
+
 [[package]]
 name = "s3s-rfc2047"
-version = "0.16.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f9d3e6b36be7cca3da377b4cabcd668ffe5b16c3b73a83f7a8f7b63ff6a178ee"
+version = "0.16.0"
+source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
 dependencies = [
 "base64-simd",
 "thiserror 2.0.20",
@@ -11327,9 +11343,8 @@ dependencies = [

 [[package]]
 name = "s3s-sigv2"
-version = "0.16.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d96880b5ad8d639c9224155126741708aeb62ad3682e0476d2fd14290af808ef"
+version = "0.16.0"
+source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
 dependencies = [
 "base64-simd",
 "hmac 0.13.0",
@@ -11341,9 +11356,8 @@ dependencies = [

 [[package]]
 name = "s3s-sigv4"
-version = "0.16.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fed605cbaaa1cf4d82e8ee15098fb3e42d49c92bb6215eb1359f63bfc0c6b020"
+version = "0.16.0"
+source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
 dependencies = [
 "arrayvec",
 "base64-simd",
@@ -313,8 +313,8 @@ rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { version = "0.16.1", features = ["minio"] }
-s3s-sigv4 = { version = "0.16.1" }
+s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6", features = ["minio"] }
+s3s-sigv4 = { git = "https://github.com/s3s-project/s3s.git", rev = "50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6" }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
@@ -30,6 +30,9 @@ pub const ENV_OBS_ENDPOINT_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TIMEOUT_M
 pub const ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS";
 pub const ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS";
 pub const ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS";
+/// Absolute path to a PEM bundle whose certificates are added to the system
+/// trust roots used by OTLP/HTTP exporters.
+pub const ENV_OBS_TLS_CA_FILE: &str = "RUSTFS_OBS_TLS_CA_FILE";
 pub const ENV_OBS_PROFILING_ENDPOINT: &str = "RUSTFS_OBS_PROFILING_ENDPOINT";
 pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
 pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
@@ -55,6 +58,13 @@ pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
 pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
 pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";

+/// Span lifecycle events emitted by the JSON log layers (`none`|`close`|`full`).
+///
+/// Defaults to `none`: a record per span lifecycle multiplies log volume on
+/// object hot paths, and emitting the ancestor span list on every record
+/// produced multi-megabyte lines that journald truncated (backlog#2642).
+pub const ENV_OBS_LOG_SPAN_EVENTS: &str = "RUSTFS_OBS_LOG_SPAN_EVENTS";
+
 /// Log cleanup related configurations
 pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES";
 pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
@@ -99,6 +109,8 @@ pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
 pub const DEFAULT_OBS_LOG_MATCH_MODE_PREFIX: &str = "prefix";
 pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";

+pub const DEFAULT_OBS_LOG_SPAN_EVENTS: &str = "none";
+
 /// Default values for observability configuration
 // ### Supported Environment Values
 // - `production` - Secure file-only logging
@@ -141,6 +153,7 @@ mod tests {
        assert_eq!(ENV_OBS_LOG_FILENAME, "RUSTFS_OBS_LOG_FILENAME");
        assert_eq!(ENV_OBS_LOG_ROTATION_TIME, "RUSTFS_OBS_LOG_ROTATION_TIME");
        assert_eq!(ENV_OBS_LOG_KEEP_FILES, "RUSTFS_OBS_LOG_KEEP_FILES");
+        assert_eq!(ENV_OBS_LOG_SPAN_EVENTS, "RUSTFS_OBS_LOG_SPAN_EVENTS");
        assert_eq!(ENV_OBS_TRACES_EXPORT_ENABLED, "RUSTFS_OBS_TRACES_EXPORT_ENABLED");
        assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
        assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
@@ -178,6 +191,7 @@ mod tests {
        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP, "gzip");
        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD, "zstd");
        assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE_PREFIX, "prefix");
+        assert_eq!(DEFAULT_OBS_LOG_SPAN_EVENTS, "none");
        assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE, "suffix");
        assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, "zstd");
    }
@@ -184,7 +184,7 @@ async fn four_node_health_inventory_metrics_and_audit_delivery_are_consistent()
        let (status, metrics_body) = admin_request(
            &node.url,
            Method::GET,
-            "/rustfs/admin/v3/metrics?n=1&by-host=true&by-disk=true",
+            "/rustfs/admin/v3/realtime?n=1&by-host=true&by-disk=true",
            None,
            &dist.cluster.access_key,
            &dist.cluster.secret_key,
@@ -257,7 +257,7 @@ async fn node_admin_body(dist: &DistCluster, node: usize, path: &str) -> TestRes
 }

 async fn http_put_counts(dist: &DistCluster, node: usize) -> TestResult<[u64; 2]> {
-    let body = node_admin_body(dist, node, "/rustfs/admin/v3/metrics?types=512&by-host=true&n=1").await?;
+    let body = node_admin_body(dist, node, "/rustfs/admin/v3/realtime?types=512&by-host=true&n=1").await?;
    let sample: RealtimeMetrics = serde_json::from_str(body.lines().next().ok_or("empty HTTP metrics stream")?)?;
    assert!(sample.errors.is_empty(), "HTTP metrics returned errors: {:?}", sample.errors);
    let http = sample.aggregated.http.ok_or("HTTP metrics missing at WARN log level")?;
@@ -504,6 +504,13 @@ pub mod notification {
    };
 }

+pub mod integrity {
+    pub use crate::services::integrity::{
+        IntegrityError, InventoryItem, InventoryPage, ItemRequest, ItemResult, ItemState, Job, JobMode, JobRequest, JobState,
+        Protection, Readiness, control_job, create_job, get_job, inventory, readiness, resume_job,
+    };
+}
+
 pub mod object {
    pub use crate::object_api::{
        BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
@@ -1291,6 +1291,28 @@ pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_in(
    }
 }

+/// Inspect all migration-relevant settings while the caller holds an Object Lock
+/// snapshot's lifecycle and metadata transaction guards. Cached settings are not
+/// sufficient to authorize a direct storage writer that cannot apply S3 defaults.
+pub(crate) async fn integrity_migration_metadata_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bucket: &str,
+) -> Result<Arc<BucketMetadata>> {
+    let sys = bucket_metadata_sys_of(ctx)?.read().await.clone();
+    match sys
+        .read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
+        .await?
+    {
+        BucketMetadataAuthority::Authoritative(metadata)
+            if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
+        {
+            Ok(metadata)
+        }
+        BucketMetadataAuthority::MissingBucket => Err(Error::BucketNotFound(bucket.to_string())),
+        _ => Err(Error::other("migration requires authoritative bucket metadata")),
+    }
+}
+
 /// Re-read the quota configuration and bucket incarnation from the same
 /// authoritative metadata blob while the caller holds the bucket metadata
 /// transaction read lock.
@@ -2141,7 +2141,7 @@ impl RemoteDisk {
                let file_info_bin = encode_file_info_msgpack(fi)?;
                let mut client = self.get_client().await?;
                let mut request = Request::new(RenameDataRequest {
-                    bucket_incarnation_id: crate::store::bucket_heal_scope(dst_volume)
+                    bucket_incarnation_id: crate::store::bucket_heal_scope_for_object(dst_volume, dst_path)
                        .map(|scope| scope.incarnation.as_bytes().to_vec().into())
                        .unwrap_or_default(),
                    disk: self.endpoint.to_string(),
@@ -4308,6 +4308,13 @@ mod tests {
        object_read_all_disks: Arc<StdMutex<Vec<String>>>,
        format_data: Bytes,
        read_all_data: Bytes,
+        next_format_read_gate: Arc<StdMutex<Option<Arc<TestFormatReadGate>>>>,
+    }
+
+    #[derive(Debug, Default)]
+    struct TestFormatReadGate {
+        entered: tokio::sync::Notify,
+        release: CancellationToken,
    }

    impl AuthenticatedReadPeer {
@@ -4319,9 +4326,16 @@ mod tests {
                object_read_all_disks: Arc::default(),
                format_data,
                read_all_data,
+                next_format_read_gate: Arc::default(),
            }
        }

+        fn pause_next_format_read(&self) -> Arc<TestFormatReadGate> {
+            let gate = Arc::new(TestFormatReadGate::default());
+            *self.next_format_read_gate.lock().expect("format gate lock poisoned") = Some(Arc::clone(&gate));
+            gate
+        }
+
        fn disk_info_calls(&self) -> u32 {
            self.disk_info_calls.load(Ordering::Acquire)
        }
@@ -4429,6 +4443,11 @@ mod tests {
                                let disk = request.disk;
                                peer.read_all_calls.fetch_add(1, Ordering::AcqRel);
                                let data = if is_format_read {
+                                    let gate = peer.next_format_read_gate.lock().expect("format gate lock poisoned").take();
+                                    if let Some(gate) = gate {
+                                        gate.entered.notify_one();
+                                        gate.release.cancelled().await;
+                                    }
                                    peer.format_data.clone()
                                } else {
                                    peer.object_read_all_disks
@@ -7131,6 +7150,319 @@ mod tests {
        peer.stop().await;
    }

+    async fn missing_read_set(peer: &TestGrpcPeer, format: crate::layout::format::FormatV3) -> Arc<crate::set_disk::SetDisks> {
+        let count = format.erasure.sets[0].len();
+        let endpoints = (0..count)
+            .map(|index| Endpoint {
+                url: url::Url::parse(&format!("{}/data/rustfs{index}", peer.addr)).expect("endpoint should parse"),
+                is_local: false,
+                pool_idx: 0,
+                set_idx: 0,
+                disk_idx: i32::try_from(index).expect("fixture index fits"),
+            })
+            .collect();
+        crate::set_disk::SetDisks::new(
+            "missing-read-test".to_string(),
+            Arc::new(tokio::sync::RwLock::new(vec![None; count])),
+            count,
+            0,
+            0,
+            0,
+            endpoints,
+            format,
+            Vec::new(),
+        )
+        .await
+    }
+
+    #[tokio::test]
+    #[serial(remote_disk_recovery_probe)]
+    async fn missing_read_disks_reconnect_before_periodic_monitor_and_coalesce() {
+        runtime_sources::ensure_test_rpc_secret();
+        let mut format = crate::layout::format::FormatV3::new(1, 1);
+        let disk_id = format.erasure.sets[0][0];
+        format.erasure.this = disk_id;
+        let Some(peer) = TestGrpcPeer::spawn(
+            Bytes::from(format.to_json().expect("format should serialize")),
+            Bytes::from_static(b"startup-recovered-data"),
+        )
+        .await
+        else {
+            panic!("authenticated regression peer is required");
+        };
+        let set = missing_read_set(&peer, format).await;
+        let snapshots = futures::future::join_all((0..8).map(|_| set.get_disks_for_data_read())).await;
+        let disk = snapshots[0][0]
+            .as_ref()
+            .expect("a ready peer must be usable before the periodic reconnect");
+        for snapshot in &snapshots {
+            assert!(Arc::ptr_eq(
+                disk,
+                snapshot[0]
+                    .as_ref()
+                    .expect("every waiting read should observe the same handle")
+            ));
+        }
+        assert_eq!(peer.peer.read_all_calls(), 1, "concurrent requests must share the format probe");
+        assert_eq!(disk.get_disk_id().await.expect("disk identity should be available"), Some(disk_id));
+        let raw = disk
+            .read_all("bucket", "object")
+            .await
+            .expect("reconnected handle should read data");
+        assert_eq!(raw, Bytes::from_static(b"startup-recovered-data"));
+        assert_eq!(peer.peer.object_read_all_disks(), vec![disk_id.to_string()]);
+        let healthy = set.get_disks_for_data_read().await;
+        assert!(Arc::ptr_eq(disk, healthy[0].as_ref().expect("healthy handle must remain registered")));
+        assert_eq!(peer.peer.read_all_calls(), 2, "healthy reads must not re-probe the format");
+        disk.close().await.expect("reconnected disk should close");
+        peer.stop().await;
+    }
+
+    #[tokio::test]
+    #[serial(remote_disk_recovery_probe)]
+    async fn missing_read_disks_reject_foreign_format_and_back_off() {
+        runtime_sources::ensure_test_rpc_secret();
+        let reference = crate::layout::format::FormatV3::new(1, 1);
+        let mut foreign = reference.clone();
+        foreign.id = Uuid::new_v4();
+        foreign.erasure.this = reference.erasure.sets[0][0];
+        let Some(peer) = TestGrpcPeer::spawn(
+            Bytes::from(foreign.to_json().expect("foreign format should serialize")),
+            Bytes::from_static(b"foreign-data"),
+        )
+        .await
+        else {
+            panic!("authenticated regression peer is required");
+        };
+        let set = missing_read_set(&peer, reference).await;
+        assert!(
+            set.get_disks_for_data_read().await[0].is_none(),
+            "a matching disk ID cannot authorize a foreign deployment"
+        );
+        assert_eq!(peer.peer.read_all_calls(), 1, "the foreign format must actually be inspected");
+        assert!(set.get_disks_for_data_read().await[0].is_none());
+        assert_eq!(peer.peer.read_all_calls(), 1, "failed probes must not repeat on every degraded GET");
+        assert!(peer.peer.object_read_all_disks().is_empty(), "foreign object data must never be read");
+        peer.stop().await;
+    }
+
+    #[tokio::test]
+    #[serial(remote_disk_recovery_probe)]
+    async fn missing_read_disks_reject_wrong_slot_and_invalid_formats() {
+        runtime_sources::ensure_test_rpc_secret();
+        let mut reference = crate::layout::format::FormatV3::new(1, 2);
+        reference.erasure.this = reference.erasure.sets[0][0];
+        let mut nil = reference.clone();
+        nil.erasure.this = Uuid::nil();
+        let mut unknown = reference.clone();
+        unknown.erasure.this = Uuid::new_v4();
+        let mut future = reference.clone();
+        future.erasure.version = crate::layout::format::FormatErasureVersion::Unknown;
+        for (name, payload, first_accepted) in [
+            ("duplicate slot zero", reference.to_json().expect("format should serialize"), true),
+            ("nil drive", nil.to_json().expect("nil format should serialize"), false),
+            ("unknown drive", unknown.to_json().expect("unknown format should serialize"), false),
+            ("future version", future.to_json().expect("future format should serialize"), false),
+            ("malformed", "not-json".to_string(), false),
+        ] {
+            let peer = TestGrpcPeer::spawn(Bytes::from(payload), Bytes::new())
+                .await
+                .expect("authenticated regression peer is required");
+            let set = missing_read_set(&peer, reference.clone()).await;
+            let disks = set.get_disks_for_data_read().await;
+            assert_eq!(disks[0].is_some(), first_accepted, "{name}: slot zero identity decision");
+            assert!(disks[1].is_none(), "{name}: a peer cannot claim another topology slot");
+            assert_eq!(peer.peer.read_all_calls(), 2, "{name}: both missing slots must be probed");
+            assert!(peer.peer.object_read_all_disks().is_empty(), "{name}: no object payload may be read");
+            for disk in disks.into_iter().flatten() {
+                disk.close().await.expect("accepted disk should close");
+            }
+            peer.stop().await;
+        }
+    }
+
+    #[tokio::test]
+    #[serial(remote_disk_recovery_probe)]
+    async fn missing_read_disks_do_not_claim_local_or_misconfigured_endpoints() {
+        runtime_sources::ensure_test_rpc_secret();
+        let mut reference = crate::layout::format::FormatV3::new(1, 1);
+        reference.erasure.this = reference.erasure.sets[0][0];
+        let peer = TestGrpcPeer::spawn(Bytes::from(reference.to_json().expect("format should serialize")), Bytes::new())
+            .await
+            .expect("authenticated regression peer is required");
+        for (local, pool, set_index, disk_index) in [(true, 0, 0, 0), (false, -1, 0, 0), (false, 0, 1, 0), (false, 0, 0, 1)] {
+            let mut set = missing_read_set(&peer, reference.clone()).await;
+            let endpoint = &mut Arc::get_mut(&mut set)
+                .expect("fixture set must be uniquely owned")
+                .set_endpoints[0];
+            endpoint.is_local = local;
+            endpoint.pool_idx = pool;
+            endpoint.set_idx = set_index;
+            endpoint.disk_idx = disk_index;
+            assert!(set.get_disks_for_data_read().await[0].is_none());
+        }
+        assert_eq!(peer.peer.read_all_calls(), 0, "invalid endpoint ownership must fail before I/O");
+        peer.stop().await;
+    }
+
+    #[tokio::test]
+    #[serial(remote_disk_recovery_probe)]
+    async fn missing_read_disks_preserve_concurrently_published_handles() {
+        runtime_sources::ensure_test_rpc_secret();
+        let mut reference = crate::layout::format::FormatV3::new(1, 1);
+        reference.erasure.this = reference.erasure.sets[0][0];
+        for reader_first in [false, true] {
+            let peer = TestGrpcPeer::spawn(Bytes::from(reference.to_json().expect("format should serialize")), Bytes::new())
+                .await
+                .expect("authenticated regression peer is required");
+            let set = missing_read_set(&peer, reference.clone()).await;
+            let gate = peer.peer.pause_next_format_read();
+            let pending_set = Arc::clone(&set);
+            let pending = tokio::spawn(async move {
+                if reader_first {
+                    pending_set.get_disks_for_data_read().await;
+                } else {
+                    pending_set.renew_disk(&pending_set.set_endpoints[0]).await;
+                }
+            });
+            time::timeout(Duration::from_secs(2), gate.entered.notified())
+                .await
+                .expect("the first reconnect must reach the format barrier");
+            if reader_first {
+                set.renew_disk(&set.set_endpoints[0]).await;
+            } else {
+                set.get_disks_for_data_read().await;
+            }
+            let published = set.disks.read().await[0]
+                .clone()
+                .expect("the competing reconnect must publish");
+            gate.release.cancel();
+            pending.await.expect("paused reconnect must complete");
+            assert!(
+                Arc::ptr_eq(&published, set.disks.read().await[0].as_ref().expect("published slot must remain")),
+                "neither reconnect may replace a handle published while its format I/O was pending"
+            );
+            published.close().await.expect("published disk should close");
+            peer.stop().await;
+        }
+    }
+
+    #[tokio::test]
+    #[serial(remote_disk_recovery_probe, internode_metrics)]
+    async fn missing_read_disks_expired_waiter_does_not_start_another_probe() {
+        // Disable cooldown so this independently proves deadline admission,
+        // rather than passing only because the leader has just finished.
+        temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_RETURNING_PROBE_INTERVAL_SECS, Some("0"))], async {
+            runtime_sources::ensure_test_rpc_secret();
+            let mut reference = crate::layout::format::FormatV3::new(1, 1);
+            reference.erasure.this = reference.erasure.sets[0][0];
+            let peer = TestGrpcPeer::spawn(Bytes::from(reference.to_json().expect("format should serialize")), Bytes::new())
+                .await
+                .expect("authenticated regression peer is required");
+            let set = missing_read_set(&peer, reference).await;
+            let gate = peer.peer.pause_next_format_read();
+            let mut leader = Box::pin(set.get_disks_for_data_read());
+            time::timeout(Duration::from_secs(2), async {
+                tokio::select! {
+                    _ = gate.entered.notified() => {}
+                    _ = &mut leader => panic!("leader must remain blocked in its format RPC"),
+                }
+            })
+            .await
+            .expect("leader must reach the format barrier");
+            time::pause();
+            let mut waiter = Box::pin(set.get_disks_for_data_read());
+            assert!(futures::poll!(&mut waiter).is_pending(), "waiter must queue behind the leader");
+            let requests_before =
+                crate::cluster::rpc::runtime_sources::internode_metrics_snapshot_for_test().outgoing_requests_total;
+
+            time::advance(crate::disk::disk_store::get_drive_active_check_timeout()).await;
+            assert!(leader.await[0].is_none(), "expired leader must not publish a disk");
+            assert!(waiter.await[0].is_none(), "expired waiter must not publish a disk");
+            time::resume();
+
+            // Count client dispatches, not server arrivals: a cancelled RPC can still
+            // be buffered in the transport when the expired waiter returns.
+            assert_eq!(
+                crate::cluster::rpc::runtime_sources::internode_metrics_snapshot_for_test().outgoing_requests_total,
+                requests_before,
+                "an expired waiter must not dispatch another format RPC after the leader releases the lock"
+            );
+            gate.release.cancel();
+            peer.stop().await;
+        })
+        .await;
+    }
+
+    #[tokio::test]
+    #[serial(remote_disk_recovery_probe)]
+    async fn missing_read_disks_timeout_releases_singleflight_and_allows_retry() {
+        runtime_sources::ensure_test_rpc_secret();
+        let mut reference = crate::layout::format::FormatV3::new(1, 1);
+        reference.erasure.this = reference.erasure.sets[0][0];
+        let peer = TestGrpcPeer::spawn(Bytes::from(reference.to_json().expect("format should serialize")), Bytes::new())
+            .await
+            .expect("authenticated regression peer is required");
+        let set = missing_read_set(&peer, reference).await;
+        let gate = peer.peer.pause_next_format_read();
+        let deadline = crate::disk::disk_store::get_drive_active_check_timeout();
+        let snapshots = time::timeout(
+            deadline + Duration::from_secs(2),
+            futures::future::join_all((0..4).map(|_| set.get_disks_for_data_read())),
+        )
+        .await
+        .expect("all waiting requests must respect the bounded probe deadline");
+        assert!(snapshots.iter().all(|snapshot| snapshot[0].is_none()));
+        assert_eq!(peer.peer.read_all_calls(), 1, "only one format RPC may remain stalled");
+        gate.release.cancel();
+        time::pause();
+        time::advance(crate::disk::health_state::get_drive_returning_probe_interval()).await;
+        time::resume();
+        let disks = set.get_disks_for_data_read().await;
+        let disk = disks[0]
+            .as_ref()
+            .expect("a timed-out probe must not prevent a later read from reconnecting");
+        assert_eq!(peer.peer.read_all_calls(), 2, "retry must validate the format again");
+        disk.close().await.expect("reconnected disk should close");
+        peer.stop().await;
+    }
+
+    #[tokio::test]
+    #[serial(remote_disk_recovery_probe)]
+    async fn missing_read_disks_cancellation_leaves_no_published_handle() {
+        runtime_sources::ensure_test_rpc_secret();
+        let mut reference = crate::layout::format::FormatV3::new(1, 1);
+        reference.erasure.this = reference.erasure.sets[0][0];
+        let peer = TestGrpcPeer::spawn(Bytes::from(reference.to_json().expect("format should serialize")), Bytes::new())
+            .await
+            .expect("authenticated regression peer is required");
+        let set = missing_read_set(&peer, reference).await;
+        let gate = peer.peer.pause_next_format_read();
+        let pending_set = Arc::clone(&set);
+        let pending = tokio::spawn(async move { pending_set.get_disks_for_data_read().await });
+        time::timeout(Duration::from_secs(2), gate.entered.notified())
+            .await
+            .expect("probe must reach the format barrier");
+        // Cancellation can happen after the start-based cooldown has elapsed.
+        // It must still cool down before another caller probes the same peer.
+        time::pause();
+        time::advance(crate::disk::health_state::get_drive_returning_probe_interval()).await;
+        time::resume();
+        pending.abort();
+        assert!(pending.await.expect_err("probe must be cancelled").is_cancelled());
+        assert!(
+            set.disks.read().await[0].is_none(),
+            "cancelled format I/O must not publish an unvalidated handle"
+        );
+        let waiting = time::timeout(Duration::from_secs(1), set.get_disks_for_data_read())
+            .await
+            .expect("cancelled probe must release the singleflight lock");
+        assert!(waiting[0].is_none(), "retry cooldown must still apply after cancellation");
+        assert_eq!(peer.peer.read_all_calls(), 1);
+        gate.release.cancel();
+        peer.stop().await;
+    }
+
    #[tokio::test]
    async fn test_copy_stream_with_buffer_copies_full_payload() {
        let payload = b"walk-dir-stream".repeat(1024);
@@ -4253,6 +4253,13 @@ mod decommission_lock_order_tests {
                panic!("cancellation probe finished before observing target contention: {result:?}");
            }
        }
+        // The mutation absorbs the gate contention inline before the outer wait
+        // loop ever sees it, and that inline budget is bounded.
+        assert_eq!(
+            retry_observer.target_gate_inline_retries(),
+            crate::core::pools::DECOMMISSION_MUTATION_GATE_MAX_INLINE_ATTEMPTS - 1,
+            "inline target-gate retries must stop at their bounded attempt budget"
+        );
        tokio::time::sleep(Duration::from_millis(800)).await;
        assert_eq!(
            retry_observer.target_gate_exact_reloads(),
@@ -244,13 +244,16 @@ impl StdError for ConditionalFileNotCommittedError {
    }
 }

-fn classify_internode_missing_error(error: &InternodeHttpError) -> Option<DiskError> {
+fn classify_internode_disk_error(error: &InternodeHttpError) -> Option<DiskError> {
    if error.is_remote_file_not_found() {
        return Some(DiskError::FileNotFound);
    }
    if error.is_remote_volume_not_found() {
        return Some(DiskError::VolumeNotFound);
    }
+    if error.is_remote_file_corrupt() {
+        return Some(DiskError::FileCorrupt);
+    }
    None
 }

@@ -526,13 +529,10 @@ fn io_error_chain_contains_kind(io_error: &std::io::Error, kind: std::io::ErrorK

 impl From<std::io::Error> for DiskError {
    fn from(e: std::io::Error) -> Self {
-        if let Some(error) = e.get_ref().and_then(|source| source.downcast_ref::<InternodeHttpError>()) {
-            if error.is_remote_file_not_found() {
-                return DiskError::FileNotFound;
-            }
-            if error.is_remote_volume_not_found() {
-                return DiskError::VolumeNotFound;
-            }
+        if let Some(error) = e.get_ref().and_then(|source| source.downcast_ref::<InternodeHttpError>())
+            && let Some(classified) = classify_internode_disk_error(error)
+        {
+            return classified;
        }
        let e = match e.downcast::<TerminalReadError>() {
            Ok(terminal_error) => {
@@ -541,7 +541,7 @@ impl From<std::io::Error> for DiskError {
                    && let Some(internode_error) = io_error
                        .get_ref()
                        .and_then(|source| source.downcast_ref::<InternodeHttpError>())
-                    && let Some(classified) = classify_internode_missing_error(internode_error)
+                    && let Some(classified) = classify_internode_disk_error(internode_error)
                {
                    return classified;
                }
@@ -963,6 +963,7 @@ mod tests {
        for (remote_error, expected) in [
            (rustfs_rio::new_test_remote_file_not_found_http_io_error(), DiskError::FileNotFound),
            (rustfs_rio::new_test_remote_volume_not_found_http_io_error(), DiskError::VolumeNotFound),
+            (rustfs_rio::new_test_remote_file_corrupt_http_io_error(), DiskError::FileCorrupt),
        ] {
            let wrapped = terminal_read_error_to_io(DiskError::Io(remote_error));
            assert_eq!(DiskError::from(wrapped), expected);
@@ -1533,25 +1534,23 @@ mod tests {
    }

    #[test]
-    fn test_internode_missing_errors_preserve_disk_error_types() {
+    fn test_internode_disk_errors_preserve_disk_error_types() {
        let file_missing = DiskError::from(rustfs_rio::new_test_remote_file_not_found_http_io_error());
        let volume_missing = DiskError::from(rustfs_rio::new_test_remote_volume_not_found_http_io_error());
+        let file_corrupt = DiskError::from(rustfs_rio::new_test_remote_file_corrupt_http_io_error());
        let unmarked_server_error = DiskError::from(rustfs_rio::new_test_internode_http_io_error(
            rustfs_rio::InternodeHttpErrorKind::HttpStatus(http::StatusCode::INTERNAL_SERVER_ERROR),
        ));

        assert_eq!(file_missing, DiskError::FileNotFound);
        assert_eq!(volume_missing, DiskError::VolumeNotFound);
+        assert_eq!(file_corrupt, DiskError::FileCorrupt);
        assert!(matches!(unmarked_server_error, DiskError::Io(_)));
-        for missing in [file_missing, volume_missing] {
-            assert_eq!(missing.clone(), missing);
+        for error in [file_missing, volume_missing, file_corrupt] {
+            assert_eq!(error.clone(), error);
            assert_eq!(
-                crate::disk::error_reduce::reduce_write_quorum_errs(
-                    &[Some(missing.clone()), Some(missing.clone()), None],
-                    &[],
-                    2
-                ),
-                Some(missing)
+                crate::disk::error_reduce::reduce_write_quorum_errs(&[Some(error.clone()), Some(error.clone()), None], &[], 2),
+                Some(error)
            );
        }
    }
@@ -825,7 +825,7 @@ impl Disk {
        dst_volume: &str,
        dst_path: &str,
    ) -> Result<RenameDataResp> {
-        let Some(scope) = crate::store::bucket_heal_scope(dst_volume) else {
+        let Some(scope) = crate::store::bucket_heal_scope_for_object(dst_volume, dst_path) else {
            return self
                .rename_data_borrowed_with_fence(src_volume, src_path, fi, dst_volume, dst_path, None)
                .await;
@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+use crate::disk::error::Error as DiskError;
 use crate::erasure::codec::buffer_pool::{get_ec_buffer, return_ec_buffer};
 use pin_project_lite::pin_project;
 use rustfs_utils::HashAlgorithm;
@@ -239,7 +240,7 @@ fn short_shard_read(got: usize, want: usize) -> std::io::Error {
        want,
        "short shard read: got {got} of {want} bytes"
    );
-    std::io::Error::new(std::io::ErrorKind::UnexpectedEof, format!("short shard read: got {got} of {want} bytes"))
+    std::io::Error::new(std::io::ErrorKind::UnexpectedEof, DiskError::FileCorrupt)
 }

 /// Split a `[hash][data]` block, verify the hash (unless `skip_verify`), and
@@ -265,7 +266,7 @@ fn split_and_verify<'a>(hash_algo: &HashAlgorithm, skip_verify: bool, block: &'a
            data_len = data.len(),
            "bitrot hash mismatch"
        );
-        return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "bitrot hash mismatch"));
+        return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, DiskError::FileCorrupt));
    }
    Ok((data, verify))
 }
@@ -434,7 +435,7 @@ where
                            data_len = want,
                            "bitrot hash mismatch"
                        );
-                        return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "bitrot hash mismatch"));
+                        return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, DiskError::FileCorrupt));
                    }
                    hash_offset += take;
                    remaining -= take;
@@ -1141,6 +1142,26 @@ mod tests {
    use std::time::Duration;
    use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt, ReadBuf};

+    #[test]
+    fn bitrot_damage_preserves_typed_repair_evidence() {
+        use crate::disk::error::Error as DiskError;
+        let algo = HashAlgorithm::HighwayHash256;
+        let data = b"repair evidence";
+        let mut frame = algo.hash_encode(data).as_ref().to_vec();
+        frame.extend_from_slice(data);
+        *frame.last_mut().expect("test frame has payload") ^= 1;
+        let error = super::split_and_verify(&algo, false, &frame).expect_err("a hash mismatch must reject the shard");
+        assert_eq!(error.kind(), io::ErrorKind::InvalidData);
+        assert_eq!(DiskError::from(error), DiskError::FileCorrupt);
+        let error = super::short_shard_read(3, 4);
+        assert_eq!(error.kind(), io::ErrorKind::UnexpectedEof);
+        assert_eq!(DiskError::from(error), DiskError::FileCorrupt);
+        assert!(matches!(
+            DiskError::from(io::Error::new(io::ErrorKind::InvalidData, "unrelated invalid data")),
+            DiskError::Io(_)
+        ));
+    }
+
    struct FragmentedSource {
        chunks: VecDeque<Bytes>,
    }
@@ -26,7 +26,7 @@ use crate::erasure::codec::workspace::ShardBufferPool;
 use crate::erasure::coding::{BitrotReader, Erasure};
 use crate::io_support::bitrot::DeferredReaderStripeHandle;
 use crate::set_disk::shard_source::{
-    INLINE_SHARD_SLOTS, ShardBuffers, ShardErrors, ShardReadCost, ShardStripeSource, StripeReadState,
+    INLINE_SHARD_SLOTS, ShardBuffers, ShardErrors, ShardReadCost, ShardReadRepair, ShardStripeSource, StripeReadState,
 };
 use futures::FutureExt;
 use futures::stream::{FuturesUnordered, StreamExt};
@@ -46,7 +46,13 @@ type ShardReadFuture<'a> = Pin<Box<dyn Future<Output = (usize, ShardReadCost, Re
 type OwnedShardReadFuture<'a, R> =
    Pin<Box<dyn Future<Output = (usize, ShardReadCost, Result<Vec<u8>, Error>, Option<BitrotReader<R>>, bool)> + Send + 'a>>;
 pub(crate) type DeferredReaderReopener<R> = Arc<dyn Fn(usize) -> Option<BitrotReader<R>> + Send + Sync>;
-pub(crate) type DecodeOutcome = (usize, Option<std::io::Error>, bool);
+#[derive(Debug, Default)]
+pub(crate) struct DecodeOutcome {
+    pub(crate) written: usize,
+    pub(crate) error: Option<std::io::Error>,
+    pub(crate) exact_quorum: bool,
+    pub(crate) repair: ShardReadRepair,
+}

 type ShardIndexes = SmallVec<[usize; INLINE_SHARD_SLOTS]>;
 type ActiveReaders = SmallVec<[bool; INLINE_SHARD_SLOTS]>;
@@ -596,6 +602,7 @@ pub(crate) struct ParallelReader<R> {
    hedged_readers: ActiveReaders,
    stripe_index: usize,
    integrity: Option<std::sync::Arc<crate::io_support::shard_integrity::PartProofReader>>,
+    repair_evidence: Option<Arc<ShardReadRepair>>,
 }
 }

@@ -808,9 +815,15 @@ where
            deferred_reopeners: Vec::new(),
            hedged_readers,
            stripe_index: 0,
+            repair_evidence: None,
        }
    }

+    pub(crate) fn with_repair_evidence(mut self, evidence: Arc<ShardReadRepair>) -> Self {
+        self.repair_evidence = Some(evidence);
+        self
+    }
+
    /// Attach the per-slot deferred stripe handles produced during bitrot
    /// reader setup. Only parity slots are consulted: a handle lets the
    /// lockstep path open a parity shard aligned to the stripe where a data
@@ -2027,6 +2040,9 @@ where
            .take()
            .unwrap_or_else(|| Box::new(StripeReadState::with_slot_count(self.readers.len(), self.data_shards)));
        self.read_into_state(&mut state).await;
+        if let Some(evidence) = &self.repair_evidence {
+            evidence.observe(state.parts_mut().1);
+        }
        if state.can_decode()
            && let Some(proof) = &self.integrity
        {
@@ -2231,10 +2247,10 @@ impl Erasure {
        W: AsyncWrite + Send + Sync + Unpin,
        R: crate::erasure::coding::ShardSource,
    {
-        let (written, error, _) = self
+        let outcome = self
            .decode_inner(writer, readers, offset, length, total_length, None, Vec::new(), Vec::new())
            .await;
-        (written, error)
+        (outcome.written, outcome.error)
    }

    #[allow(dead_code, reason = "read-cost decode path asserted by this file's tests (backlog#1823)")]
@@ -2251,10 +2267,10 @@ impl Erasure {
        W: AsyncWrite + Send + Sync + Unpin,
        R: crate::erasure::coding::ShardSource,
    {
-        let (written, error, _) = self
+        let outcome = self
            .decode_inner(writer, readers, offset, length, total_length, Some(read_costs), Vec::new(), Vec::new())
            .await;
-        (written, error)
+        (outcome.written, outcome.error)
    }

    /// GET decode entry point that also carries the deferred-parity stripe
@@ -2311,7 +2327,7 @@ impl Erasure {
        W: AsyncWrite + Send + Sync + Unpin,
        R: crate::erasure::coding::ShardSource,
    {
-        let (written, error, _) = self
+        let outcome = self
            .decode_inner(
                writer,
                readers,
@@ -2323,7 +2339,7 @@ impl Erasure {
                deferred_reopeners,
            )
            .await;
-        (written, error)
+        (outcome.written, outcome.error)
    }

    #[allow(clippy::too_many_arguments)]
@@ -2374,6 +2390,7 @@ impl Erasure {
        block_length: usize,
        written: &mut usize,
        ret_err: &mut Option<std::io::Error>,
+        repair: &ShardReadRepair,
        stage_metrics_enabled: bool,
        require_surplus_source: bool,
        integrity: Option<&std::sync::Arc<crate::io_support::shard_integrity::PartProofReader>>,
@@ -2382,6 +2399,7 @@ impl Erasure {
    where
        W: AsyncWrite + Send + Sync + Unpin,
    {
+        repair.observe(errs);
        if ret_err.is_none()
            && let (_, Some(err)) = reduce_errs(errs, &[])
            && (err == Error::FileNotFound || err == Error::FileCorrupt)
@@ -2516,41 +2534,42 @@ impl Erasure {
    {
        if readers.len() != self.data_shards + self.parity_shards {
            record_get_object_pipeline_failure(GET_STAGE_RANGE, GetObjectFailureReason::RangeOrLengthInvalid);
-            return (0, Some(io::Error::new(ErrorKind::InvalidInput, "Invalid number of readers")), false);
+            return DecodeOutcome {
+                error: Some(io::Error::new(ErrorKind::InvalidInput, "Invalid number of readers")),
+                ..Default::default()
+            };
        }

        // block_size/data_shards come from on-disk metadata; a corrupt FileInfo with a
        // zero here must surface as an error, not a divide-by-zero panic on every GET.
        if self.block_size == 0 || self.data_shards == 0 {
            record_get_object_pipeline_failure(GET_STAGE_RANGE, GetObjectFailureReason::RangeOrLengthInvalid);
-            return (
-                0,
-                Some(io::Error::new(ErrorKind::InvalidInput, "Invalid erasure coding parameters")),
-                false,
-            );
+            return DecodeOutcome {
+                error: Some(io::Error::new(ErrorKind::InvalidInput, "Invalid erasure coding parameters")),
+                ..Default::default()
+            };
        }

        let Some(end_offset) = offset.checked_add(length) else {
            record_get_object_pipeline_failure(GET_STAGE_RANGE, GetObjectFailureReason::RangeOrLengthInvalid);
-            return (
-                0,
-                Some(io::Error::new(ErrorKind::InvalidInput, "offset + length exceeds total length")),
-                false,
-            );
+            return DecodeOutcome {
+                error: Some(io::Error::new(ErrorKind::InvalidInput, "offset + length exceeds total length")),
+                ..Default::default()
+            };
        };
        if end_offset > total_length {
            record_get_object_pipeline_failure(GET_STAGE_RANGE, GetObjectFailureReason::RangeOrLengthInvalid);
-            return (
-                0,
-                Some(io::Error::new(ErrorKind::InvalidInput, "offset + length exceeds total length")),
-                false,
-            );
+            return DecodeOutcome {
+                error: Some(io::Error::new(ErrorKind::InvalidInput, "offset + length exceeds total length")),
+                ..Default::default()
+            };
        }

        let mut ret_err = None;
+        let repair = ShardReadRepair::default();

        if length == 0 {
-            return (0, ret_err, false);
+            return DecodeOutcome::default();
        }

        let mut written = 0;
@@ -2573,6 +2592,11 @@ impl Erasure {
        .with_deferred_parity_reopeners(deferred_reopeners);

        let integrity = reader.integrity.clone();
+        // Copy sources use demand-bound scheduling for backpressure, but retain
+        // the ordinary GET read quorum. Only the opt-in GET rollout requires a
+        // surplus source; available surplus shards and integrity proofs are
+        // still verified before any reconstructed copy bytes are emitted.
+        let require_surplus_source = reader.demand_bound_lockstep && matches!(decode_read_policy(), DecodeReadPolicy::Default);

        let start = offset / self.block_size;
        let end = end_offset.saturating_sub(1) / self.block_size;
@@ -2667,7 +2691,6 @@ impl Erasure {
                        // `shards` are borrowed again below. In the `Stop` case that
                        // drop is what cancels the still-in-flight prefetch read.
                        let (flow, next): (Option<StripeFlow>, Option<StripeReadOutput>) = {
-                            let require_surplus_source = reader.demand_bound_lockstep;
                            let read_fut = read_stripe_timed(&mut reader, stage_metrics_enabled);
                            let emit_fut = self.emit_decoded_stripe(
                                writer,
@@ -2677,6 +2700,7 @@ impl Erasure {
                                block_length,
                                &mut written,
                                &mut ret_err,
+                                &repair,
                                stage_metrics_enabled,
                                require_surplus_source,
                                integrity.as_ref(),
@@ -2727,8 +2751,9 @@ impl Erasure {
                                block_length,
                                &mut written,
                                &mut ret_err,
+                                &repair,
                                stage_metrics_enabled,
-                                reader.demand_bound_lockstep,
+                                require_surplus_source,
                                integrity.as_ref(),
                                offset,
                            )
@@ -2770,8 +2795,9 @@ impl Erasure {
                        block_length,
                        &mut written,
                        &mut ret_err,
+                        &repair,
                        stage_metrics_enabled,
-                        reader.demand_bound_lockstep,
+                        require_surplus_source,
                        integrity.as_ref(),
                        offset,
                    )
@@ -2785,15 +2811,16 @@ impl Erasure {
            }
        }

-        if ret_err.is_some() {
-            return (written, ret_err, exact_quorum);
-        }
-
-        if written < length {
+        if ret_err.is_none() && written < length {
            ret_err = Some(Error::LessData.into());
        }

-        (written, ret_err, exact_quorum)
+        DecodeOutcome {
+            written,
+            error: ret_err,
+            exact_quorum,
+            repair,
+        }
    }
 }

@@ -3449,6 +3476,150 @@ mod tests {
        }
    }

+    async fn copy_source_bitrot_shards(erasure: &Erasure, payload: &[u8], corrupt_parity: bool) -> Vec<Vec<u8>> {
+        let mut writers: Vec<_> = (0..erasure.data_shards + erasure.parity_shards)
+            .map(|_| BitrotWriter::new(Cursor::new(Vec::new()), erasure.shard_size(), HashAlgorithm::HighwayHash256))
+            .collect();
+        for block in payload.chunks(erasure.block_size) {
+            let shards = erasure.encode_data(block).expect("copy source should encode");
+            for (index, shard) in shards.iter().enumerate() {
+                let mut bytes = shard.to_vec();
+                if corrupt_parity && index == erasure.data_shards {
+                    // A stale source can have a valid per-shard bitrot checksum.
+                    // Reframe the changed parity to exercise reconstruction verification.
+                    bytes[0] ^= 0x80;
+                }
+                writers[index].write(&bytes).await.expect("copy source should frame");
+            }
+        }
+        writers.into_iter().map(|writer| writer.into_inner().into_inner()).collect()
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn copy_source_reads_exact_quorum_across_stripes_and_ranges() {
+        const BLOCK_SIZE: usize = 768;
+        let payload: Vec<_> = (0..BLOCK_SIZE * 3 + 37)
+            .map(|index| (index.wrapping_mul(31) % 251) as u8)
+            .collect();
+
+        for uses_legacy in [false, true] {
+            let erasure = Erasure::new_with_options(12, 4, BLOCK_SIZE, uses_legacy);
+            let buffers = copy_source_bitrot_shards(&erasure, &payload, false).await;
+            for offline in [[0, 1, 2, 3], [0, 5, 12, 15]] {
+                for (offset, length) in [(0, payload.len()), (BLOCK_SIZE + 17, payload.len() - BLOCK_SIZE - 17)] {
+                    let frame_offset = offset / BLOCK_SIZE * (erasure.shard_size() + HashAlgorithm::HighwayHash256.size());
+                    let positioned: Vec<_> = buffers.iter().map(|bytes| bytes[frame_offset..].to_vec()).collect();
+                    let (mut readers, handles) =
+                        readers_with_deferred_parity(&positioned, 12, erasure.shard_size(), &HashAlgorithm::HighwayHash256, &[]);
+                    for index in offline {
+                        readers[index] = None;
+                    }
+                    let mut output = Vec::new();
+                    let (written, error) = crate::set_disk::with_get_object_read_policy(
+                        crate::set_disk::GetObjectReadPolicy::CopySource,
+                        erasure.decode_with_stripe_handles(&mut output, readers, offset, length, payload.len(), None, handles),
+                    )
+                    .await;
+
+                    assert!(error.is_none(), "legacy={uses_legacy}, offline={offline:?}, offset={offset}: {error:?}");
+                    assert_eq!(written, length);
+                    assert_eq!(output, payload[offset..offset + length]);
+                }
+            }
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn copy_source_recovers_exact_quorum_after_first_stripe() {
+        const BLOCK_SIZE: usize = 768;
+        let payload: Vec<_> = (0..BLOCK_SIZE * 3 + 37)
+            .map(|index| (index.wrapping_mul(31) % 251) as u8)
+            .collect();
+        for uses_legacy in [false, true] {
+            let erasure = Erasure::new_with_options(12, 4, BLOCK_SIZE, uses_legacy);
+            let buffers = copy_source_bitrot_shards(&erasure, &payload, false).await;
+            let first_frame = erasure.shard_size() + HashAlgorithm::HighwayHash256.size();
+            let (readers, handles) = readers_with_deferred_parity(
+                &buffers,
+                12,
+                erasure.shard_size(),
+                &HashAlgorithm::HighwayHash256,
+                &[(0, first_frame), (1, first_frame), (2, first_frame), (3, first_frame)],
+            );
+            let mut output = Vec::new();
+            let (written, error) = crate::set_disk::with_get_object_read_policy(
+                crate::set_disk::GetObjectReadPolicy::CopySource,
+                erasure.decode_with_stripe_handles(&mut output, readers, 0, payload.len(), payload.len(), None, handles),
+            )
+            .await;
+
+            assert!(error.is_none(), "legacy={uses_legacy}: late shard loss should recover: {error:?}");
+            assert_eq!(written, payload.len());
+            assert_eq!(output, payload);
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn copy_source_below_read_quorum_fails_before_output() {
+        const BLOCK_SIZE: usize = 768;
+        let payload = vec![0x5a; BLOCK_SIZE * 2 + 37];
+        for uses_legacy in [false, true] {
+            let erasure = Erasure::new_with_options(12, 4, BLOCK_SIZE, uses_legacy);
+            let buffers = copy_source_bitrot_shards(&erasure, &payload, false).await;
+            let (mut readers, handles) =
+                readers_with_deferred_parity(&buffers, 12, erasure.shard_size(), &HashAlgorithm::HighwayHash256, &[]);
+            for reader in readers.iter_mut().take(5) {
+                *reader = None;
+            }
+            let mut output = Vec::new();
+            let (written, error) = crate::set_disk::with_get_object_read_policy(
+                crate::set_disk::GetObjectReadPolicy::CopySource,
+                erasure.decode_with_stripe_handles(&mut output, readers, 0, payload.len(), payload.len(), None, handles),
+            )
+            .await;
+
+            assert_eq!(written, 0);
+            assert!(output.is_empty());
+            assert_eq!(
+                error
+                    .expect("eleven shards cannot reconstruct twelve data shards")
+                    .to_string(),
+                Error::ErasureReadQuorum.to_string()
+            );
+        }
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn copy_source_rejects_corrupt_surplus_parity_before_output() {
+        const BLOCK_SIZE: usize = 768;
+        let payload = vec![0x5a; BLOCK_SIZE * 2 + 37];
+        for uses_legacy in [false, true] {
+            let erasure = Erasure::new_with_options(12, 4, BLOCK_SIZE, uses_legacy);
+            let buffers = copy_source_bitrot_shards(&erasure, &payload, true).await;
+            let (mut readers, handles) =
+                readers_with_deferred_parity(&buffers, 12, erasure.shard_size(), &HashAlgorithm::HighwayHash256, &[]);
+            for reader in readers.iter_mut().take(3) {
+                *reader = None;
+            }
+            let mut output = Vec::new();
+            let (written, error) = crate::set_disk::with_get_object_read_policy(
+                crate::set_disk::GetObjectReadPolicy::CopySource,
+                erasure.decode_with_stripe_handles(&mut output, readers, 0, payload.len(), payload.len(), None, handles),
+            )
+            .await;
+
+            assert_eq!(written, 0);
+            assert!(output.is_empty());
+            let error = error.expect("copy must reject inconsistent surplus parity despite valid bitrot framing");
+            assert_eq!(error.kind(), ErrorKind::InvalidData);
+            assert!(error.to_string().contains("inconsistent read source shards"));
+        }
+    }
+
    /// Guards the shard-buffer reuse in `ParallelReader`: a multi-stripe
    /// `Erasure::decode` that reconstructs missing data shards on every stripe
    /// must still return byte-exact output. Reconstructed and parity buffers are
@@ -3834,10 +4005,45 @@ mod tests {
            temp_env::async_with_vars(vars, async {
                let readers = build_prefetch_readers(&shard_bufs, shard_size, &hash_algo, BLOCK_SIZE, 0, &[], &[1]);
                let mut output = Vec::new();
-                let (written, err) = erasure.decode(&mut output, readers, 0, total_len, total_len).await;
-                assert!(err.is_none(), "{label}: corrupt-shard read errored {err:?}");
-                assert_eq!(written, total_len, "{label}: corrupt-shard short write");
+                let outcome = erasure
+                    .decode_with_stripe_handles_and_reopeners_with_diagnostics(
+                        &mut output,
+                        readers,
+                        0,
+                        total_len,
+                        total_len,
+                        None,
+                        Vec::new(),
+                        Vec::new(),
+                    )
+                    .await;
+                assert!(outcome.error.is_none(), "{label}: corrupt-shard read errored {:?}", outcome.error);
+                assert_eq!(outcome.written, total_len, "{label}: corrupt-shard short write");
                assert_eq!(output, total_data, "{label}: corrupt bytes leaked into output");
+                assert!(
+                    outcome.repair.needed(false),
+                    "{label}: minority corruption must survive successful reconstruction"
+                );
+                let readers = build_prefetch_readers(&shard_bufs, shard_size, &hash_algo, BLOCK_SIZE, 0, &[], &[]);
+                let mut healthy_output = Vec::new();
+                let healthy = erasure
+                    .decode_with_stripe_handles_and_reopeners_with_diagnostics(
+                        &mut healthy_output,
+                        readers,
+                        0,
+                        total_len,
+                        total_len,
+                        None,
+                        Vec::new(),
+                        Vec::new(),
+                    )
+                    .await;
+                assert!(healthy.error.is_none());
+                assert_eq!(healthy_output, total_data);
+                assert!(
+                    !healthy.repair.needed(true),
+                    "{label}: healthy/deferred shards cannot create repair evidence"
+                );
            })
            .await;
        }
@@ -4635,40 +4841,38 @@ mod tests {
                (ENV_RUSTFS_GET_SHARD_LOCALITY_PREFERENCE_ENABLE, None::<&str>),
            ],
            async {
-            const NUM_SHARDS: usize = 1;
-            const BLOCK_SIZE: usize = 64;
-            const DATA_SHARDS: usize = 4;
-            const PARITY_SHARDS: usize = 2;
-            const SHARD_SIZE: usize = BLOCK_SIZE / DATA_SHARDS;
+                const NUM_SHARDS: usize = 1;
+                const BLOCK_SIZE: usize = 64;
+                const DATA_SHARDS: usize = 4;
+                const PARITY_SHARDS: usize = 2;
+                const SHARD_SIZE: usize = BLOCK_SIZE / DATA_SHARDS;

-            let hash_algo = HashAlgorithm::HighwayHash256;
-            let readers = make_test_readers(DATA_SHARDS + PARITY_SHARDS, SHARD_SIZE, NUM_SHARDS, &hash_algo, &[], &[1]).await;
-            let read_costs = vec![
-                ShardReadCost::Local,
-                ShardReadCost::Local,
-                ShardReadCost::Local,
-                ShardReadCost::Local,
-                ShardReadCost::Remote,
-                ShardReadCost::Remote,
-            ];
-            let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
-            let mut parallel_reader = ParallelReader::new_with_metrics_path_and_read_costs(
-                readers,
-                erasure,
-                0,
-                NUM_SHARDS * BLOCK_SIZE,
-                None,
-                read_costs,
-            );
+                let hash_algo = HashAlgorithm::HighwayHash256;
+                let readers = make_test_readers(DATA_SHARDS + PARITY_SHARDS, SHARD_SIZE, NUM_SHARDS, &hash_algo, &[], &[1]).await;
+                let read_costs = vec![
+                    ShardReadCost::Local,
+                    ShardReadCost::Local,
+                    ShardReadCost::Local,
+                    ShardReadCost::Local,
+                    ShardReadCost::Remote,
+                    ShardReadCost::Remote,
+                ];
+                let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+                let mut parallel_reader = ParallelReader::new_with_metrics_path_and_read_costs(
+                    readers,
+                    erasure,
+                    0,
+                    NUM_SHARDS * BLOCK_SIZE,
+                    None,
+                    read_costs,
+                );

-            let (bufs, errs) = parallel_reader.read().await;
+                let (bufs, errs) = parallel_reader.read().await;

-            assert_eq!(DATA_SHARDS, bufs.iter().filter(|buf| buf.is_some()).count());
-            assert!(
-                matches!(&errs[1], Some(DiskError::Io(err)) if err.kind() == ErrorKind::InvalidData && err.to_string().contains("bitrot"))
-            );
-            assert_eq!(bufs[4].as_deref(), Some(&[4u8; SHARD_SIZE][..]));
-            assert!(bufs[5].is_none());
+                assert_eq!(DATA_SHARDS, bufs.iter().filter(|buf| buf.is_some()).count());
+                assert_eq!(errs[1], Some(DiskError::FileCorrupt));
+                assert_eq!(bufs[4].as_deref(), Some(&[4u8; SHARD_SIZE][..]));
+                assert!(bufs[5].is_none());
            },
        )
        .await;
@@ -4814,16 +5018,7 @@ mod tests {
            assert_eq!(DATA_SHARDS, bufs.iter().filter(|buf| buf.is_some()).count());
            assert_eq!(
                BITROT_DISKS,
-                errs.iter()
-                    .filter(|err| {
-                        match err {
-                            Some(DiskError::Io(err)) => {
-                                err.kind() == std::io::ErrorKind::InvalidData && err.to_string().contains("bitrot")
-                            }
-                            _ => false,
-                        }
-                    })
-                    .count()
+                errs.iter().filter(|err| matches!(err, Some(DiskError::FileCorrupt))).count()
            );
        }
    }
@@ -587,10 +587,8 @@ pub(crate) async fn verify_deep_parts(
            return Err(DiskError::FileCorrupt);
        }
        let part_status = statuses.get_mut(&part_index).ok_or(DiskError::FileCorrupt)?;
-        // Deep verification reads the complete encoded shard. `shard_file_offset`
-        // is a range-read helper and can stop before the final encoded byte for
-        // a partial data block, which would let a one-byte tail truncation pass
-        // as healthy. Use the physical shard length for the integrity proof.
+        // Deep verification must consume every encoded stripe, including a
+        // partial final stripe, before certifying the shard's integrity.
        let length = usize::try_from(erasure.shard_file_size(part.size as i64)).map_err(|_| DiskError::FileCorrupt)?;
        let mut readers = Vec::with_capacity(disks.len());
        for (index, disk) in disks.iter().enumerate() {
@@ -635,6 +633,9 @@ pub(crate) async fn verify_deep_parts(
                    part_status[index] = CHECK_PART_FILE_CORRUPT;
                }
                Err(error) => {
+                    if !matches!(error, DiskError::FileNotFound | DiskError::FileVersionNotFound | DiskError::FileCorrupt) {
+                        return Err(error);
+                    }
                    readers.push(None);
                    part_status[index] = conv_part_err_to_int(&Some(error));
                }
@@ -935,6 +935,8 @@ pub struct ObjectOptions {
    /// Persisted bucket incarnation observed before authorization.
    pub expected_bucket_incarnation_id: Option<Uuid>,
    pub no_lock: bool,
+    /// Internal read-only inspection must not enqueue metadata or payload repairs.
+    pub suppress_read_repair: bool,
    /// Control-plane writers that immediately read or CAS the same namespace
    /// key use TailDrained without changing namespace lock ownership.
    #[doc(hidden)]
@@ -0,0 +1,290 @@
+// Copyright 2024 RustFS Team
+// Licensed under the Apache License, Version 2.0.
+
+//! Explicit, bounded integrity operations. This service does not participate in ordinary GET or Heal.
+//! Lock order: cluster worker lock, bucket incarnation fence, object commit lock.
+//! Control requests use checkpoint CAS without taking the worker lock; a worker that loses CAS stops.
+
+mod model;
+mod runner;
+#[cfg(test)]
+mod tests;
+
+pub use model::{
+    InventoryItem, InventoryPage, ItemRequest, ItemResult, ItemState, Job, JobMode, JobRequest, JobState, Protection, Readiness,
+    readiness,
+};
+
+use crate::config::com::{read_config_limited_preserve_empty_with_metadata, save_config_with_opts_quiet};
+use crate::disk::RUSTFS_META_BUCKET;
+use crate::error::Error as StorageError;
+use crate::object_api::{ObjectOptions, WriteCompletion};
+use crate::storage_api_contracts::{
+    list::ListOperations,
+    namespace::NamespaceLocking,
+    object::{HTTPPreconditions, ObjectOperations},
+};
+use crate::store::ECStore;
+use model::InventoryItem as Item;
+use std::sync::Arc;
+use std::time::Duration;
+use uuid::Uuid;
+
+type Result<T> = std::result::Result<T, IntegrityError>;
+
+#[derive(Debug, thiserror::Error)]
+pub enum IntegrityError {
+    #[error("invalid integrity request: {0}")]
+    Invalid(&'static str),
+    #[error("integrity job or target changed concurrently")]
+    Conflict,
+    #[error("an integrity worker is already active")]
+    Busy,
+    #[error("integrity job not found")]
+    NotFound,
+    #[error("integrity operation was stopped")]
+    Stopped,
+    #[error("protected writes are not enabled on this coordinator")]
+    NotActivated,
+    #[error("unsupported migration bucket configuration")]
+    UnsupportedBucket,
+    #[error("integrity storage operation failed: {0}")]
+    Storage(#[from] StorageError),
+    #[error("integrity stream operation failed: {0}")]
+    Io(#[from] std::io::Error),
+    #[error("integrity checkpoint is malformed: {0}")]
+    Json(#[from] serde_json::Error),
+}
+
+fn check_bucket(bucket: &str) -> Result<()> {
+    crate::bucket::utils::check_valid_bucket_name_strict(bucket).map_err(|_| IntegrityError::Invalid("invalid source bucket"))
+}
+
+fn path(bucket: &str, id: Uuid) -> String {
+    format!("buckets/{bucket}/integrity-jobs/{id}.json")
+}
+
+pub async fn inventory(
+    api: Arc<ECStore>,
+    bucket: &str,
+    prefix: &str,
+    key_marker: Option<String>,
+    version_marker: Option<String>,
+    limit: i32,
+) -> Result<InventoryPage> {
+    check_bucket(bucket)?;
+    if !(1..=100).contains(&limit) || prefix.len() > 1024 || (version_marker.is_some() && key_marker.is_none()) {
+        return Err(IntegrityError::Invalid("invalid inventory page"));
+    }
+    let incarnation = api.bucket_incarnation_id_from_disk(bucket).await?;
+    let page = api
+        .clone()
+        .list_object_versions(bucket, prefix, key_marker.clone(), version_marker.clone(), None, limit)
+        .await?;
+    if page.is_truncated && (page.next_marker.clone(), page.next_version_idmarker.clone()) == (key_marker, version_marker) {
+        return Err(IntegrityError::Invalid("inventory cursor did not advance"));
+    }
+    let mut items = Vec::with_capacity(page.objects.len());
+    for listed in page.objects {
+        if listed.delete_marker {
+            items.push(Item::from_info(&listed));
+            continue;
+        }
+        let opts = ObjectOptions {
+            version_id: Some(listed.version_id.unwrap_or_else(Uuid::nil).to_string()),
+            include_part_checksums: true,
+            suppress_read_repair: true,
+            ..Default::default()
+        };
+        match api.get_object_info(bucket, &listed.name, &opts).await {
+            Ok(info) => items.push(Item::from_info(&info)),
+            Err(_) => {
+                let mut item = Item::from_info(&listed);
+                item.observation_error = Some("metadata_unavailable_or_changed".into());
+                item.protection = Protection::Unknown;
+                item.audit_unsupported = Some("metadata_unavailable_or_changed".into());
+                items.push(item);
+            }
+        }
+    }
+    if api.bucket_incarnation_id_from_disk(bucket).await? != incarnation {
+        return Err(IntegrityError::Conflict);
+    }
+    Ok(InventoryPage {
+        observed_at: time::OffsetDateTime::now_utc().to_string(),
+        bucket_incarnation: incarnation,
+        items,
+        is_truncated: page.is_truncated,
+        next_key_marker: page.next_marker,
+        next_version_marker: page.next_version_idmarker,
+    })
+}
+
+pub async fn create_job(api: Arc<ECStore>, bucket: &str, request: JobRequest) -> Result<Job> {
+    check_bucket(bucket)?;
+    request.validate()?;
+    if request.mode == JobMode::Migrate && !readiness().new_writes_enabled_here {
+        return Err(IntegrityError::NotActivated);
+    }
+    let incarnation = api.bucket_incarnation_id_from_disk(bucket).await?;
+    let job = Job {
+        format_version: 1,
+        id: Uuid::new_v4(),
+        bucket: bucket.into(),
+        bucket_incarnation: incarnation,
+        results: vec![ItemResult::default(); request.items.len()],
+        request,
+        state: JobState::Paused,
+        revision: 0,
+        last_error: None,
+    };
+    save(&api, &job, None, None).await?;
+    Ok(job)
+}
+
+pub async fn get_job(api: Arc<ECStore>, bucket: &str, id: Uuid) -> Result<Job> {
+    Ok(load(&api, bucket, id).await?.0)
+}
+
+async fn load(api: &Arc<ECStore>, bucket: &str, id: Uuid) -> Result<(Job, String)> {
+    check_bucket(bucket)?;
+    let (bytes, info) = match read_config_limited_preserve_empty_with_metadata(api.clone(), &path(bucket, id), 1024 * 1024).await
+    {
+        Ok(value) => value,
+        Err(StorageError::ConfigNotFound | StorageError::FileNotFound) => return Err(IntegrityError::NotFound),
+        Err(error) => return Err(error.into()),
+    };
+    let job: Job = serde_json::from_slice(&bytes)?;
+    job.request.validate()?;
+    if job.format_version != 1
+        || job.bucket != bucket
+        || job.id != id
+        || job.results.len() != job.request.items.len()
+        || api.bucket_incarnation_id_from_disk(bucket).await? != job.bucket_incarnation
+    {
+        return Err(IntegrityError::Conflict);
+    }
+    Ok((job, info.etag.ok_or(IntegrityError::Conflict)?))
+}
+
+async fn save(
+    api: &Arc<ECStore>,
+    job: &Job,
+    etag: Option<&str>,
+    worker: Option<&rustfs_lock::NamespaceLockGuard>,
+) -> Result<String> {
+    let fence = api
+        .acquire_bucket_incarnation_fence(&job.bucket, job.bucket_incarnation)
+        .await?;
+    let mut opts = ObjectOptions {
+        max_parity: true,
+        write_completion: WriteCompletion::TailDrained,
+        http_preconditions: Some(match etag {
+            Some(value) => HTTPPreconditions {
+                if_match: Some(value.into()),
+                ..Default::default()
+            },
+            None => HTTPPreconditions {
+                if_none_match: Some("*".into()),
+                ..Default::default()
+            },
+        }),
+        ..Default::default()
+    };
+    fence.attach_to_object_options(&mut opts);
+    if let Some(guard) = worker {
+        opts.add_namespace_lock_guard(guard);
+    }
+    let api = api.clone();
+    let job = job.clone();
+    // Keep the lifecycle guard through the drained commit even if the HTTP waiter is cancelled.
+    tokio::spawn(async move {
+        save_config_with_opts_quiet(api.clone(), &path(&job.bucket, job.id), serde_json::to_vec(&job)?, &opts).await?;
+        let (stored, next) = load(&api, &job.bucket, job.id).await?;
+        if stored.revision != job.revision {
+            return Err(IntegrityError::Conflict);
+        }
+        drop(fence);
+        Ok(next)
+    })
+    .await
+    .map_err(|_| IntegrityError::Invalid("checkpoint commit task failed"))?
+}
+
+/// Pause/cancel acknowledge intent. An in-flight conditional publication is allowed to drain.
+pub async fn control_job(api: Arc<ECStore>, bucket: &str, id: Uuid, cancel: bool) -> Result<Job> {
+    let (mut job, etag) = load(&api, bucket, id).await?;
+    match job.state {
+        JobState::Complete | JobState::Cancelled | JobState::CancelRequested => return Ok(job),
+        JobState::Paused | JobState::Failed => job.state = if cancel { JobState::Cancelled } else { JobState::Paused },
+        _ => {
+            job.state = if cancel {
+                JobState::CancelRequested
+            } else {
+                JobState::PauseRequested
+            }
+        }
+    }
+    job.revision = job.revision.checked_add(1).ok_or(IntegrityError::Conflict)?;
+    save(&api, &job, Some(&etag), None).await?;
+    Ok(job)
+}
+
+/// Resume is explicit after restart. The distributed lock excludes another worker, including another node.
+pub async fn resume_job(api: Arc<ECStore>, bucket: &str, id: Uuid) -> Result<Job> {
+    check_bucket(bucket)?;
+    let lock = api.new_ns_lock(RUSTFS_META_BUCKET, "integrity-jobs/worker.lock").await?;
+    let guard = lock
+        .get_write_lock_quiet(Duration::from_secs(1))
+        .await
+        .map_err(|_| IntegrityError::Busy)?;
+    let (mut job, etag) = load(&api, bucket, id).await?;
+    if matches!(job.state, JobState::Complete | JobState::Cancelled) {
+        return Ok(job);
+    }
+    if job.state == JobState::CancelRequested {
+        job.state = JobState::Cancelled;
+        job.revision = job.revision.checked_add(1).ok_or(IntegrityError::Conflict)?;
+        save(&api, &job, Some(&etag), Some(&guard)).await?;
+        return Ok(job);
+    }
+    if job.request.mode == JobMode::Migrate && !readiness().new_writes_enabled_here {
+        return Err(IntegrityError::NotActivated);
+    }
+    job.state = JobState::Running;
+    job.last_error = None;
+    job.revision = job.revision.checked_add(1).ok_or(IntegrityError::Conflict)?;
+    let etag = save(&api, &job, Some(&etag), Some(&guard)).await?;
+    let response = job.clone();
+    tokio::spawn(async move {
+        let result = runner::run(&api, &mut job, etag, &guard).await;
+        if let Err(error) = result {
+            // A failed write/CAS stops publication. Never overwrite a concurrent control request.
+            if let Ok((mut latest, etag)) = load(&api, &job.bucket, job.id).await {
+                match latest.state {
+                    JobState::PauseRequested => latest.state = JobState::Paused,
+                    JobState::CancelRequested => latest.state = JobState::Cancelled,
+                    JobState::Running if latest.revision <= job.revision => {
+                        latest.state = JobState::Failed;
+                        latest.last_error = Some(
+                            match error {
+                                IntegrityError::Conflict => "concurrent_change",
+                                IntegrityError::Stopped => "worker_stopped",
+                                IntegrityError::UnsupportedBucket => "unsupported_destination_bucket_configuration",
+                                IntegrityError::NotActivated => "protected_writes_not_activated",
+                                _ => "operation_failed_retry_requires_resume",
+                            }
+                            .into(),
+                        );
+                    }
+                    _ => return,
+                }
+                if let Some(revision) = latest.revision.checked_add(1) {
+                    latest.revision = revision;
+                    let _ = save(&api, &latest, Some(&etag), Some(&guard)).await;
+                }
+            }
+        }
+    });
+    Ok(response)
+}
@@ -0,0 +1,343 @@
+// Copyright 2024 RustFS Team
+// Licensed under the Apache License, Version 2.0.
+
+use super::{IntegrityError, Result};
+use crate::object_api::ObjectInfo;
+use rustfs_rio::{Checksum, read_checksums};
+use serde::{Deserialize, Serialize};
+use sha2::{Digest, Sha256};
+use std::collections::{BTreeMap, HashSet};
+use uuid::Uuid;
+
+pub const MAX_ITEMS: usize = 64;
+pub const MAX_OBJECT_BYTES: u64 = 5 * 1024 * 1024 * 1024;
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct ItemRequest {
+    pub key: String,
+    pub version_id: Option<String>,
+    /// Base64 SHA-256 supplied by the administrator from an independent source.
+    pub expected_sha256: Option<String>,
+    pub target_key: Option<String>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum JobMode {
+    Audit,
+    Migrate,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct JobRequest {
+    pub mode: JobMode,
+    pub items: Vec<ItemRequest>,
+    /// Bounds each source read, destination PUT, and verification stream; one worker runs per cluster.
+    pub bytes_per_second: u64,
+    pub max_object_bytes: u64,
+}
+
+impl JobRequest {
+    pub(crate) fn validate(&self) -> Result<()> {
+        if self.items.is_empty() || self.items.len() > MAX_ITEMS {
+            return Err(IntegrityError::Invalid("a job requires 1..=64 explicit source versions"));
+        }
+        if !(64 * 1024..=1024 * 1024 * 1024).contains(&self.bytes_per_second)
+            || self.max_object_bytes == 0
+            || self.max_object_bytes > MAX_OBJECT_BYTES
+        {
+            return Err(IntegrityError::Invalid("invalid byte rate or staging limit"));
+        }
+        let mut targets = HashSet::new();
+        let sources: HashSet<_> = self.items.iter().map(|i| i.key.as_str()).collect();
+        for item in &self.items {
+            validate_key(&item.key)?;
+            if let Some(version) = &item.version_id
+                && version != "null"
+                && Uuid::parse_str(version).ok().is_none_or(|id| id.is_nil())
+            {
+                return Err(IntegrityError::Invalid("invalid source version"));
+            }
+            if let Some(value) = &item.expected_sha256 {
+                sha256_bytes(value)?;
+            }
+            match (self.mode, &item.target_key) {
+                (JobMode::Audit, None) => {}
+                (JobMode::Migrate, Some(target)) => {
+                    validate_key(target)?;
+                    if sources.contains(target.as_str()) || !targets.insert(target) {
+                        return Err(IntegrityError::Invalid("targets must be distinct from every source and target"));
+                    }
+                }
+                _ => return Err(IntegrityError::Invalid("only migrations require an explicit target key")),
+            }
+        }
+        Ok(())
+    }
+}
+
+fn validate_key(key: &str) -> Result<()> {
+    if key.is_empty() || key.len() > 1024 || key.ends_with('/') || key.contains('\0') {
+        return Err(IntegrityError::Invalid("invalid object key"));
+    }
+    Ok(())
+}
+
+pub(crate) fn sha256_bytes(value: &str) -> Result<Vec<u8>> {
+    let raw = base64_simd::STANDARD
+        .decode_to_vec(value)
+        .map_err(|_| IntegrityError::Invalid("SHA256 must be canonical Base64"))?;
+    if raw.len() != 32 || base64_simd::STANDARD.encode_to_string(&raw) != value {
+        return Err(IntegrityError::Invalid("SHA256 must be canonical Base64 of 32 bytes"));
+    }
+    Ok(raw)
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum Protection {
+    Unknown,
+    Legacy,
+    IndependentCommitment,
+    InvalidDeclaration,
+    NoPayload,
+}
+
+pub(crate) fn protection(info: &ObjectInfo) -> Protection {
+    if info.delete_marker {
+        return Protection::NoPayload;
+    }
+    match rustfs_filemeta::shard_integrity::descriptor_from_metadata(&info.user_defined) {
+        Ok(None) if info.parts.iter().all(|p| p.integrity.is_none()) => Protection::Legacy,
+        Ok(Some(parts))
+            if parts.len() == info.parts.len()
+                && !parts.is_empty()
+                && parts
+                    .iter()
+                    .zip(info.parts.iter())
+                    .all(|(declared, part)| part.integrity.as_ref() == Some(declared)) =>
+        {
+            Protection::IndependentCommitment
+        }
+        _ => Protection::InvalidDeclaration,
+    }
+}
+
+/// Checksums shown in inventory are not automatically accepted as migration evidence.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct InventoryItem {
+    pub key: String,
+    /// S3 selector, including the literal "null" for the null version slot.
+    pub version_id: String,
+    pub data_dir: Option<Uuid>,
+    pub size: i64,
+    pub protection: Protection,
+    pub checksums: BTreeMap<String, String>,
+    pub checksum_is_multipart: bool,
+    pub audit_unsupported: Option<String>,
+    pub source_fingerprint: String,
+    pub observation_error: Option<String>,
+}
+
+impl InventoryItem {
+    pub(crate) fn from_info(info: &ObjectInfo) -> Self {
+        let (checksums, checksum_is_multipart) = info.checksum.as_ref().map_or_else(Default::default, |b| read_checksums(b, 0));
+        Self {
+            key: info.name.clone(),
+            version_id: info
+                .version_id
+                .filter(|id| !id.is_nil())
+                .map_or_else(|| "null".into(), |id| id.to_string()),
+            data_dir: info.data_dir,
+            size: info.size,
+            protection: protection(info),
+            checksums: checksums.into_iter().collect(),
+            checksum_is_multipart,
+            audit_unsupported: unsupported(info).map(str::to_owned),
+            source_fingerprint: fingerprint(info),
+            observation_error: None,
+        }
+    }
+}
+
+pub(crate) fn unsupported(info: &ObjectInfo) -> Option<&'static str> {
+    if info.delete_marker {
+        Some("delete_marker")
+    } else if !info.transitioned_object.status.is_empty() {
+        Some("transitioned_object")
+    } else if info.is_encrypted() || info.is_compressed() {
+        Some("transformed_object")
+    } else if info.is_multipart() {
+        Some("multipart_object")
+    } else if protection(info) == Protection::InvalidDeclaration {
+        Some("invalid_protection_declaration")
+    } else {
+        None
+    }
+}
+
+pub(crate) fn stored_sha256(info: &ObjectInfo) -> Option<String> {
+    let bytes = info.checksum.as_ref()?;
+    let (sums, multipart) = read_checksums(bytes, 0);
+    let value = sums.get("SHA256")?;
+    let canonical = Checksum::new_from_string("SHA256", value)?.to_bytes(&[]);
+    // The general display decoder tolerates truncated suffixes. Eligibility does not.
+    (!multipart && canonical.as_ref() == bytes.as_ref()).then(|| value.clone())
+}
+
+pub(crate) fn fingerprint(info: &ObjectInfo) -> String {
+    let mut hash = Sha256::new();
+    // Ordered metadata includes transformation and protection declarations as well as user metadata.
+    for value in [
+        info.bucket.clone(),
+        info.name.clone(),
+        format!("{:?}", info.version_id),
+        format!("{:?}", info.data_dir),
+        format!("{:?}", info.mod_time),
+        info.size.to_string(),
+        info.actual_size.to_string(),
+        format!("{:?}", info.etag),
+        format!("{:?}", info.checksum),
+        format!("{:?}", info.transitioned_object),
+        info.user_tags.to_string(),
+        format!("{:?}", info.expires),
+        format!("{:?}", info.content_type),
+        format!("{:?}", info.content_encoding),
+        format!("{:?}", info.storage_class),
+    ] {
+        hash.update(value.len().to_le_bytes());
+        hash.update(value.as_bytes());
+    }
+    for part in info.parts.iter() {
+        let values = format!(
+            "{}:{}:{}:{:?}:{:?}:{:?}:{:?}",
+            part.number, part.size, part.actual_size, part.etag, part.mod_time, part.index, part.integrity
+        );
+        hash.update(values.len().to_le_bytes());
+        hash.update(values.as_bytes());
+        if let Some(checksums) = &part.checksums {
+            let mut checksums: Vec<_> = checksums.iter().collect();
+            checksums.sort_unstable_by_key(|(key, _)| *key);
+            for (key, value) in checksums {
+                hash.update(key.len().to_le_bytes());
+                hash.update(key.as_bytes());
+                hash.update(value.len().to_le_bytes());
+                hash.update(value.as_bytes());
+            }
+        }
+    }
+    let mut metadata: Vec<_> = info.user_defined.iter().collect();
+    metadata.sort_unstable_by_key(|(key, _)| *key);
+    for (key, value) in metadata {
+        hash.update(key.len().to_le_bytes());
+        hash.update(key.as_bytes());
+        hash.update(value.len().to_le_bytes());
+        hash.update(value.as_bytes());
+    }
+    hex_simd::encode_to_string(hash.finalize(), hex_simd::AsciiCase::Lower)
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum JobState {
+    Paused,
+    Running,
+    PauseRequested,
+    CancelRequested,
+    Cancelled,
+    Complete,
+    Failed,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum ItemState {
+    Pending,
+    Prepared,
+    Verified,
+    Migrated,
+    Unsupported,
+    Unavailable,
+    Mismatch,
+    Stale,
+    Conflict,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct ItemResult {
+    pub state: ItemState,
+    pub source_fingerprint: Option<String>,
+    pub expected_sha256: Option<String>,
+    pub evidence_source: Option<String>,
+    pub source_size: Option<u64>,
+    pub target_version_id: Option<Uuid>,
+    pub detail: Option<String>,
+}
+
+impl Default for ItemResult {
+    fn default() -> Self {
+        Self {
+            state: ItemState::Pending,
+            source_fingerprint: None,
+            expected_sha256: None,
+            evidence_source: None,
+            source_size: None,
+            target_version_id: None,
+            detail: None,
+        }
+    }
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct Job {
+    pub format_version: u32,
+    pub id: Uuid,
+    pub bucket: String,
+    pub bucket_incarnation: Uuid,
+    pub request: JobRequest,
+    pub state: JobState,
+    pub revision: u64,
+    pub results: Vec<ItemResult>,
+    pub last_error: Option<String>,
+}
+
+#[derive(Debug, Serialize)]
+pub struct InventoryPage {
+    pub observed_at: String,
+    pub bucket_incarnation: Uuid,
+    pub items: Vec<InventoryItem>,
+    pub is_truncated: bool,
+    pub next_key_marker: Option<String>,
+    pub next_version_marker: Option<String>,
+}
+
+#[derive(Debug, Serialize)]
+pub struct Readiness {
+    pub local_reader_supported: bool,
+    pub write_requested: bool,
+    pub fleet_operator_attested: bool,
+    pub new_writes_enabled_here: bool,
+    pub fleet_capability_verified: bool,
+    pub unverified_requirements: [&'static str; 3],
+}
+
+pub fn readiness() -> Readiness {
+    let write = rustfs_utils::get_env_bool(rustfs_config::ENV_SHARD_INTEGRITY_WRITE, false);
+    let fleet = rustfs_utils::get_env_bool(rustfs_config::ENV_SHARD_INTEGRITY_FLEET_CONFIRMED, false);
+    Readiness {
+        local_reader_supported: true,
+        write_requested: write,
+        fleet_operator_attested: fleet,
+        new_writes_enabled_here: write && fleet,
+        fleet_capability_verified: false,
+        unverified_requirements: [
+            "peer_reader_writer_and_repair_capabilities",
+            "old_process_storage_admission",
+            "deployment_failure_and_performance_qualification",
+        ],
+    }
+}
@@ -0,0 +1,468 @@
+// Copyright 2024 RustFS Team
+// Licensed under the Apache License, Version 2.0.
+
+use super::model::{Protection, fingerprint, protection, sha256_bytes, stored_sha256, unsupported};
+use super::{IntegrityError, ItemState, Job, JobMode, JobState, Result, load, readiness, save};
+use crate::object_api::{
+    ObjectInfo, ObjectOptions, PutObjReader, ShardIntegrityWriteMode, WriteCompletion, without_get_object_body_cache_hook,
+};
+use crate::storage_api_contracts::object::{HTTPPreconditions, ObjectIO, ObjectOperations};
+use crate::store::ECStore;
+use rustfs_rio::{Checksum, HashReader};
+use sha2::{Digest, Sha256};
+use std::{collections::HashMap, sync::Arc, time::Duration};
+use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
+
+const MARKER: &str = "integrity-migration-job-v1";
+
+pub(super) async fn run(
+    api: &Arc<ECStore>,
+    job: &mut Job,
+    mut etag: String,
+    guard: &rustfs_lock::NamespaceLockGuard,
+) -> Result<()> {
+    without_get_object_body_cache_hook(async {
+        for index in 0..job.results.len() {
+            ensure_running(api, job, guard).await?;
+            if !matches!(
+                job.results[index].state,
+                ItemState::Pending | ItemState::Prepared | ItemState::Unavailable
+            ) {
+                continue;
+            }
+            if job.request.mode == JobMode::Migrate {
+                validate_destination(api, job).await?;
+            }
+            if job.results[index].state == ItemState::Prepared
+                && job.request.mode == JobMode::Migrate
+                && reconcile_target(api, job, index, guard).await?
+            {
+                etag = checkpoint(api, job, &etag, guard).await?;
+                continue;
+            }
+            let request = job.request.items[index].clone();
+            let opts = ObjectOptions {
+                version_id: request.version_id.as_ref().map(|version| {
+                    if version == "null" {
+                        uuid::Uuid::nil().to_string()
+                    } else {
+                        version.clone()
+                    }
+                }),
+                include_part_checksums: true,
+                suppress_read_repair: true,
+                ..Default::default()
+            };
+            let info = match api.get_object_info(&job.bucket, &request.key, &opts).await {
+                Ok(info) => info,
+                Err(_) => {
+                    job.results[index].state = ItemState::Unavailable;
+                    job.results[index].detail = Some("source_metadata_unavailable".into());
+                    etag = checkpoint(api, job, &etag, guard).await?;
+                    continue;
+                }
+            };
+            if let Some(reason) = unsupported(&info) {
+                job.results[index].state = ItemState::Unsupported;
+                job.results[index].detail = Some(reason.into());
+                etag = checkpoint(api, job, &etag, guard).await?;
+                continue;
+            }
+            let identity = fingerprint(&info);
+            if job.results[index]
+                .source_fingerprint
+                .as_ref()
+                .is_some_and(|old| old != &identity)
+            {
+                job.results[index].state = ItemState::Stale;
+                etag = checkpoint(api, job, &etag, guard).await?;
+                continue;
+            }
+            let expected = request.expected_sha256.clone().or_else(|| stored_sha256(&info));
+            let Some(expected) = expected else {
+                job.results[index].state = ItemState::Unsupported;
+                job.results[index].detail = Some("no_canonical_single_part_sha256".into());
+                etag = checkpoint(api, job, &etag, guard).await?;
+                continue;
+            };
+            let size = u64::try_from(info.size).map_err(|_| IntegrityError::Invalid("negative source size"))?;
+            if size > job.request.max_object_bytes {
+                job.results[index].state = ItemState::Unsupported;
+                job.results[index].detail = Some("object_exceeds_staging_budget".into());
+                etag = checkpoint(api, job, &etag, guard).await?;
+                continue;
+            }
+            let result = &mut job.results[index];
+            result.source_fingerprint = Some(identity);
+            result.expected_sha256 = Some(expected.clone());
+            result.evidence_source = Some(
+                if request.expected_sha256.is_some() {
+                    "administrator_supplied_sha256"
+                } else {
+                    "stored_sha256"
+                }
+                .into(),
+            );
+            result.source_size = Some(size);
+            result.state = ItemState::Prepared;
+            result.detail = None;
+            etag = checkpoint(api, job, &etag, guard).await?;
+            let mut file = match stage(api, job, index, &info, &opts, guard).await {
+                Ok(file) => file,
+                Err(IntegrityError::Conflict) => {
+                    job.results[index].state = ItemState::Stale;
+                    etag = checkpoint(api, job, &etag, guard).await?;
+                    continue;
+                }
+                Err(IntegrityError::Invalid("content_sha256_mismatch")) => {
+                    job.results[index].state = ItemState::Mismatch;
+                    etag = checkpoint(api, job, &etag, guard).await?;
+                    continue;
+                }
+                Err(error) => return Err(error),
+            };
+            ensure_running(api, job, guard).await?;
+            if job.request.mode == JobMode::Audit {
+                job.results[index].state = ItemState::Verified;
+            } else {
+                let snapshot = validate_destination(api, job).await?;
+                file.rewind().await?;
+                let raw = sha256_bytes(&expected)?;
+                let expected_hex = hex_simd::encode_to_string(raw, hex_simd::AsciiCase::Lower);
+                let mut reader = HashReader::from_stream(
+                    ThrottledFile::new(file, job.request.bytes_per_second),
+                    info.size,
+                    info.size,
+                    None,
+                    Some(expected_hex),
+                    false,
+                )?;
+                reader.add_non_trailing_checksum(Checksum::new_from_string("SHA256", &expected), false)?;
+                let mut body = PutObjReader::new(reader);
+                let mut metadata: HashMap<String, String> = info
+                    .user_defined
+                    .iter()
+                    .filter(|(key, _)| {
+                        key.starts_with("x-amz-meta-")
+                            || matches!(key.as_str(), "cache-control" | "content-disposition" | "content-language" | "expires")
+                    })
+                    .map(|(key, value)| (key.clone(), value.clone()))
+                    .collect();
+                if let Some(content_type) = &info.content_type {
+                    metadata.insert("content-type".into(), content_type.clone());
+                }
+                if let Some(content_encoding) = &info.content_encoding {
+                    metadata.insert("content-encoding".into(), content_encoding.clone());
+                }
+                if let Some(expires) = info.expires {
+                    metadata.insert(
+                        "expires".into(),
+                        expires
+                            .format(&time::format_description::well_known::Rfc3339)
+                            .map_err(|_| IntegrityError::Invalid("invalid source expiration"))?,
+                    );
+                }
+                if let Some(storage_class) = &info.storage_class {
+                    metadata.insert("x-amz-storage-class".into(), storage_class.clone());
+                }
+                if !info.user_tags.is_empty() {
+                    metadata.insert(rustfs_utils::http::headers::AMZ_OBJECT_TAGGING.into(), info.user_tags.to_string());
+                }
+                rustfs_utils::http::insert_str(&mut metadata, MARKER, marker(job, index)?);
+                let mut write = ObjectOptions {
+                    user_defined: metadata,
+                    preserve_delete_marker: true,
+                    shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Protected),
+                    write_completion: WriteCompletion::TailDrained,
+                    http_preconditions: Some(HTTPPreconditions {
+                        if_none_match: Some("*".into()),
+                        ..Default::default()
+                    }),
+                    ..Default::default()
+                };
+                write.add_namespace_lock_guard(guard);
+                write.expected_bucket_incarnation_id = Some(job.bucket_incarnation);
+                snapshot.add_lock_fences(&mut write);
+                write.object_lock_config_snapshot = Some(snapshot.clone());
+                let target = request
+                    .target_key
+                    .as_deref()
+                    .ok_or(IntegrityError::Invalid("missing target"))?;
+                // Even an error may follow a committed PUT. Reconcile before retrying; never overwrite.
+                let put = api.put_object(&job.bucket, target, &mut body, &write).await;
+                drop(write);
+                drop(snapshot);
+                if !reconcile_target(api, job, index, guard).await? {
+                    put?;
+                    return Err(IntegrityError::Conflict);
+                }
+            }
+            etag = checkpoint(api, job, &etag, guard).await?;
+        }
+        job.state = if job.results.iter().any(|item| item.state == ItemState::Unavailable) {
+            job.last_error = Some("source_unavailable_retry_requires_resume".into());
+            JobState::Failed
+        } else {
+            JobState::Complete
+        };
+        checkpoint(api, job, &etag, guard).await?;
+        Ok(())
+    })
+    .await
+}
+
+async fn checkpoint(api: &Arc<ECStore>, job: &mut Job, etag: &str, guard: &rustfs_lock::NamespaceLockGuard) -> Result<String> {
+    ensure_running(api, job, guard).await?;
+    job.revision = job.revision.checked_add(1).ok_or(IntegrityError::Conflict)?;
+    save(api, job, Some(etag), Some(guard)).await
+}
+
+async fn ensure_running(api: &Arc<ECStore>, job: &Job, guard: &rustfs_lock::NamespaceLockGuard) -> Result<()> {
+    if guard.lock_lost_signal().is_some_and(|signal| signal.is_lost()) {
+        return Err(IntegrityError::Stopped);
+    }
+    let (current, _) = load(api, &job.bucket, job.id).await?;
+    if current.state != JobState::Running {
+        return Err(IntegrityError::Stopped);
+    }
+    if current.revision != job.revision {
+        return Err(IntegrityError::Conflict);
+    }
+    Ok(())
+}
+
+async fn stage(
+    api: &Arc<ECStore>,
+    job: &Job,
+    index: usize,
+    info: &ObjectInfo,
+    opts: &ObjectOptions,
+    guard: &rustfs_lock::NamespaceLockGuard,
+) -> Result<tokio::fs::File> {
+    let mut reader = api
+        .get_object_reader(&job.bucket, &job.request.items[index].key, None, Default::default(), opts)
+        .await?;
+    if fingerprint(&reader.object_info) != fingerprint(info) {
+        return Err(IntegrityError::Conflict);
+    }
+    let std_file = tokio::task::spawn_blocking(tempfile::tempfile)
+        .await
+        .map_err(|_| IntegrityError::Invalid("staging task failed"))??;
+    let mut file = tokio::fs::File::from_std(std_file);
+    let expected = job.results[index]
+        .expected_sha256
+        .as_deref()
+        .ok_or(IntegrityError::Invalid("missing expected digest"))?;
+    copy_verified(api, job, &mut reader.stream, Some(&mut file), info.size, expected, guard).await?;
+    file.flush().await?;
+    let after = api.get_object_info(&job.bucket, &job.request.items[index].key, opts).await?;
+    if fingerprint(&after) != fingerprint(info) {
+        return Err(IntegrityError::Conflict);
+    }
+    Ok(file)
+}
+
+async fn copy_verified(
+    api: &Arc<ECStore>,
+    job: &Job,
+    input: &mut (dyn tokio::io::AsyncRead + Unpin + Send + Sync),
+    mut output: Option<&mut tokio::fs::File>,
+    size: i64,
+    expected: &str,
+    guard: &rustfs_lock::NamespaceLockGuard,
+) -> Result<()> {
+    let expected = sha256_bytes(expected)?;
+    let expected_size = u64::try_from(size).map_err(|_| IntegrityError::Invalid("negative object size"))?;
+    let mut buffer = vec![0u8; 64 * 1024];
+    let mut count = 0u64;
+    let mut hash = Sha256::new();
+    let started = tokio::time::Instant::now();
+    let mut checked = started;
+    loop {
+        if checked.elapsed() >= Duration::from_secs(1) {
+            ensure_running(api, job, guard).await?;
+            checked = tokio::time::Instant::now();
+        }
+        let n = tokio::time::timeout(Duration::from_secs(30), input.read(&mut buffer))
+            .await
+            .map_err(|_| std::io::Error::new(std::io::ErrorKind::TimedOut, "integrity read timed out"))??;
+        if n == 0 {
+            break;
+        }
+        count = count
+            .checked_add(u64::try_from(n).map_err(|_| IntegrityError::Invalid("read size overflow"))?)
+            .ok_or(IntegrityError::Invalid("read size overflow"))?;
+        if count > expected_size || count > job.request.max_object_bytes {
+            return Err(IntegrityError::Invalid("content_sha256_mismatch"));
+        }
+        hash.update(&buffer[..n]);
+        if let Some(file) = output.as_deref_mut() {
+            file.write_all(&buffer[..n]).await?;
+        }
+        let delay = Duration::from_secs_f64(count as f64 / job.request.bytes_per_second as f64);
+        tokio::time::sleep_until(started + delay).await;
+    }
+    if count != expected_size || hash.finalize().as_slice() != expected {
+        return Err(IntegrityError::Invalid("content_sha256_mismatch"));
+    }
+    Ok(())
+}
+
+fn marker(job: &Job, index: usize) -> Result<String> {
+    let result = &job.results[index];
+    Ok(format!(
+        "{}:{}:{}:{}",
+        job.id,
+        index,
+        result
+            .source_fingerprint
+            .as_deref()
+            .ok_or(IntegrityError::Invalid("missing source identity"))?,
+        result
+            .expected_sha256
+            .as_deref()
+            .ok_or(IntegrityError::Invalid("missing source digest"))?
+    ))
+}
+
+async fn reconcile_target(
+    api: &Arc<ECStore>,
+    job: &mut Job,
+    index: usize,
+    guard: &rustfs_lock::NamespaceLockGuard,
+) -> Result<bool> {
+    let key = job.request.items[index]
+        .target_key
+        .as_deref()
+        .ok_or(IntegrityError::Invalid("missing target"))?;
+    let opts = ObjectOptions {
+        include_part_checksums: true,
+        suppress_read_repair: true,
+        ..Default::default()
+    };
+    let target = match api.get_object_info(&job.bucket, key, &opts).await {
+        Ok(info) => info,
+        Err(crate::error::Error::FileNotFound | crate::error::Error::ObjectNotFound(_, _)) => return Ok(false),
+        Err(error) => return Err(error.into()),
+    };
+    let expected_marker = marker(job, index)?;
+    if rustfs_utils::http::get_consistent_str(&target.user_defined, MARKER) != Some(expected_marker.as_str()) {
+        job.results[index].state = ItemState::Conflict;
+        job.results[index].detail = Some("target_owned_by_another_operation".into());
+        return Ok(true);
+    }
+    if protection(&target) != Protection::IndependentCommitment {
+        return Err(IntegrityError::Invalid("target_not_protected"));
+    }
+    let mut reader = api
+        .get_object_reader(&job.bucket, key, None, Default::default(), &opts)
+        .await?;
+    if fingerprint(&target) != fingerprint(&reader.object_info) {
+        return Err(IntegrityError::Conflict);
+    }
+    let size = job.results[index]
+        .source_size
+        .ok_or(IntegrityError::Invalid("missing source size"))?;
+    let expected = job.results[index]
+        .expected_sha256
+        .clone()
+        .ok_or(IntegrityError::Invalid("missing source digest"))?;
+    copy_verified(
+        api,
+        job,
+        &mut reader.stream,
+        None,
+        i64::try_from(size).map_err(|_| IntegrityError::Invalid("source size overflow"))?,
+        &expected,
+        guard,
+    )
+    .await?;
+    let after = api.get_object_info(&job.bucket, key, &opts).await?;
+    if fingerprint(&after) != fingerprint(&target) {
+        return Err(IntegrityError::Conflict);
+    }
+    job.results[index].state = ItemState::Migrated;
+    job.results[index].target_version_id = target.version_id;
+    Ok(true)
+}
+
+async fn validate_destination(api: &Arc<ECStore>, job: &Job) -> Result<Arc<crate::object_api::ObjectLockConfigSnapshot>> {
+    if !readiness().new_writes_enabled_here {
+        return Err(IntegrityError::NotActivated);
+    }
+    let snapshot = api.object_lock_config_snapshot(&job.bucket).await?;
+    if snapshot
+        .metadata_transaction_guard_for(api.id, &job.bucket, Some(job.bucket_incarnation))
+        .is_none()
+    {
+        return Err(IntegrityError::Conflict);
+    }
+    let metadata = crate::bucket::metadata_sys::integrity_migration_metadata_in(&api.ctx, &job.bucket).await?;
+    // The direct storage writer is not the S3 orchestration layer. Reject configurations it cannot preserve.
+    if metadata.bucket_incarnation_id != job.bucket_incarnation {
+        return Err(IntegrityError::Conflict);
+    }
+    if !metadata.versioning_config_xml.is_empty()
+        || !metadata.encryption_config_xml.is_empty()
+        || !metadata.object_lock_config_xml.is_empty()
+        || !metadata.replication_config_xml.is_empty()
+        || !metadata.notification_config_xml.is_empty()
+        || !metadata.quota_config_json.is_empty()
+        || !metadata.on_demand_migration_config_json.is_empty()
+        || !metadata.lifecycle_config_xml.is_empty()
+        || !metadata.table_bucket_config_json.is_empty()
+        || !metadata.bucket_acl_config_json.is_empty()
+        || !metadata.logging_config_xml.is_empty()
+    {
+        return Err(IntegrityError::UnsupportedBucket);
+    }
+    Ok(snapshot)
+}
+
+/// Bound publication throughput as well as the source and destination verification reads.
+struct ThrottledFile {
+    file: tokio::fs::File,
+    started: tokio::time::Instant,
+    count: u64,
+    rate: u64,
+    sleep: std::pin::Pin<Box<tokio::time::Sleep>>,
+}
+
+impl ThrottledFile {
+    fn new(file: tokio::fs::File, rate: u64) -> Self {
+        let started = tokio::time::Instant::now();
+        Self {
+            file,
+            started,
+            count: 0,
+            rate,
+            sleep: Box::pin(tokio::time::sleep_until(started)),
+        }
+    }
+}
+
+impl tokio::io::AsyncRead for ThrottledFile {
+    fn poll_read(
+        mut self: std::pin::Pin<&mut Self>,
+        cx: &mut std::task::Context<'_>,
+        output: &mut tokio::io::ReadBuf<'_>,
+    ) -> std::task::Poll<std::io::Result<()>> {
+        use std::future::Future;
+        if self.sleep.as_mut().poll(cx).is_pending() {
+            return std::task::Poll::Pending;
+        }
+        let capacity = output.remaining().min(64 * 1024);
+        let mut buffer = tokio::io::ReadBuf::new(output.initialize_unfilled_to(capacity));
+        match std::pin::Pin::new(&mut self.file).poll_read(cx, &mut buffer) {
+            std::task::Poll::Ready(Ok(())) => {
+                let count = buffer.filled().len();
+                output.advance(count);
+                self.count += count as u64;
+                let deadline = self.started + Duration::from_secs_f64(self.count as f64 / self.rate as f64);
+                self.sleep.as_mut().reset(deadline);
+                std::task::Poll::Ready(Ok(()))
+            }
+            result => result,
+        }
+    }
+}
@@ -0,0 +1,387 @@
+// Copyright 2024 RustFS Team
+// Licensed under the Apache License, Version 2.0.
+
+use super::*;
+use crate::object_api::{ObjectOptions, PutObjReader, ShardIntegrityWriteMode};
+use crate::storage_api_contracts::{
+    bucket::{BucketOperations, MakeBucketOptions},
+    object::{ObjectIO, ObjectOperations},
+};
+use tokio::io::AsyncReadExt;
+
+async fn fixture() -> (Vec<tempfile::TempDir>, Arc<crate::store::ECStore>, String) {
+    let (dirs, store) = crate::services::rebalance::test_store_with_persisted_rebalance_meta(Default::default()).await;
+    crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+    let bucket = format!("integrity-{}", Uuid::new_v4());
+    store
+        .make_bucket(&bucket, &MakeBucketOptions::default())
+        .await
+        .expect("fixture bucket");
+    (dirs, store, bucket)
+}
+
+fn request(mode: JobMode, digest: String) -> JobRequest {
+    JobRequest {
+        mode,
+        items: vec![ItemRequest {
+            key: "source".into(),
+            version_id: Some("null".into()),
+            expected_sha256: Some(digest),
+            target_key: (mode == JobMode::Migrate).then(|| "target".into()),
+        }],
+        bytes_per_second: 1024 * 1024,
+        max_object_bytes: 16 * 1024 * 1024,
+    }
+}
+
+async fn finish(store: Arc<crate::store::ECStore>, bucket: &str, id: Uuid) -> Job {
+    tokio::time::timeout(std::time::Duration::from_secs(60), async {
+        loop {
+            let job = get_job(store.clone(), bucket, id).await.expect("durable job");
+            if job.state != JobState::Running {
+                return job;
+            }
+            tokio::time::sleep(std::time::Duration::from_millis(20)).await;
+        }
+    })
+    .await
+    .expect("job must finish")
+}
+
+#[tokio::test]
+#[serial_test::serial(shard_integrity_rollout)]
+async fn audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts() {
+    temp_env::async_with_vars(
+        [
+            (rustfs_config::ENV_SHARD_INTEGRITY_WRITE, Some("true")),
+            (rustfs_config::ENV_SHARD_INTEGRITY_FLEET_CONFIRMED, Some("true")),
+        ],
+        async {
+            let (_dirs, store, bucket) = fixture().await;
+            let payload = vec![42; 256 * 1024 + 17];
+            let expected = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::SHA256, &payload)
+                .expect("digest")
+                .encoded;
+            store
+                .put_object(
+                    &bucket,
+                    "source",
+                    &mut PutObjReader::from_vec(payload.clone()),
+                    &ObjectOptions {
+                        user_defined: [
+                            ("cache-control".into(), "max-age=300".into()),
+                            ("content-disposition".into(), "attachment; filename=sample.bin".into()),
+                            ("expires".into(), "2030-01-01T00:00:00Z".into()),
+                            (rustfs_utils::http::headers::AMZ_OBJECT_TAGGING.into(), "kind=archive".into()),
+                        ]
+                        .into(),
+                        shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy),
+                        write_completion: crate::object_api::WriteCompletion::TailDrained,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("source");
+            let source = store
+                .get_object_info(&bucket, "source", &ObjectOptions::default())
+                .await
+                .expect("source metadata");
+            assert_eq!(source.user_tags.as_str(), "kind=archive", "source fixture tags");
+            let before = model::fingerprint(&source);
+            let page = inventory(store.clone(), &bucket, "", None, None, 1).await.expect("inventory");
+            assert_eq!(page.items.len(), 1);
+            assert_eq!(page.items[0].protection, Protection::Legacy);
+            assert_eq!(page.items[0].version_id, "null", "inventory must round-trip an explicit null selector");
+            let audit = create_job(store.clone(), &bucket, request(JobMode::Audit, expected.clone()))
+                .await
+                .expect("create audit");
+            resume_job(store.clone(), &bucket, audit.id).await.expect("resume audit");
+            let audited = finish(store.clone(), &bucket, audit.id).await;
+            assert_eq!(audited.state, JobState::Complete, "{audited:?}");
+            assert_eq!(audited.results[0].state, ItemState::Verified);
+
+            let migration = create_job(store.clone(), &bucket, request(JobMode::Migrate, expected.clone()))
+                .await
+                .expect("create migration");
+            resume_job(store.clone(), &bucket, migration.id)
+                .await
+                .expect("resume migration");
+            let migrated = finish(store.clone(), &bucket, migration.id).await;
+            assert_eq!(migrated.state, JobState::Complete, "{migrated:?}");
+            assert_eq!(migrated.results[0].state, ItemState::Migrated);
+            let mut reader = store
+                .get_object_reader(&bucket, "target", None, Default::default(), &ObjectOptions::default())
+                .await
+                .expect("target");
+            assert_eq!(model::protection(&reader.object_info), Protection::IndependentCommitment);
+            assert_eq!(reader.object_info.user_tags.as_str(), "kind=archive");
+            assert_eq!(reader.object_info.expires, source.expires);
+            assert_eq!(
+                reader.object_info.user_defined.get("cache-control"),
+                source.user_defined.get("cache-control")
+            );
+            assert_eq!(
+                reader.object_info.user_defined.get("content-disposition"),
+                source.user_defined.get("content-disposition")
+            );
+            let mut actual = Vec::new();
+            reader.stream.read_to_end(&mut actual).await.expect("target content");
+            assert_eq!(actual, payload);
+            let after = store
+                .get_object_info(&bucket, "source", &ObjectOptions::default())
+                .await
+                .expect("source after");
+            assert_eq!(model::fingerprint(&after), before);
+
+            // Simulate a lost completion checkpoint after the create-only PUT committed.
+            let (mut replay, etag) = load(&store, &bucket, migration.id).await.expect("load receipt");
+            replay.state = JobState::Paused;
+            replay.results[0].state = ItemState::Prepared;
+            replay.revision += 1;
+            save(&store, &replay, Some(&etag), None)
+                .await
+                .expect("lost acknowledgement fixture");
+            resume_job(store.clone(), &bucket, replay.id)
+                .await
+                .expect("reconcile after restart");
+            let reconciled = finish(store.clone(), &bucket, replay.id).await;
+            assert_eq!(reconciled.results[0].state, ItemState::Migrated, "{reconciled:?}");
+
+            let conflict = create_job(store.clone(), &bucket, request(JobMode::Migrate, expected))
+                .await
+                .expect("conflicting job");
+            resume_job(store.clone(), &bucket, conflict.id)
+                .await
+                .expect("resume conflict");
+            let conflict = finish(store.clone(), &bucket, conflict.id).await;
+            assert_eq!(conflict.results[0].state, ItemState::Conflict, "{conflict:?}");
+            let target = store
+                .get_object_info(&bucket, "target", &ObjectOptions::default())
+                .await
+                .expect("unchanged target");
+            assert_eq!(target.data_dir, reader.object_info.data_dir);
+        },
+    )
+    .await;
+}
+
+#[tokio::test]
+async fn mismatch_never_publishes_and_paused_job_survives_reload() {
+    let (_dirs, store, bucket) = fixture().await;
+    store
+        .put_object(
+            &bucket,
+            "source",
+            &mut PutObjReader::from_vec(b"actual".to_vec()),
+            &ObjectOptions {
+                shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy),
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("source");
+    let wrong = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::SHA256, b"different")
+        .expect("digest")
+        .encoded;
+    let job = create_job(store.clone(), &bucket, request(JobMode::Audit, wrong))
+        .await
+        .expect("job");
+    assert_eq!(get_job(store.clone(), &bucket, job.id).await.expect("reload").state, JobState::Paused);
+    resume_job(store.clone(), &bucket, job.id).await.expect("resume");
+    let job = finish(store.clone(), &bucket, job.id).await;
+    assert_eq!(job.results[0].state, ItemState::Mismatch, "{job:?}");
+    assert!(
+        store
+            .get_object_info(&bucket, "target", &ObjectOptions::default())
+            .await
+            .is_err()
+    );
+}
+
+#[test]
+fn fingerprint_is_independent_of_metadata_insertion_order() {
+    let mut a = crate::object_api::ObjectInfo::default();
+    let mut first = std::collections::HashMap::new();
+    first.insert("a".into(), "one".into());
+    first.insert("b".into(), "two".into());
+    a.user_defined = Arc::new(first);
+    let mut b = a.clone();
+    let mut second = std::collections::HashMap::new();
+    second.insert("b".into(), "two".into());
+    second.insert("a".into(), "one".into());
+    b.user_defined = Arc::new(second);
+    assert_eq!(model::fingerprint(&a), model::fingerprint(&b));
+    b.data_dir = Some(Uuid::new_v4());
+    assert_ne!(model::fingerprint(&a), model::fingerprint(&b));
+}
+
+#[test]
+fn fingerprint_is_independent_of_part_checksum_insertion_order() {
+    let mut checksums = [
+        ("SHA256", "sha256"),
+        ("SHA1", "sha1"),
+        ("CRC32", "crc32"),
+        ("CRC32C", "crc32c"),
+    ];
+    let a = crate::object_api::ObjectInfo {
+        parts: Arc::new(vec![rustfs_filemeta::ObjectPartInfo {
+            checksums: Some(checksums.map(|(key, value)| (key.to_string(), value.to_string())).into()),
+            ..Default::default()
+        }]),
+        ..Default::default()
+    };
+    let mut b = a.clone();
+    let expected = model::fingerprint(&a);
+    for _ in 0..checksums.len() {
+        checksums.rotate_left(1);
+        Arc::make_mut(&mut b.parts)[0].checksums =
+            Some(checksums.map(|(key, value)| (key.to_string(), value.to_string())).into());
+        assert_eq!(expected, model::fingerprint(&b));
+    }
+    Arc::make_mut(&mut b.parts)[0]
+        .checksums
+        .as_mut()
+        .expect("part checksums")
+        .insert("SHA256".into(), "changed".into());
+    assert_ne!(expected, model::fingerprint(&b));
+}
+
+#[test]
+fn rejects_migration_into_a_source_key() {
+    let request = JobRequest {
+        mode: JobMode::Migrate,
+        items: vec![ItemRequest {
+            key: "source".into(),
+            version_id: None,
+            expected_sha256: None,
+            target_key: Some("source".into()),
+        }],
+        bytes_per_second: 1024 * 1024,
+        max_object_bytes: 1024,
+    };
+    assert!(request.validate().is_err());
+}
+
+#[test]
+fn rejects_truncated_checksum_suffix_as_migration_evidence() {
+    let checksum = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::SHA256, b"payload").expect("checksum");
+    let mut info = crate::object_api::ObjectInfo {
+        checksum: Some(checksum.to_bytes(&[])),
+        ..Default::default()
+    };
+    assert_eq!(model::stored_sha256(&info), Some(checksum.encoded));
+    let mut bytes = info.checksum.take().expect("bytes").to_vec();
+    bytes.push(0x80);
+    info.checksum = Some(bytes.into());
+    assert!(model::stored_sha256(&info).is_none());
+}
+
+#[tokio::test]
+async fn unavailable_source_can_resume_and_cancel_intent_cannot_restart() {
+    let (_dirs, store, bucket) = fixture().await;
+    let payload = b"recovered".to_vec();
+    let digest = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::SHA256, &payload)
+        .expect("digest")
+        .encoded;
+    let job = create_job(store.clone(), &bucket, request(JobMode::Audit, digest))
+        .await
+        .expect("create");
+    resume_job(store.clone(), &bucket, job.id).await.expect("start");
+    let failed = finish(store.clone(), &bucket, job.id).await;
+    assert_eq!(failed.state, JobState::Failed);
+    assert_eq!(failed.results[0].state, ItemState::Unavailable);
+    store
+        .put_object(
+            &bucket,
+            "source",
+            &mut PutObjReader::from_vec(payload),
+            &ObjectOptions {
+                shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy),
+                write_completion: WriteCompletion::TailDrained,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("recovered source");
+    resume_job(store.clone(), &bucket, job.id).await.expect("retry");
+    let completed = finish(store.clone(), &bucket, job.id).await;
+    assert_eq!(completed.state, JobState::Complete, "{completed:?}");
+    assert_eq!(completed.results[0].state, ItemState::Verified);
+
+    // A process restart may leave the durable cancel intent without a live worker.
+    let (mut interrupted, etag) = load(&store, &bucket, job.id).await.expect("load");
+    interrupted.state = JobState::CancelRequested;
+    interrupted.revision += 1;
+    save(&store, &interrupted, Some(&etag), None).await.expect("persist intent");
+    assert_eq!(
+        control_job(store.clone(), &bucket, job.id, false).await.expect("pause").state,
+        JobState::CancelRequested
+    );
+    assert_eq!(
+        resume_job(store.clone(), &bucket, job.id)
+            .await
+            .expect("recover cancel")
+            .state,
+        JobState::Cancelled
+    );
+}
+
+#[tokio::test]
+async fn pause_and_cancel_are_durable_during_content_read() {
+    let (_dirs, store, bucket) = fixture().await;
+    let payload = vec![7; 512 * 1024];
+    let digest = rustfs_rio::Checksum::new_from_data(rustfs_rio::ChecksumType::SHA256, &payload)
+        .expect("digest")
+        .encoded;
+    store
+        .put_object(
+            &bucket,
+            "source",
+            &mut PutObjReader::from_vec(payload),
+            &ObjectOptions {
+                shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy),
+                write_completion: WriteCompletion::TailDrained,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("source");
+    let mut req = request(JobMode::Audit, digest);
+    req.bytes_per_second = 64 * 1024;
+    let job = create_job(store.clone(), &bucket, req).await.expect("create");
+    resume_job(store.clone(), &bucket, job.id).await.expect("start");
+    tokio::time::timeout(std::time::Duration::from_secs(20), async {
+        loop {
+            if get_job(store.clone(), &bucket, job.id).await.expect("status").results[0].state == ItemState::Prepared {
+                break;
+            }
+            tokio::time::sleep(std::time::Duration::from_millis(20)).await;
+        }
+    })
+    .await
+    .expect("prepared");
+    let intent = control_job(store.clone(), &bucket, job.id, false).await.expect("pause");
+    assert_eq!(intent.state, JobState::PauseRequested);
+    tokio::time::timeout(std::time::Duration::from_secs(20), async {
+        loop {
+            let paused = get_job(store.clone(), &bucket, job.id).await.expect("status");
+            if paused.state == JobState::Paused {
+                break;
+            }
+            tokio::time::sleep(std::time::Duration::from_millis(20)).await;
+        }
+    })
+    .await
+    .expect("pause acknowledged");
+    assert_eq!(
+        control_job(store.clone(), &bucket, job.id, true).await.expect("cancel").state,
+        JobState::Cancelled
+    );
+    assert_eq!(
+        resume_job(store.clone(), &bucket, job.id)
+            .await
+            .expect("terminal resume")
+            .state,
+        JobState::Cancelled
+    );
+}
@@ -16,6 +16,7 @@

 pub(crate) mod batch_processor;
 pub(crate) mod event_notification;
+pub(crate) mod integrity;
 pub(crate) mod metrics_realtime;
 pub(crate) mod notification_sys;
 pub(crate) mod rebalance;
@@ -7,8 +7,9 @@ use super::meta::{
    validate_start_rebalance_state,
 };
 use super::worker::{
-    resolve_rebalance_bucket_result, resolve_rebalance_meta_save_result, resolve_rebalance_save_task_result,
-    resolve_rebalance_terminal_error, send_rebalance_done_signal,
+    rebalance_max_attempts, resolve_rebalance_bucket_result, resolve_rebalance_meta_save_result,
+    resolve_rebalance_save_task_result, resolve_rebalance_terminal_error, retry_rebalance_metadata_access,
+    send_rebalance_done_signal,
 };
 use super::{
    EVENT_REBALANCE_BUCKET, EVENT_REBALANCE_STATE, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_REBALANCE,
@@ -449,13 +450,14 @@ impl ECStore {
                        };

                        if terminal_state_present {
-                            if let Err(err) = store
-                                .save_rebalance_stats_inner(
+                            if let Err(err) = retry_rebalance_metadata_access(None, rebalance_max_attempts(), || {
+                                store.save_rebalance_stats_inner(
                                    pool_index,
                                    RebalSaveOpt::Stats,
                                    Some(save_rebalance_id.as_ref()),
                                )
-                                .await
+                            })
+                            .await
                            {
                                let mut rebalance_meta = store.rebalance_meta.write().await;
                                *rebalance_meta = previous_meta;
@@ -475,9 +477,10 @@ impl ECStore {
                }

                if !terminal_state_saved
-                    && let Err(err) = store
-                        .save_rebalance_stats_for_id(pool_index, RebalSaveOpt::Stats, save_rebalance_id.as_ref())
-                        .await
+                    && let Err(err) = retry_rebalance_metadata_access(None, rebalance_max_attempts(), || {
+                        store.save_rebalance_stats_for_id(pool_index, RebalSaveOpt::Stats, save_rebalance_id.as_ref())
+                    })
+                    .await
                {
                    let wrapped = Error::other(format!("rebalance save_task stats save failed for pool {pool_index}: {err}"));
                    error!("{} err: {:?}", msg, wrapped);
@@ -109,7 +109,13 @@ pub(super) fn resolve_rebalance_save_task_result(
 }

 pub(super) fn resolve_rebalance_meta_save_result(result: Result<()>, stage: &str) -> Result<()> {
-    result.map_err(|err| Error::other(format!("rebalance meta save failed during {stage}: {err}")))
+    // Keep the source error reachable: the metadata retry policy classifies
+    // transient lock timeouts by inspecting the source chain, so collapsing the
+    // failure into a plain string here would make that retry a no-op.
+    result.map_err(|err| {
+        let rendered = format!("rebalance meta save failed during {stage}: {err}");
+        crate::data_movement::data_movement_context_error(rendered, err)
+    })
 }

 pub(super) fn rebalance_meta_lock_error(err: rustfs_lock::LockError, mode: &'static str) -> Error {
@@ -739,6 +745,29 @@ mod error_source_tests {
        }
    }

+    #[tokio::test]
+    async fn rebalance_metadata_retry_engages_for_wrapped_meta_save_lock_timeout() {
+        let mut attempts = 0;
+        let result = retry_rebalance_metadata_access(None, 3, || {
+            attempts += 1;
+            std::future::ready(resolve_rebalance_meta_save_result(
+                Err(rebalance_meta_lock_error(
+                    rustfs_lock::LockError::timeout(".rustfs.sys/rebalance.bin@latest", Duration::from_secs(5)),
+                    "write",
+                )),
+                "save_rebalance_stats for pool 0 opt Stats",
+            ))
+        })
+        .await;
+
+        assert_eq!(attempts, 3, "a wrapped meta save lock timeout must stay retryable");
+        let err = result.expect_err("persistent lock contention must not become success");
+        assert!(matches!(
+            rebalance_error_source(&err),
+            Error::Lock(rustfs_lock::LockError::Timeout { .. })
+        ));
+    }
+
    #[tokio::test]
    async fn rebalance_metadata_retry_cancels_a_pending_attempt() {
        struct DropProbe(Arc<std::sync::atomic::AtomicUsize>);
@@ -1163,13 +1163,10 @@ pub(in crate::set_disk) struct ReadRepairHealSubmission<'a> {
    pub(in crate::set_disk) set_index: usize,
    pub(in crate::set_disk) part_number: Option<usize>,
    pub(in crate::set_disk) reason: &'static str,
-    /// Durable MRF journal intent to file alongside the read-repair request
-    /// (backlog#1894 axis A): the intent kind plus its native `Uuid`
-    /// version id (the submission's string form stays display-only). Bound
-    /// to the reservation — the intent is only delivered when this sighting
-    /// wins the dedup TTL, so a burst of reads failing on the same object
-    /// books exactly one journal record instead of one per retry. `None`
-    /// keeps the historical no-intent behavior.
+    /// Best-effort ingress for the durable MRF consumer. Its identity coalescer
+    /// is independent of the heal reservation: an earlier metadata repair must
+    /// not suppress newly observed payload damage, and rejected ingress must
+    /// remain retryable. Channel admission does not prove checkpoint commit.
    pub(in crate::set_disk) mrf_intent: Option<(rustfs_common::mrf_channel::MrfKind, Option<uuid::Uuid>)>,
 }

@@ -1224,6 +1221,27 @@ pub(in crate::set_disk) async fn submit_read_repair_heal_with_submitter(
        mrf_intent,
    } = submission;

+    if let Some((kind, version_uuid)) = mrf_intent
+        && let (Ok(pool_index), Ok(set_index)) = (u32::try_from(pool_index), u32::try_from(set_index))
+    {
+        let scope = rustfs_common::mrf_channel::MrfScope { pool_index, set_index };
+        let ingress = rustfs_common::mrf_channel::try_send_mrf_intent_typed(kind, bucket, object, version_uuid, Some(scope));
+        debug!(
+            event = EVENT_SET_DISK_READ,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_SET_DISK,
+            state = "mrf_ingress",
+            mrf_kind = ?kind,
+            bucket,
+            object,
+            version_id,
+            pool_index,
+            set_index,
+            result = ?ingress,
+            "Read-repair MRF ingress result"
+        );
+    }
+
    let Some(dedup_key) = reserve_read_repair_heal(bucket, object, version_id, pool_index, set_index).await else {
        record_read_repair_dedup("duplicate");
        debug!(
@@ -1233,15 +1251,6 @@ pub(in crate::set_disk) async fn submit_read_repair_heal_with_submitter(
        return;
    };

-    // Reservation won: this sighting owns the repair records for the object,
-    // including the durable journal intent when the caller asked for one.
-    if let Some((kind, version_uuid)) = mrf_intent
-        && let (Ok(pool_index), Ok(set_index)) = (u32::try_from(pool_index), u32::try_from(set_index))
-    {
-        let scope = rustfs_common::mrf_channel::MrfScope { pool_index, set_index };
-        let _ = rustfs_common::mrf_channel::try_send_mrf_intent_typed(kind, bucket, object, version_uuid, Some(scope));
-    }
-
    let mut request = rustfs_heal_contracts::heal_channel::create_heal_request_with_options(
        bucket.to_string(),
        Some(object.to_string()),
@@ -12961,7 +12970,7 @@ mod tests {

    #[tokio::test]
    #[serial_test::serial]
-    async fn mrf_intent_is_filed_once_per_read_repair_reservation() {
+    async fn mrf_ingress_is_coalesced_independently_of_read_repair_reservations() {
        // Serial: owns the process-global MRF channel for this test binary
        // (same key as the other channel-owning tests above).
        let bucket = format!("mrf-intent-bucket-{}", Uuid::new_v4());
@@ -12982,8 +12991,13 @@ mod tests {
            }
        }

-        // First sighting wins the reservation: the journal intent is filed
-        // synchronously before the admission task is spawned.
+        // An earlier metadata/missing-shard admission has no payload MRF intent.
+        let mut metadata_submission = intent_submission(&bucket, &object);
+        metadata_submission.mrf_intent = None;
+        submit_read_repair_heal_with_submitter(metadata_submission, accepted_read_repair_submitter).await;
+        assert!(receiver.try_recv().is_err());
+
+        // Newly observed corruption must reach MRF despite that reservation.
        submit_read_repair_heal_with_submitter(intent_submission(&bucket, &object), accepted_read_repair_submitter).await;
        let first = receiver.try_recv().expect("first sighting must file exactly one MRF intent");
        assert_eq!(*first.bucket, bucket);
@@ -12993,6 +13007,20 @@ mod tests {
        // second journal record.
        submit_read_repair_heal_with_submitter(intent_submission(&bucket, &object), accepted_read_repair_submitter).await;
        assert!(receiver.try_recv().is_err(), "duplicate sighting must not file another MRF intent");
+
+        let retry_object = format!("retry-object-{}", Uuid::new_v4());
+        rustfs_common::mrf_channel::set_mrf_delivery_enabled(false);
+        submit_read_repair_heal_with_submitter(intent_submission(&bucket, &retry_object), accepted_read_repair_submitter).await;
+        assert!(receiver.try_recv().is_err());
+        rustfs_common::mrf_channel::set_mrf_delivery_enabled(true);
+        submit_read_repair_heal_with_submitter(intent_submission(&bucket, &retry_object), accepted_read_repair_submitter).await;
+        let retried = receiver
+            .try_recv()
+            .expect("a heal reservation cannot suppress a previously rejected MRF intent");
+        assert_eq!(*retried.object, retry_object);
+        rustfs_common::mrf_channel::release_mrf_intent(&first);
+        rustfs_common::mrf_channel::release_mrf_intent(&retried);
+        rustfs_common::mrf_channel::set_mrf_delivery_enabled(false);
    }

    #[tokio::test]
@@ -3877,6 +3877,9 @@ pub struct SetDisks {
    pub format: FormatV3,
    #[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
    disk_health_cache: Arc<RwLock<Vec<Option<DiskHealthEntry>>>>,
+    /// Coalesce bounded GET probes of missing remote slots. Shared by set clones;
+    /// the timestamp also limits repeated probes while a peer remains absent.
+    read_reconnect: Arc<tokio::sync::Mutex<Option<tokio::time::Instant>>>,
    get_object_metadata_cache: moka::future::Cache<GetObjectMetadataCacheKey, Arc<GetObjectMetadataCacheEntry>>,
    get_object_metadata_cache_hash_builder: std::collections::hash_map::RandomState,
    get_object_metadata_cache_generations: Arc<[AtomicU64]>,
@@ -4743,6 +4746,7 @@ impl SetDisks {
            format,
            set_endpoints,
            disk_health_cache: Arc::new(RwLock::new(Vec::new())),
+            read_reconnect: Arc::new(tokio::sync::Mutex::new(None)),
            get_object_metadata_cache: moka::future::Cache::builder()
                .max_capacity(get_object_metadata_cache_max_entries() as u64)
                .time_to_live(GET_OBJECT_METADATA_CACHE_TTL)
@@ -6473,6 +6477,28 @@ async fn verify_inline_part_bitrot(meta: &FileInfo) -> disk::error::Result<()> {
        .map_err(|_| DiskError::FileCorrupt)
 }

+fn validate_deep_scan_results(results: &[usize], expected_parts: usize) -> disk::error::Result<()> {
+    if results.len() != expected_parts {
+        return Err(DiskError::other(format!(
+            "incomplete deep scan result: expected {expected_parts} parts, received {}",
+            results.len()
+        )));
+    }
+    for (part, status) in results.iter().enumerate() {
+        match *status {
+            CHECK_PART_SUCCESS | CHECK_PART_FILE_NOT_FOUND | CHECK_PART_FILE_CORRUPT => {}
+            CHECK_PART_DISK_NOT_FOUND => return Err(DiskError::DiskNotFound),
+            crate::disk::CHECK_PART_VOLUME_NOT_FOUND => return Err(DiskError::VolumeNotFound),
+            _ => {
+                return Err(DiskError::other(format!(
+                    "incomplete deep scan result: part {part} has unverified status {status}"
+                )));
+            }
+        }
+    }
+    Ok(())
+}
+
 /// disks_with_all_partsv2 is a corrected version based on Go implementation.
 /// It sets partsMetadata and onlineDisks when xl.meta is inexistant/corrupted or outdated.
 /// It also checks if the status of each part (corrupted, missing, ok) in each drive.
@@ -6693,9 +6719,13 @@ async fn disks_with_all_parts(
            // it needs healing too.
            match disk.verify_file(bucket, object, meta).await {
                Ok(v) => {
+                    validate_deep_scan_results(&v.results, latest_meta.parts.len())?;
                    verify_resp = v;
                }
                Err(err) => {
+                    if !matches!(err, DiskError::FileNotFound | DiskError::FileVersionNotFound | DiskError::FileCorrupt) {
+                        return Err(err);
+                    }
                    debug!(
                        event = EVENT_SET_DISK_HEAL,
                        component = LOG_COMPONENT_ECSTORE,
@@ -11109,6 +11139,50 @@ mod tests {
        assert_eq!(conv_part_err_to_int(&Some(other_err)), CHECK_PART_UNKNOWN); // Other errors should return UNKNOWN, not SUCCESS
    }

+    #[test]
+    fn deep_scan_results_accept_only_complete_verified_or_repairable_parts() {
+        for (results, expected_parts) in [
+            (vec![], 0),
+            (vec![CHECK_PART_SUCCESS], 1),
+            (vec![CHECK_PART_FILE_NOT_FOUND], 1),
+            (vec![CHECK_PART_FILE_CORRUPT], 1),
+            (vec![CHECK_PART_SUCCESS, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_FILE_CORRUPT], 3),
+        ] {
+            validate_deep_scan_results(&results, expected_parts)
+                .expect("complete results must allow healthy or repairable parts");
+        }
+    }
+
+    #[test]
+    fn deep_scan_results_reject_unknown_and_incomplete_observations() {
+        for (results, expected_parts) in [
+            (vec![CHECK_PART_UNKNOWN], 1),
+            (vec![usize::MAX], 1),
+            (vec![CHECK_PART_SUCCESS, CHECK_PART_UNKNOWN], 2),
+            (vec![], 1),
+            (vec![CHECK_PART_SUCCESS], 2),
+            (vec![CHECK_PART_SUCCESS, CHECK_PART_SUCCESS], 1),
+            (vec![CHECK_PART_SUCCESS], 0),
+        ] {
+            let error =
+                validate_deep_scan_results(&results, expected_parts).expect_err("unverified parts must not become healthy");
+            assert!(matches!(error, DiskError::Io(_)), "an incomplete observation is not proven corruption");
+            assert!(error.to_string().contains("incomplete deep scan result"));
+        }
+    }
+
+    #[test]
+    fn deep_scan_results_preserve_disk_and_volume_failures() {
+        for (status, expected_error) in [
+            (CHECK_PART_DISK_NOT_FOUND, DiskError::DiskNotFound),
+            (CHECK_PART_VOLUME_NOT_FOUND, DiskError::VolumeNotFound),
+        ] {
+            let error = validate_deep_scan_results(&[CHECK_PART_SUCCESS, status], 2)
+                .expect_err("an unavailable part cannot certify a healthy disk");
+            assert_eq!(error, expected_error);
+        }
+    }
+
    #[test]
    fn test_has_part_err() {
        // Test checking for part errors
@@ -13628,6 +13702,7 @@ mod tests {
                true,
                false,
                false,
+                false,
                GET_OBJECT_PATH_LEGACY_DUPLEX,
                GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART,
                metrics_size_bucket,
@@ -13741,6 +13816,7 @@ mod tests {
                true,
                false,
                false,
+                false,
                GET_OBJECT_PATH_LEGACY_DUPLEX,
                GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART,
                metrics_size_bucket,
@@ -854,7 +854,7 @@ impl SetDisks {
        );

        let disks = self.get_disks_internal().await;
-        let bucket_heal_scope = crate::store::bucket_heal_scope(bucket);
+        let bucket_heal_scope = crate::store::bucket_heal_scope_for_object(bucket, object);
        if let Some(scope) = &bucket_heal_scope {
            scope.check()?;
        }
@@ -4378,63 +4378,243 @@ mod heal_result_report_tests {
        );
    }

-    #[tokio::test]
-    async fn deep_heal_rebuilds_h2_near_tail_truncated_part() {
-        let (temp_dirs, disks, set) = hermetic_set_disks_for_pool_with_default_parity_isolated(16, 0, 4).await;
-        let bucket = "deep-heal-h2-near-tail-truncation";
-        let object = "object.bin";
-        for disk in &disks {
-            disk.make_volume(bucket).await.expect("bucket volume should be created");
+    #[derive(Debug)]
+    struct FailedIntegrityReadTransport {
+        error: DiskError,
+        reads: std::sync::atomic::AtomicUsize,
+    }
+
+    #[async_trait::async_trait]
+    impl crate::cluster::rpc::internode_data_transport::InternodeDataTransport for FailedIntegrityReadTransport {
+        async fn open_read(
+            &self,
+            _request: crate::cluster::rpc::internode_data_transport::ReadStreamRequest,
+        ) -> crate::disk::error::Result<crate::disk::FileReader> {
+            self.reads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+            let error = match &self.error {
+                DiskError::FileCorrupt => DiskError::from(rustfs_rio::new_test_remote_file_corrupt_http_io_error()),
+                DiskError::FileNotFound => DiskError::from(rustfs_rio::new_test_remote_file_not_found_http_io_error()),
+                DiskError::VolumeNotFound => DiskError::from(rustfs_rio::new_test_remote_volume_not_found_http_io_error()),
+                error => error.clone(),
+            };
+            Err(error)
        }

-        let expected_payload = vec![0x6d; 5 * 1024 * 1024 + 123];
+        async fn open_write(
+            &self,
+            _request: crate::cluster::rpc::internode_data_transport::WriteStreamRequest,
+        ) -> crate::disk::error::Result<crate::disk::FileWriter> {
+            panic!("an integrity scan must not write remote data");
+        }
+
+        async fn open_walk_dir(
+            &self,
+            _request: crate::cluster::rpc::internode_data_transport::WalkDirStreamRequest,
+        ) -> crate::disk::error::Result<crate::disk::FileReader> {
+            panic!("an object integrity scan must not walk remote directories");
+        }
+
+        fn name(&self) -> &'static str {
+            "failed-integrity-read"
+        }
+
+        fn capabilities(&self) -> crate::cluster::rpc::internode_data_transport::InternodeDataTransportCapabilities {
+            crate::cluster::rpc::internode_data_transport::InternodeDataTransportCapabilities::tcp_http()
+        }
+    }
+
+    async fn assert_remote_integrity_read_failure(expected_error: DiskError) {
+        use crate::disk::{CHECK_PART_FILE_CORRUPT, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_SUCCESS, CHECK_PART_UNKNOWN};
+        use crate::object_api::ShardIntegrityWriteMode;
+
+        let (dirs, disks, set) = hermetic_set_disks_for_pool_with_default_parity_isolated(4, 0, 2).await;
+        let bucket = "remote-integrity-read-failure";
+        let object = "object.bin";
+        for disk in &disks {
+            disk.make_volume(bucket).await.expect("create fixture bucket");
+        }
        set.put_object(
            bucket,
            object,
-            &mut PutObjReader::from_vec(expected_payload.clone()),
+            &mut PutObjReader::from_vec(vec![0x6d; 1024 * 1024 + 123]),
            &ObjectOptions {
+                shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Protected),
                no_lock: true,
                ..Default::default()
            },
        )
        .await
-        .expect("source object should be written before shard truncation");
-        let source = disks[2]
-            .read_version("", bucket, object, "", &ReadOptions::default())
-            .await
-            .expect("source metadata should be readable");
-        let data_dir = source.data_dir.expect("non-inline source should have a data directory");
-        let truncated_part = temp_dirs[1]
-            .path()
-            .join(bucket)
-            .join(object)
-            .join(data_dir.to_string())
-            .join("part.1");
-        let original_len = tokio::fs::metadata(&truncated_part)
-            .await
-            .expect("target shard should exist")
-            .len();
-        assert!(original_len > 1, "test shard must be large enough to truncate");
-        let file = tokio::fs::OpenOptions::new()
-            .write(true)
-            .open(&truncated_part)
-            .await
-            .expect("target shard should be writable");
-        file.set_len(original_len - 1)
-            .await
-            .expect("target shard should be truncated");
+        .expect("write protected external fixture");

-        let mut reader = set
-            .get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
-            .await
-            .expect("GET should remain readable after a one-byte shard truncation");
-        let mut read_back = Vec::new();
-        tokio::io::copy(&mut reader, &mut read_back)
-            .await
-            .expect("GET should reconstruct the truncated shard through EC");
-        assert_eq!(read_back, expected_payload, "EC GET must preserve the object bytes");
+        let mut files = Vec::new();
+        let mut snapshots = Vec::new();
+        for (dir, disk) in dirs.iter().zip(&disks) {
+            let file = disk
+                .read_version("", bucket, object, "", &ReadOptions::default())
+                .await
+                .expect("read fixture metadata");
+            assert!(file.data.is_none(), "the remote test must open an external shard");
+            let object_dir = dir.path().join(bucket).join(object);
+            let part_path = object_dir
+                .join(file.data_dir.expect("external shard directory").to_string())
+                .join("part.1");
+            for path in [part_path, object_dir.join("xl.meta")] {
+                let bytes = tokio::fs::read(&path).await.expect("snapshot fixture before scan");
+                snapshots.push((path, bytes));
+            }
+            files.push(file);
+        }
+        let transport = Arc::new(FailedIntegrityReadTransport {
+            error: expected_error.clone(),
+            reads: std::sync::atomic::AtomicUsize::new(0),
+        });
+        let endpoint = Endpoint {
+            url: url::Url::parse("http://remote-integrity.invalid:9000/data/disk0").expect("fixture endpoint"),
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+        let remote = crate::cluster::rpc::RemoteDisk::new(
+            &endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+            transport.clone(),
+        )
+        .await
+        .expect("construct remote disk with failing data transport");
+        let mut verification_disks: Vec<_> = disks.iter().cloned().map(Some).collect();
+        verification_disks[0] = Some(Arc::new(crate::disk::Disk::Remote(Box::new(remote))));
+        let mut statuses = [(0, vec![CHECK_PART_UNKNOWN; disks.len()])].into_iter().collect();
+        let result = crate::io_support::shard_integrity::verify_deep_parts(
+            &files,
+            &verification_disks,
+            &files[0],
+            bucket,
+            object,
+            &mut statuses,
+        )
+        .await;
+        assert!(transport.reads.load(std::sync::atomic::Ordering::Relaxed) > 0);
+        if matches!(expected_error, DiskError::FileCorrupt | DiskError::FileNotFound) {
+            result.expect("typed remote payload damage must identify a repairable shard");
+            let expected_status = if expected_error == DiskError::FileCorrupt {
+                CHECK_PART_FILE_CORRUPT
+            } else {
+                CHECK_PART_FILE_NOT_FOUND
+            };
+            assert_eq!(statuses[&0][0], expected_status);
+            assert!(statuses[&0][1..].iter().all(|status| *status == CHECK_PART_SUCCESS));
+            let (heal, metadata, reason) = super::should_heal_object_on_disk(&None, &[statuses[&0][0]], &files[0], &files[0]);
+            assert!(heal, "remote payload damage must schedule repair: {expected_error:?}");
+            assert!(!metadata);
+            let expected_reason = if expected_error == DiskError::FileCorrupt {
+                DiskError::FileCorrupt
+            } else {
+                DiskError::PartMissingOrCorrupt
+            };
+            assert_eq!(reason, Some(expected_reason));
+        } else {
+            let error = result.expect_err("unavailable payload storage must abort health verification");
+            if expected_error.is_internode_http_status(500) {
+                assert!(error.is_internode_http_status(500), "preserve the original transport error: {error:?}");
+            } else {
+                assert_eq!(error, expected_error, "preserve the original storage failure");
+            }
+            assert_eq!(statuses[&0][0], CHECK_PART_UNKNOWN, "failed verification cannot certify the shard");
+        }
+        for (path, before) in snapshots {
+            assert_eq!(tokio::fs::read(path).await.expect("read fixture after scan"), before);
+        }
+    }

-        let (result, error) = set
+    #[tokio::test]
+    async fn deep_scan_remote_payload_damage_schedules_repair() {
+        for error in [DiskError::FileCorrupt, DiskError::FileNotFound] {
+            assert_remote_integrity_read_failure(error).await;
+        }
+    }
+
+    #[tokio::test]
+    async fn deep_scan_remote_storage_failure_cannot_certify_healthy() {
+        for error in [
+            DiskError::from(rustfs_rio::new_test_internode_http_io_error(
+                rustfs_rio::InternodeHttpErrorKind::HttpStatus(reqwest::StatusCode::INTERNAL_SERVER_ERROR),
+            )),
+            DiskError::DiskNotFound,
+            DiskError::VolumeNotFound,
+            DiskError::FileAccessDenied,
+        ] {
+            assert_remote_integrity_read_failure(error).await;
+        }
+    }
+
+    #[cfg(unix)]
+    #[tokio::test]
+    async fn deep_heal_legacy_verification_error_cannot_certify_healthy() {
+        use crate::object_api::ShardIntegrityWriteMode;
+
+        let (dirs, disks, set) = hermetic_set_disks_isolated(4).await;
+        let bucket = "legacy-deep-scan-verification-error";
+        let object = "object.bin";
+        for disk in &disks {
+            disk.make_volume(bucket).await.expect("create legacy fixture bucket");
+        }
+        set.put_object(
+            bucket,
+            object,
+            &mut PutObjReader::from_vec(vec![0x6d; 1024 * 1024 + 123]),
+            &ObjectOptions {
+                shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy),
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("write external legacy fixture");
+
+        let mut files = Vec::new();
+        let mut part_paths = Vec::new();
+        let mut preserved = Vec::new();
+        for (index, (dir, disk)) in dirs.iter().zip(&disks).enumerate() {
+            let file = disk
+                .read_version("", bucket, object, "", &ReadOptions::default())
+                .await
+                .expect("read legacy fixture metadata");
+            assert!(file.data.is_none(), "legacy verification must open the physical shard");
+            assert!(file.parts.iter().all(|part| part.integrity.is_none()));
+            let object_dir = dir.path().join(bucket).join(object);
+            let part = object_dir
+                .join(file.data_dir.expect("external legacy shard directory").to_string())
+                .join("part.1");
+            let metadata = object_dir.join("xl.meta");
+            let bytes = tokio::fs::read(&metadata).await.expect("snapshot legacy metadata");
+            preserved.push((metadata, bytes));
+            if index != 0 {
+                let bytes = tokio::fs::read(&part).await.expect("snapshot healthy legacy shard");
+                preserved.push((part.clone(), bytes));
+            }
+            part_paths.push(part);
+            files.push(file);
+        }
+        let broken_part = &part_paths[0];
+        tokio::fs::remove_file(broken_part)
+            .await
+            .expect("remove fixture shard before symlink fault");
+        std::os::unix::fs::symlink("part.1", broken_part).expect("create self-referential shard symlink");
+        let open_error = tokio::fs::File::open(broken_part)
+            .await
+            .expect_err("self-referential symlink must fail to open");
+        assert_eq!(open_error.raw_os_error(), Some(libc::ELOOP));
+
+        let verification_error = disks[0]
+            .verify_file(bucket, object, &files[0])
+            .await
+            .expect_err("legacy disk verifier must reject the symlink path");
+        assert_eq!(verification_error, DiskError::InvalidPath);
+        let error = set
            .heal_object(
                bucket,
                object,
@@ -4446,19 +4626,229 @@ mod heal_result_report_tests {
                },
            )
            .await
-            .expect("deep heal should finish after a one-byte H2 tail truncation");
-
-        assert!(error.is_none(), "deep heal should recover the H2 truncated shard: {error:?}");
-        assert_eq!(result.drives_healed(), Some(1));
-        assert_eq!(result.before.drives[1].state, DriveState::Corrupt.to_string());
+            .expect_err("an unverified legacy part must fail the complete heal operation");
+        assert_eq!(error, DiskError::InvalidPath, "complete heal must preserve the verifier's path error");
        assert_eq!(
-            tokio::fs::metadata(&truncated_part)
+            tokio::fs::read_link(broken_part)
                .await
-                .expect("repaired H2 shard should exist")
-                .len(),
-            original_len,
-            "deep heal must restore the complete H2 shard length"
+                .expect("failed heal must preserve the symlink fault"),
+            std::path::Path::new("part.1")
        );
+        for (path, before) in preserved {
+            assert_eq!(tokio::fs::read(path).await.expect("read unchanged legacy fixture"), before);
+        }
+    }
+
+    async fn assert_h2_truncated_shards(
+        mode: crate::object_api::ShardIntegrityWriteMode,
+        coding_indexes: &[usize],
+        retained_len: fn(usize, usize) -> usize,
+    ) {
+        use crate::object_api::ShardIntegrityWriteMode;
+
+        let (temp_dirs, disks, set) = hermetic_set_disks_for_pool_with_default_parity_isolated(16, 0, 4).await;
+        let bucket = "deep-heal-h2-truncation";
+        let object = "object.bin";
+        for disk in &disks {
+            disk.make_volume(bucket).await.expect("create bucket volume");
+        }
+        let expected_payload: Vec<_> = (0..5 * 1024 * 1024 + 123)
+            .map(|index| u8::try_from((index / 4096 + index) % 251).expect("payload byte fits"))
+            .collect();
+        set.put_object(
+            bucket,
+            object,
+            &mut PutObjReader::from_vec(expected_payload.clone()),
+            &ObjectOptions {
+                shard_integrity_write_mode: Some(mode),
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .expect("commit all source shards");
+
+        let mut files = Vec::new();
+        let mut paths = Vec::new();
+        let mut original_shards = Vec::new();
+        let mut original_metadata = Vec::new();
+        let mut damaged = Vec::new();
+        for (slot, disk) in disks.iter().enumerate() {
+            let source = disk
+                .read_version("", bucket, object, "", &ReadOptions::default())
+                .await
+                .expect("read source metadata");
+            assert_eq!((source.erasure.data_blocks, source.erasure.parity_blocks), (12, 4));
+            assert_eq!(source.parts[0].integrity.is_some(), mode == ShardIntegrityWriteMode::Protected);
+            let directory = temp_dirs[slot].path().join(bucket).join(object);
+            let path = directory
+                .join(source.data_dir.expect("external shard data directory").to_string())
+                .join("part.1");
+            let original = tokio::fs::read(&path).await.expect("read original shard");
+            original_metadata.push(tokio::fs::read(directory.join("xl.meta")).await.expect("read xl.meta"));
+            if coding_indexes.contains(&source.erasure.index) {
+                let erasure = crate::erasure::coding::Erasure::try_new_with_options(
+                    12,
+                    4,
+                    source.erasure.block_size,
+                    source.uses_legacy_checksum,
+                )
+                .expect("valid erasure layout");
+                let stripe_size = erasure.shard_size() + source.erasure.get_checksum_info(1).algorithm.size();
+                let keep = retained_len(original.len(), stripe_size);
+                assert!(keep < original.len(), "fixture must truncate the shard");
+                tokio::fs::OpenOptions::new()
+                    .write(true)
+                    .open(&path)
+                    .await
+                    .expect("open target shard")
+                    .set_len(u64::try_from(keep).expect("shard length fits"))
+                    .await
+                    .expect("truncate target shard");
+                damaged.push((slot, keep));
+            }
+            paths.push(path);
+            original_shards.push(original);
+            files.push(source);
+        }
+        assert_eq!(damaged.len(), coding_indexes.len());
+        if mode == ShardIntegrityWriteMode::Protected {
+            let mut statuses = [(0, vec![crate::disk::CHECK_PART_UNKNOWN; disks.len()])]
+                .into_iter()
+                .collect();
+            let online_disks: Vec<_> = disks.iter().cloned().map(Some).collect();
+            crate::io_support::shard_integrity::verify_deep_parts(
+                &files,
+                &online_disks,
+                &files[0],
+                bucket,
+                object,
+                &mut statuses,
+            )
+            .await
+            .expect("verify protected shards directly");
+            for (slot, status) in statuses[&0].iter().enumerate() {
+                let expected = if damaged.iter().any(|(index, _)| *index == slot) {
+                    crate::disk::CHECK_PART_FILE_CORRUPT
+                } else {
+                    crate::disk::CHECK_PART_SUCCESS
+                };
+                assert_eq!(*status, expected, "protected shard {slot}");
+            }
+        }
+
+        let opts = HealOpts {
+            no_lock: true,
+            scan_mode: HealScanMode::Deep,
+            ..Default::default()
+        };
+        let (dry_run, error) = set
+            .heal_object(bucket, object, "", &HealOpts { dry_run: true, ..opts })
+            .await
+            .expect("dry-run should report corruption");
+        assert!(error.is_none(), "dry-run error: {error:?}");
+        assert_eq!(dry_run.drives_healed(), Some(0));
+        for (slot, keep) in &damaged {
+            assert_eq!(dry_run.after.drives[*slot].state, DriveState::Corrupt.to_string());
+            assert_eq!(
+                tokio::fs::read(&paths[*slot]).await.expect("dry-run preserved shard"),
+                original_shards[*slot][..*keep]
+            );
+        }
+
+        let recoverable = damaged.len() <= 4;
+        if recoverable {
+            let mut reader = set
+                .get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
+                .await
+                .expect("degraded GET should remain readable");
+            let mut read_back = Vec::new();
+            tokio::io::copy(&mut reader, &mut read_back)
+                .await
+                .expect("read complete degraded GET");
+            assert_eq!(read_back, expected_payload);
+        }
+        let (result, error) = set
+            .heal_object(bucket, object, "", &opts)
+            .await
+            .expect("heal should report outcome");
+        if recoverable {
+            assert!(error.is_none(), "heal error: {error:?}");
+            assert_eq!(result.drives_healed(), Some(damaged.len()));
+            for (slot, _) in &damaged {
+                assert_eq!(result.before.drives[*slot].state, DriveState::Corrupt.to_string());
+            }
+        } else {
+            assert_eq!(error, Some(DiskError::ErasureReadQuorum));
+            assert_eq!(result.drives_healed(), Some(0));
+        }
+        for (slot, path) in paths.iter().enumerate() {
+            let expected = if !recoverable && let Some((_, keep)) = damaged.iter().find(|(index, _)| *index == slot) {
+                &original_shards[slot][..*keep]
+            } else {
+                original_shards[slot].as_slice()
+            };
+            assert_eq!(
+                tokio::fs::read(path).await.expect("read shard after heal"),
+                expected,
+                "physical shard {slot}"
+            );
+            if !recoverable {
+                let metadata_path = temp_dirs[slot].path().join(bucket).join(object).join("xl.meta");
+                assert_eq!(tokio::fs::read(metadata_path).await.expect("preserved metadata"), original_metadata[slot]);
+            }
+        }
+        if recoverable {
+            let mut reader = set
+                .get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
+                .await
+                .expect("GET after repair");
+            let mut read_back = Vec::new();
+            tokio::io::copy(&mut reader, &mut read_back)
+                .await
+                .expect("read repaired object");
+            assert_eq!(read_back, expected_payload);
+            let (healthy, error) = set.heal_object(bucket, object, "", &opts).await.expect("second deep scan");
+            assert!(error.is_none());
+            assert_eq!(healthy.drives_healed(), Some(0), "repair must converge");
+            assert_eq!(healthy.integrity_verified, mode == ShardIntegrityWriteMode::Protected);
+        }
+    }
+
+    #[tokio::test]
+    async fn deep_heal_rebuilds_h2_near_tail_truncated_part() {
+        use crate::object_api::ShardIntegrityWriteMode::{Legacy, Protected};
+        for mode in [Legacy, Protected] {
+            for coding_index in [1, 13] {
+                assert_h2_truncated_shards(mode, &[coding_index], |length, _| length - 1).await;
+            }
+        }
+    }
+
+    #[tokio::test]
+    async fn deep_heal_rebuilds_h2_half_truncated_part() {
+        use crate::object_api::ShardIntegrityWriteMode::{Legacy, Protected};
+        for mode in [Legacy, Protected] {
+            for coding_index in [1, 13] {
+                assert_h2_truncated_shards(mode, &[coding_index], |length, _| length / 2).await;
+            }
+        }
+    }
+
+    #[tokio::test]
+    async fn deep_heal_rebuilds_h2_stripe_boundary_truncated_parts_at_read_quorum() {
+        use crate::object_api::ShardIntegrityWriteMode::{Legacy, Protected};
+        for mode in [Legacy, Protected] {
+            assert_h2_truncated_shards(mode, &[1, 2, 13, 16], |_, stripe| stripe).await;
+        }
+    }
+
+    #[tokio::test]
+    async fn deep_heal_rejects_h2_truncated_parts_below_read_quorum() {
+        use crate::object_api::ShardIntegrityWriteMode::{Legacy, Protected};
+        for mode in [Legacy, Protected] {
+            assert_h2_truncated_shards(mode, &[1, 2, 3, 13, 16], |length, _| length / 2).await;
+        }
    }

    #[tokio::test]
@@ -341,14 +341,17 @@ impl SetDisks {
    pub async fn renew_disk(&self, ep: &Endpoint) {
        debug!("renew_disk: start {:?}", ep);

-        let previous_health = {
+        let previous_disk = {
            let disks = self.disks.read().await;
            disks
                .iter()
                .filter_map(|disk| disk.as_ref())
                .find(|disk| disk.endpoint() == *ep)
-                .and_then(|disk| disk.local_health_tracker_epoch_for_reconnect())
+                .cloned()
        };
+        let previous_health = previous_disk
+            .as_ref()
+            .and_then(|disk| disk.local_health_tracker_epoch_for_reconnect());

        let (new_disk, fm) = match Self::connect_endpoint(ep, previous_health).await {
            Ok(res) => res,
@@ -406,6 +409,25 @@ impl SetDisks {
        }

        let _ = new_disk.set_disk_id(Some(fm.erasure.this)).await;
+
+        if !new_disk.is_local() {
+            let mut disks = self.disks.write().await;
+            let slot = &mut disks[disk_idx];
+            let unchanged = match (slot.as_ref(), previous_disk.as_ref()) {
+                (None, None) => true,
+                (Some(current), Some(previous)) => Arc::ptr_eq(current, previous),
+                _ => false,
+            };
+            // A GET probe or another reconnect may have filled this slot while
+            // format I/O was pending. Do not replace its handle or start a
+            // second health monitor for an unpublished handle.
+            if unchanged {
+                new_disk.enable_health_check();
+                *slot = Some(new_disk);
+            }
+            return;
+        }
+
        new_disk.enable_health_check();

        if new_disk.is_local() {
@@ -2776,6 +2776,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
                    self.set_index,
                    self.pool_index,
                    opts.skip_verify_bitrot,
+                    opts.suppress_read_repair,
                    true,
                    true,
                    GET_OBJECT_PATH_LEGACY_DUPLEX,
@@ -2805,6 +2806,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
                        self.set_index,
                        self.pool_index,
                        opts.skip_verify_bitrot,
+                        opts.suppress_read_repair,
                        true,
                        false,
                        GET_OBJECT_PATH_LEGACY_DUPLEX,
@@ -2893,6 +2895,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
                self.set_index,
                self.pool_index,
                opts.skip_verify_bitrot,
+                opts.suppress_read_repair,
                true,
                false,
                GET_OBJECT_PATH_DIRECT_MEMORY,
@@ -2949,6 +2952,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
                self.set_index,
                self.pool_index,
                opts.skip_verify_bitrot,
+                opts.suppress_read_repair,
                object_class.as_str(),
                size_bucket,
                false,
@@ -2995,6 +2999,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
                    self.set_index,
                    self.pool_index,
                    opts.skip_verify_bitrot,
+                    opts.suppress_read_repair,
                    object_class.as_str(),
                    size_bucket,
                    codec_streaming_gate.prefer_data_blocks_first_reader_setup,
@@ -3063,6 +3068,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
        let set_index = self.set_index;
        let pool_index = self.pool_index;
        let skip_verify = opts.skip_verify_bitrot;
+        let suppress_read_repair = opts.suppress_read_repair;
        // The producer runs in a separate Tokio task, so carry the caller's
        // read policy across the task boundary explicitly. Tokio task-local
        // values are not inherited by spawned tasks.
@@ -3093,6 +3099,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
                        set_index,
                        pool_index,
                        skip_verify,
+                        suppress_read_repair,
                        false,
                        false,
                        GET_OBJECT_PATH_LEGACY_DUPLEX,
@@ -9493,6 +9500,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
        let set_index = self.set_index;
        let pool_index = self.pool_index;
        let skip_verify = opts.skip_verify_bitrot;
+        let suppress_read_repair = opts.suppress_read_repair;
        let metrics_size_bucket = rustfs_io_metrics::get_object_size_bucket(cloned_fi.size);
        let erasure_cache = Arc::clone(&self.erasure_cache);
        let producer = async move {
@@ -9510,6 +9518,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                set_index,
                pool_index,
                skip_verify,
+                suppress_read_repair,
                false,
                false,
                GET_OBJECT_PATH_LEGACY_DUPLEX,
@@ -18,6 +18,7 @@ use crate::diagnostics::get::{
 use crate::disk::error::Error;
 use crate::layout::disks_layout::MAX_ERASURE_SET_DRIVE_COUNT;
 use smallvec::SmallVec;
+use std::sync::atomic::{AtomicU8, Ordering};

 /// Generic codec callers may exceed the production set limit; `SmallVec` then
 /// spills without changing slot semantics.
@@ -25,6 +26,35 @@ pub(crate) const INLINE_SHARD_SLOTS: usize = MAX_ERASURE_SET_DRIVE_COUNT;
 pub(crate) type ShardBuffers = SmallVec<[Option<Vec<u8>>; INLINE_SHARD_SLOTS]>;
 pub(crate) type ShardErrors = SmallVec<[Option<Error>; INLINE_SHARD_SLOTS]>;

+/// Repair evidence is independent of reconstruction quorum. Successful shards
+/// must not vote away a minority's observed corruption. Streaming decoders
+/// share this small accumulator with the reader that completes the response.
+#[derive(Debug, Default)]
+pub(crate) struct ShardReadRepair {
+    observed: AtomicU8,
+}
+
+impl ShardReadRepair {
+    pub(crate) fn observe(&self, errors: &[Option<Error>]) {
+        let flags = errors.iter().flatten().fold(0, |flags, error| {
+            flags
+                | match error {
+                    Error::FileCorrupt => 1,
+                    Error::FileNotFound => 2,
+                    _ => 0,
+                }
+        });
+        if flags != 0 {
+            self.observed.fetch_or(flags, Ordering::Relaxed);
+        }
+    }
+
+    pub(crate) fn needed(&self, allow_missing: bool) -> bool {
+        let mask = if allow_missing { 3 } else { 1 };
+        self.observed.load(Ordering::Relaxed) & mask != 0
+    }
+}
+
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub(crate) enum ShardReadCost {
    Local,
@@ -164,6 +194,19 @@ mod tests {
    use super::*;
    use std::mem::size_of;

+    #[test]
+    fn repair_evidence_survives_healthy_stripes_and_excludes_unproven_failures() {
+        let evidence = ShardReadRepair::default();
+        evidence.observe(&[None, Some(Error::Timeout), Some(Error::DiskNotFound)]);
+        assert!(!evidence.needed(true));
+        evidence.observe(&[None, None, Some(Error::FileNotFound)]);
+        assert!(!evidence.needed(false), "unattempted reader slots do not prove a missing shard");
+        assert!(evidence.needed(true));
+        evidence.observe(&[None, None, Some(Error::FileCorrupt)]);
+        evidence.observe(&[None, None, None]);
+        assert!(evidence.needed(false), "a later healthy stripe must not erase observed corruption");
+    }
+
    #[test]
    fn stripe_scratch_capacity_matches_the_production_set_limit() {
        type OversizedShardBuffers = SmallVec<[Option<Vec<u8>>; 32]>;
@@ -58,6 +58,26 @@ pub(crate) fn bucket_heal_scope(bucket: &str) -> Option<Arc<BucketHealScope>> {
        .flatten()
 }

+/// Resolve only the two canonical bucket records; other internal paths carry no bucket authority.
+pub(super) fn bucket_metadata_owner(object: &str) -> Option<&str> {
+    use crate::bucket::metadata::{BUCKET_INCARNATION_FILE, BUCKET_METADATA_FILE};
+    let (prefix, rest) = object.split_once('/')?;
+    let (bucket, file) = rest.split_once('/')?;
+    (prefix == disk::BUCKET_META_PREFIX
+        && matches!(file, BUCKET_METADATA_FILE | BUCKET_INCARNATION_FILE)
+        && check_valid_bucket_name_strict(bucket).is_ok())
+    .then_some(bucket)
+}
+
+pub(crate) fn bucket_heal_scope_for_object(bucket: &str, object: &str) -> Option<Arc<BucketHealScope>> {
+    let owner = if bucket == RUSTFS_META_BUCKET {
+        bucket_metadata_owner(object)?
+    } else {
+        bucket
+    };
+    bucket_heal_scope(owner)
+}
+
 /// Storage-owned proof for the exact version and every selected erasure location.
 /// This is an in-process result, never reconstructed from admin drive telemetry.
 #[derive(Debug)]
@@ -422,6 +442,97 @@ impl ECStore {
        Ok(Arc::ptr_eq(&target_set, &pool.get_disks_by_key(POOL_META_NAME)))
    }

+    /// Restore the bucket configuration and incarnation before retiring a replacement marker.
+    pub async fn heal_replacement_bucket_metadata(
+        self: &Arc<Self>,
+        bucket: &str,
+        opts: &HealOpts,
+        targets: &[String],
+    ) -> Result<()> {
+        use crate::bucket::metadata::{BUCKET_INCARNATION_FILE, BUCKET_METADATA_FILE};
+        use crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in;
+
+        let (Some(pool_index), Some(set_index)) = (opts.pool, opts.set) else {
+            return Err(Error::PreconditionFailed);
+        };
+        if targets.is_empty() || opts.dry_run || opts.no_lock || check_valid_bucket_name_strict(bucket).is_err() {
+            return Err(Error::PreconditionFailed);
+        }
+        let pool = self
+            .pools
+            .get(pool_index)
+            .ok_or_else(|| invalid_heal_pool_index(pool_index, self.pools.len()))?;
+        let target_set = pool.get_disks_for_heal_object(BUCKET_METADATA_FILE, opts)?;
+        if targets.iter().any(|target| {
+            target_set
+                .set_endpoints
+                .iter()
+                .filter(|endpoint| endpoint.to_string() == *target)
+                .count()
+                != 1
+        }) {
+            return Err(Error::PreconditionFailed);
+        }
+        let objects: Vec<_> = [BUCKET_METADATA_FILE, BUCKET_INCARNATION_FILE]
+            .into_iter()
+            .map(|file| format!("{}/{bucket}/{file}", disk::BUCKET_META_PREFIX))
+            .filter(|object| Arc::ptr_eq(&target_set, &pool.get_disks_by_key(object)))
+            .collect();
+        if objects.is_empty() {
+            return Ok(());
+        }
+
+        // Read the persisted identity without lazy migration: a replacement
+        // must never turn lost configuration into newly fabricated defaults.
+        let incarnation = self.bucket_incarnation_id_from_disk(bucket).await?;
+        let targets = targets.to_vec();
+        self.run_bucket_heal_at_incarnation(bucket, incarnation, opts, move |store, bucket, opts| async move {
+            // Match config writers: lifecycle, metadata transaction, object locks.
+            // Keep the transaction stable through target readback, including when
+            // the caller is cancelled and the storage owner finishes its write.
+            let transaction = acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, &bucket).await?;
+            for object in objects {
+                let scope = bucket_heal_scope(&bucket).ok_or(Error::PreconditionFailed)?;
+                scope.check()?;
+                if transaction.is_lock_lost() {
+                    return Err(Error::PreconditionFailed);
+                }
+                // Config objects occupy one pool, unlike pool.bin. Require a
+                // successful all-pool lookup before accepting another pool's ownership.
+                if !store.single_pool() {
+                    let (owner, _) = store
+                        .get_pool_info_for_delete_marker(RUSTFS_META_BUCKET, &object, &ObjectOptions::default())
+                        .await?;
+                    if owner.index != pool_index {
+                        continue;
+                    }
+                }
+                let metadata_opts = HealOpts { remove: false, ..opts };
+                let (result, error) = store.heal_object(RUSTFS_META_BUCKET, &object, "", &metadata_opts).await?;
+                if let Some(error) = error {
+                    return Err(error);
+                }
+                for target in &targets {
+                    let mut drives = result.after.drives.iter().filter(|drive| &drive.endpoint == target);
+                    if !drives.next().is_some_and(|drive| drive.state == "ok") || drives.next().is_some() {
+                        return Err(Error::PreconditionFailed);
+                    }
+                }
+                if !store
+                    .replacement_targets_have_version(RUSTFS_META_BUCKET, &object, "", pool_index, set_index, &targets)
+                    .await?
+                {
+                    return Err(Error::PreconditionFailed);
+                }
+            }
+            if transaction.is_lock_lost() {
+                return Err(Error::PreconditionFailed);
+            }
+            Ok(())
+        })
+        .await
+    }
+
    /// Return every live erasure set selected by an object-heal scope.
    pub async fn heal_erasure_set_scopes(&self, opts: &HealOpts) -> Result<Vec<(usize, usize)>> {
        let pools = self.get_pools_for_heal_object(opts)?;
@@ -1345,6 +1456,16 @@ mod tests {
                .await
                .is_err()
        );
+        for file in [".metadata.bin", ".bucket-incarnation"] {
+            let metadata = format!("buckets/{bucket}/{file}");
+            assert!(
+                store
+                    .rename_local_data_at_incarnation(&disk_ref, (&bucket, object), &fi, (RUSTFS_META_BUCKET, &metadata), old)
+                    .await
+                    .is_err(),
+                "stale remote heal must not replace the successor's {file}"
+            );
+        }
        assert!(store.heal_bucket_at_incarnation(&bucket, old, &opts).await.is_err());
        assert!(
            store
@@ -1743,6 +1864,194 @@ mod tests {
        (temp_dir, store, shutdown)
    }

+    #[test]
+    fn bucket_metadata_heal_owner_rejects_path_aliases_and_unrelated_records() {
+        for file in [".metadata.bin", ".bucket-incarnation"] {
+            assert_eq!(bucket_metadata_owner(&format!("buckets/example/{file}")), Some("example"));
+        }
+        for path in [
+            "/buckets/example/.metadata.bin",
+            "buckets/../.metadata.bin",
+            "buckets/example/../.metadata.bin",
+            "buckets/example/.metadata.bin/extra",
+            "buckets/example//.metadata.bin",
+            "buckets/example/.metadata.bin.old",
+            "config/iam/example/.metadata.bin",
+            "buckets/example/usage.json",
+        ] {
+            assert_eq!(bucket_metadata_owner(path), None, "unexpected bucket authority for {path}");
+        }
+    }
+
+    #[tokio::test]
+    async fn replacement_bucket_metadata_respects_pool_ownership_and_required_records() {
+        let (root, store, shutdown) = multi_pool_heal_store().await;
+        let bucket = "replacement-bucket-metadata";
+        for invalid_bucket in ["MixedCase", "../example", RUSTFS_META_BUCKET] {
+            assert!(matches!(
+                store
+                    .heal_replacement_bucket_metadata(
+                        invalid_bucket,
+                        &HealOpts {
+                            pool: Some(0),
+                            set: Some(0),
+                            ..Default::default()
+                        },
+                        &[root.path().join("pool0-disk0").to_string_lossy().into_owned()],
+                    )
+                    .await,
+                Err(Error::PreconditionFailed)
+            ));
+        }
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create bucket and its persisted records");
+        let mut owned = 0;
+        for pool in 0..2 {
+            let target = root.path().join(format!("pool{pool}-disk0"));
+            let records: Vec<_> = [".metadata.bin", ".bucket-incarnation"]
+                .into_iter()
+                .map(|file| {
+                    let path = target.join(RUSTFS_META_BUCKET).join("buckets").join(bucket).join(file);
+                    let exists = path.join("xl.meta").exists();
+                    if exists {
+                        std::fs::remove_dir_all(&path).expect("remove replacement metadata shard");
+                        owned += 1;
+                    }
+                    (path, exists)
+                })
+                .collect();
+            store
+                .heal_replacement_bucket_metadata(
+                    bucket,
+                    &HealOpts {
+                        pool: Some(pool),
+                        set: Some(0),
+                        recreate: true,
+                        ..Default::default()
+                    },
+                    &[target.to_string_lossy().into_owned()],
+                )
+                .await
+                .expect("repair owned records and accept authoritative ownership in the other pool");
+            for (path, existed) in records {
+                assert_eq!(path.join("xl.meta").exists(), existed, "wrong placement for {}", path.display());
+            }
+        }
+        assert_eq!(owned, 2, "each required record must have exactly one owning pool");
+        let object = format!("buckets/{bucket}/.metadata.bin");
+        let owner = store
+            .get_pool_idx_existing_with_opts(RUSTFS_META_BUCKET, &object, &ObjectOptions::default())
+            .await
+            .expect("persisted metadata owner");
+        let disk = store.pools[owner].disk_set[0].disks.read().await[0]
+            .clone()
+            .expect("metadata target disk");
+        let fi = disk
+            .read_version(
+                "",
+                RUSTFS_META_BUCKET,
+                &object,
+                "",
+                &disk::ReadOptions {
+                    read_data: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("inline metadata to repair through target-side admission");
+        assert!(fi.data.is_some(), "fixture must contain an inline metadata shard");
+        let target_path = root.path().join(format!("pool{owner}-disk0"));
+        std::fs::remove_dir_all(target_path.join(RUSTFS_META_BUCKET).join(&object)).expect("remove remote target shard");
+        let incarnation = store
+            .bucket_incarnation_id_from_disk(bucket)
+            .await
+            .expect("current incarnation");
+        store
+            .rename_local_data_at_incarnation(
+                &disk.endpoint().to_string(),
+                (disk::RUSTFS_META_TMP_BUCKET, "metadata-heal-control"),
+                &fi,
+                (RUSTFS_META_BUCKET, &object),
+                incarnation,
+            )
+            .await
+            .expect("the target-side RPC admission must accept current bucket metadata repair");
+        assert!(target_path.join(RUSTFS_META_BUCKET).join(&object).join("xl.meta").exists());
+        let (configuration, _) = read_config_no_lock_preserve_empty_with_metadata(store.clone(), &object)
+            .await
+            .expect("read the current bucket configuration");
+        save_config(store.clone(), &object, configuration)
+            .await
+            .expect("publish a newer config revision in the same bucket incarnation");
+        let committed =
+            std::fs::read(target_path.join(RUSTFS_META_BUCKET).join(&object).join("xl.meta")).expect("new target configuration");
+        assert!(
+            store
+                .rename_local_data_at_incarnation(
+                    &disk.endpoint().to_string(),
+                    (disk::RUSTFS_META_TMP_BUCKET, "delayed-metadata-heal"),
+                    &fi,
+                    (RUSTFS_META_BUCKET, &object),
+                    incarnation,
+                )
+                .await
+                .is_err(),
+            "a delayed repair must not overwrite a newer config in the same incarnation"
+        );
+        assert_eq!(
+            std::fs::read(target_path.join(RUSTFS_META_BUCKET).join(&object).join("xl.meta")).expect("retained configuration"),
+            committed,
+            "rejected repair must leave the newer configuration unchanged"
+        );
+        for pool in 0..2 {
+            for disk in 0..4 {
+                let path = root
+                    .path()
+                    .join(format!("pool{pool}-disk{disk}"))
+                    .join(RUSTFS_META_BUCKET)
+                    .join("buckets")
+                    .join(bucket)
+                    .join(".metadata.bin");
+                if path.exists() {
+                    std::fs::remove_dir_all(path).expect("remove every copy of required bucket configuration");
+                }
+            }
+        }
+        metadata_sys::remove_bucket_metadata_in(&store.ctx, bucket)
+            .await
+            .expect("remove the cached configuration so repair cannot rely on it");
+        assert!(
+            store
+                .heal_replacement_bucket_metadata(
+                    bucket,
+                    &HealOpts {
+                        pool: Some(0),
+                        set: Some(0),
+                        recreate: true,
+                        ..Default::default()
+                    },
+                    &[root.path().join("pool0-disk0").to_string_lossy().into_owned()],
+                )
+                .await
+                .is_err(),
+            "total metadata loss must not be accepted as ownership in another pool"
+        );
+        for pool in 0..2 {
+            assert!(
+                !root
+                    .path()
+                    .join(format!("pool{pool}-disk0"))
+                    .join(RUSTFS_META_BUCKET)
+                    .join(&object)
+                    .exists(),
+                "missing required configuration must not be recreated with defaults"
+            );
+        }
+        shutdown.cancel();
+    }
+
    #[tokio::test]
    #[serial_test::serial]
    async fn absence_proof_requires_every_selected_pool() {
@@ -426,8 +426,8 @@ mod bucket_fence;
 pub(crate) use bucket::await_bucket_namespace_operation;
 pub use bucket_fence::BucketIncarnationFenceGuard;
 mod heal;
-pub(crate) use heal::bucket_heal_scope;
 pub use heal::{HealObjectAbsenceProof, HealObjectStorageResult};
+pub(crate) use heal::{bucket_heal_scope, bucket_heal_scope_for_object};
 mod heal_walk;
 pub use heal_walk::HealWalkVersion;
 mod init;
@@ -9142,6 +9142,25 @@ mod tests {
            )
            .await
            .expect("create versioned bucket");
+        let mut history = Vec::new();
+        for value in 1..=3_u8 {
+            let payload = vec![value; 4097];
+            let version = store
+                .put_object(
+                    bucket,
+                    object,
+                    &mut PutObjReader::from_vec(payload.clone()),
+                    &ObjectOptions {
+                        versioned: true,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("write the historical data versions")
+                .version_id
+                .expect("historical version must have an explicit identity");
+            history.push((version, payload));
+        }
        let marker = store
            .delete_object(
                bucket,
@@ -9216,6 +9235,55 @@ mod tests {
                Err(crate::disk::error::DiskError::FileVersionNotFound | crate::disk::error::DiskError::FileNotFound)
            ));
        }
+        let replay = store
+            .heal_object_at_incarnation(
+                bucket,
+                object,
+                &marker.to_string(),
+                current,
+                &rustfs_heal_contracts::heal_channel::HealOpts {
+                    remove: true,
+                    scan_mode: rustfs_heal_contracts::heal_channel::HealScanMode::Deep,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("replay the exact removed marker identity");
+        assert!(replay.error.is_none(), "exact marker replay: {:?}", replay.error);
+        let absence = replay.absence.expect("exact replay must prove authoritative absence");
+        assert!(!absence.removed, "replay must not claim another physical repair");
+        assert_eq!(absence.bucket_incarnation_id, current);
+        assert_eq!(absence.version_id, marker.to_string());
+        let listed = store
+            .clone()
+            .inner_list_object_versions(bucket, "", None, None, None, 100)
+            .await
+            .expect("list surviving versions after marker cleanup");
+        assert_eq!(listed.objects.len(), history.len(), "an absent marker is not enumerable in a new walk");
+        assert!(listed.objects.iter().all(|info| !info.delete_marker));
+        for (version, expected) in history {
+            assert!(listed.objects.iter().any(|info| info.version_id == Some(version)));
+            let mut reader = store
+                .handle_get_object_reader(
+                    bucket,
+                    object,
+                    None,
+                    HeaderMap::new(),
+                    &ObjectOptions {
+                        version_id: Some(version.to_string()),
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("historical data must remain readable after cleanup and replay");
+            let mut actual = Vec::new();
+            reader
+                .stream
+                .read_to_end(&mut actual)
+                .await
+                .expect("read all historical bytes");
+            assert_eq!(actual, expected);
+        }
    }

    #[tokio::test]
@@ -161,14 +161,52 @@ impl ECStore {
                "incarnation-bound rename requires namespace locking and a non-nil identity",
            ));
        }
+        let bucket = if destination.0 == RUSTFS_META_BUCKET {
+            super::heal::bucket_metadata_owner(destination.1).ok_or(DiskError::FileAccessDenied)?
+        } else {
+            destination.0
+        };
        let guard = Arc::new(
-            self.acquire_bucket_incarnation_fence(destination.0, expected)
+            self.acquire_bucket_incarnation_fence(bucket, expected)
                .await
                .map_err(|error| DiskError::other(error.to_string()))?,
        );
        if guard.is_lock_lost() {
            return Err(DiskError::other("bucket heal incarnation fence was lost"));
        }
+        let transaction = if destination.0 == RUSTFS_META_BUCKET {
+            // The receiver must retain the config fence after an RPC timeout.
+            // A delayed request must also reject a newer config in the same
+            // bucket incarnation before publishing its older repair payload.
+            let transaction = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket)
+                .await
+                .map_err(|error| DiskError::other(error.to_string()))?;
+            let (current, _) = self
+                .get_pool_info_for_delete_marker(
+                    destination.0,
+                    destination.1,
+                    &ObjectOptions {
+                        no_lock: true,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .map_err(|error| DiskError::other(error.to_string()))?;
+            let current = current.object_info;
+            if transaction.is_lock_lost()
+                || guard.is_lock_lost()
+                || fi.mod_time.is_none()
+                || current.mod_time != fi.mod_time
+                || current.data_dir != fi.data_dir
+                || current.size != fi.size
+                || current.etag != fi.get_etag()
+            {
+                return Err(DiskError::FileAccessDenied);
+            }
+            Some(transaction)
+        } else {
+            None
+        };
        rename_local_data_with_ctx(
            &self.ctx,
            disk_ref,
@@ -176,7 +214,7 @@ impl ECStore {
            fi,
            destination,
            RenameDataGuards {
-                external_guard: Some(guard),
+                external_guard: Some(Arc::new((guard, transaction))),
                ..Default::default()
            },
        )
@@ -907,6 +907,21 @@ impl ErasureSetHealer {
        }

        if failed_objects == 0 && skipped_objects == 0 && failed_buckets == 0 {
+            let targets = if self.pool_metadata_target_endpoints.is_empty() {
+                self.target_endpoints.as_ref()
+            } else {
+                self.pool_metadata_target_endpoints.as_ref()
+            };
+            if self.replacement_task_id.is_some() || (!self.heal_opts.dry_run && self.heal_opts.recreate && !targets.is_empty()) {
+                self.verify_replacement_identity_fence("bucket metadata").await?;
+                // Recheck even resumed buckets: their user-object cursor does not
+                // prove that the replacement holds the internal bucket records.
+                for bucket in buckets {
+                    self.storage
+                        .heal_replacement_bucket_metadata(bucket, &self.heal_opts, targets)
+                        .await?;
+                }
+            }
            self.heal_replacement_pool_metadata(
                set_disk_id,
                &mut ErasureSetPassCounters {
@@ -2267,6 +2282,8 @@ mod resume_loop_tests {
        list_include_lifecycle_object_info: Mutex<Vec<bool>>,
        replacement_target_identity_sequences: Mutex<VecDeque<Vec<ReplacementTargetIdentity>>>,
        pool_metadata_placement: Mutex<Option<ReplacementCommitEvidence>>,
+        bucket_metadata_calls: Mutex<Vec<String>>,
+        bucket_metadata_failure: AtomicBool,
        fail_listing: AtomicBool,
        fail_listing_buckets: Mutex<HashSet<String>>,
    }
@@ -2462,6 +2479,13 @@ mod resume_loop_tests {
                None => Ok(true),
            }
        }
+        async fn heal_replacement_bucket_metadata(&self, bucket: &str, _opts: &HealOpts, _targets: &[String]) -> Result<()> {
+            self.bucket_metadata_calls.lock().unwrap().push(bucket.to_owned());
+            if self.bucket_metadata_failure.load(Ordering::SeqCst) {
+                return Err(Error::Storage(EcstoreError::PreconditionFailed));
+            }
+            Ok(())
+        }
        async fn replacement_targets_have_version(
            &self,
            _bucket: &str,
@@ -3058,6 +3082,18 @@ mod resume_loop_tests {
        )
        .with_replacement_targets(vec!["replacement-a".to_string()], Some(replacement_task_id.clone()));

+        env.storage.bucket_metadata_failure.store(true, Ordering::SeqCst);
+        assert!(healer.heal_erasure_set(&["b".to_string()], "pool_0_set_0").await.is_err());
+        let incomplete = ResumeManager::load_replacement_intent(env.healer.disk.clone(), &replacement_task_id)
+            .await
+            .expect("failed metadata repair must retain replacement intent")
+            .get_state()
+            .await;
+        assert!(!incomplete.completed);
+        assert_ne!(incomplete.replacement_phase, crate::heal::resume::ReplacementPhase::Verified);
+        assert!(env.storage.calls().is_empty(), "metadata failure must stop completion before pool.bin");
+        env.storage.bucket_metadata_failure.store(false, Ordering::SeqCst);
+
        let error = healer
            .heal_erasure_set(&["b".to_string()], "pool_0_set_0")
            .await
@@ -3073,6 +3109,11 @@ mod resume_loop_tests {
        assert_eq!(state.replacement_phase, crate::heal::resume::ReplacementPhase::Intent);
        assert_eq!(state.retry_count, 1);
        assert_eq!(env.storage.calls(), vec![(POOL_META_NAME.to_string(), None)]);
+        assert_eq!(
+            *env.storage.bucket_metadata_calls.lock().unwrap(),
+            vec!["b", "b"],
+            "resuming a completed user scan must retry bucket metadata"
+        );
    }

    #[tokio::test]
@@ -591,6 +591,11 @@ pub trait HealStorageAPI: Send + Sync {
        Err(Error::other("replacement pool metadata placement is unsupported"))
    }

+    /// Heal and physically verify bucket configuration on its replacement targets.
+    async fn heal_replacement_bucket_metadata(&self, _bucket: &str, _opts: &HealOpts, _targets: &[String]) -> Result<()> {
+        Err(Error::Storage(StorageError::PreconditionFailed))
+    }
+
    /// Read target-specific physical evidence for one replacement version.
    ///
    /// This is only used by automatic replacement healing after the normal
@@ -1595,6 +1600,13 @@ impl HealStorageAPI for ECStoreHealStorage {
            .map_err(Error::Storage)
    }

+    async fn heal_replacement_bucket_metadata(&self, bucket: &str, opts: &HealOpts, targets: &[String]) -> Result<()> {
+        self.ecstore
+            .heal_replacement_bucket_metadata(bucket, opts, targets)
+            .await
+            .map_err(Error::Storage)
+    }
+
    async fn replacement_targets_have_version(
        &self,
        bucket: &str,
@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-//! Regression for replacement healing requiring pool.bin on a non-owning set.
+//! Replacement healing must restore internal records only on their owning sets.
 #![recursion_limit = "256"]

 use http::HeaderMap;
@@ -101,6 +101,31 @@ async fn replacement_pool_metadata_follows_real_two_set_placement() {
            storage.replacement_pool_metadata_required(&opts).is_err(),
            "invalid scope must fail closed"
        );
+        assert!(
+            storage
+                .heal_replacement_bucket_metadata(bucket, &opts, &[paths[0].to_string_lossy().into_owned()])
+                .await
+                .is_err(),
+            "bucket metadata repair must reject an incomplete or invalid replacement scope"
+        );
+    }
+
+    for targets in [vec![], vec![paths[4].to_string_lossy().into_owned()]] {
+        assert!(
+            storage
+                .heal_replacement_bucket_metadata(
+                    bucket,
+                    &HealOpts {
+                        pool: Some(0),
+                        set: Some(0),
+                        ..Default::default()
+                    },
+                    &targets,
+                )
+                .await
+                .is_err(),
+            "empty or wrong-set targets must be rejected before repairing metadata"
+        );
    }

    let mut owning_set = None;
@@ -120,6 +145,17 @@ async fn replacement_pool_metadata_follows_real_two_set_placement() {
            assert!(owning_set.replace(set).is_none(), "only one set owns pool.bin");
            std::fs::remove_dir_all(&metadata_path).unwrap();
        }
+        let bucket_records: Vec<_> = [".metadata.bin", ".bucket-incarnation"]
+            .into_iter()
+            .map(|file| {
+                let path = target_path.join(RUSTFS_META_BUCKET).join("buckets").join(bucket).join(file);
+                let existed = path.join("xl.meta").exists();
+                if existed {
+                    std::fs::remove_dir_all(&path).expect("remove bucket metadata shard from replacement");
+                }
+                (path, existed)
+            })
+            .collect();
        // Select a user key using the real placement algorithm, independently
        // of the metadata-scope decision under test.
        let key = (0..1000)
@@ -191,6 +227,14 @@ async fn replacement_pool_metadata_follows_real_two_set_placement() {
            "completed repair must clear its healing marker"
        );
        assert_eq!(metadata_path.join("xl.meta").exists(), owns_metadata);
+        for (path, owned) in bucket_records {
+            assert_eq!(
+                path.join("xl.meta").exists(),
+                owned,
+                "replacement must restore each owned bucket record without creating a wrong-set copy: {}",
+                path.display()
+            );
+        }
        for (relative, bytes) in original_parts {
            assert_eq!(std::fs::read(object_path.join(relative)).unwrap(), bytes, "reconstructed shard differs");
        }
@@ -241,7 +241,9 @@ impl RuleValidate for LifecycleRule {
            return Err(std::io::Error::other(ERR_LIFECYCLE_INVALID_DEL_MARKER_EXPIRATION_DAYS));
        }
        // Rule must have at least one action
-        let has_expiration = self.expiration.is_some();
+        let has_expiration = self.expiration.as_ref().is_some_and(|expiration| {
+            expiration.days.is_some() || expiration.date.is_some() || expiration.expired_object_delete_marker.is_some()
+        });
        let has_transition = self.transitions.as_ref().is_some_and(|t| !t.is_empty());
        // `NewerNoncurrentVersions` on its own is a MinIO extension, not an AWS
        // form: it keeps the newest N noncurrent versions and expires the rest
@@ -4455,6 +4457,19 @@ mod tests {
            .expect_err("a negative retention count on a transition must be rejected");
    }

+    #[tokio::test]
+    async fn validate_rejects_empty_expiration_action() {
+        let mut rule = rule_with_filter(LifecycleRuleFilter::default());
+        rule.expiration = Some(LifecycleExpiration::default());
+
+        let err = config_with_rules(vec![rule])
+            .validate(&ObjectLockConfiguration::default())
+            .await
+            .expect_err("an empty Expiration object must not count as an action");
+
+        assert_eq!(err.to_string(), ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION);
+    }
+
    #[tokio::test]
    async fn zero_newer_noncurrent_versions_means_no_count_constraint() {
        // Zero carries no constraint, matching how the batch limit path has
@@ -139,6 +139,7 @@ opentelemetry-stdout = { workspace = true }
 opentelemetry-otlp = { workspace = true, features = ["gzip-http", "reqwest-rustls"] }
 opentelemetry-semantic-conventions = { workspace = true, features = ["semconv_experimental"] }
 percent-encoding = { workspace = true }
+reqwest.workspace = true
 serde = { workspace = true, features = ["derive"] }
 serde_json = { workspace = true, features = ["raw_value"] }
 tracing = { workspace = true, features = ["std", "attributes"] }
@@ -167,3 +168,8 @@ tempfile = { workspace = true }
 temp-env = { workspace = true }
 log.workspace = true
 metrics-util = { workspace = true, features = ["debugging"] }
+rcgen.workspace = true
+rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "std"] }
+rustls-pki-types.workspace = true
+tokio = { workspace = true, features = ["io-util", "net"] }
+tokio-rustls = { workspace = true, default-features = false, features = ["aws-lc-rs"] }
@@ -115,6 +115,7 @@ All configuration is read from environment variables at startup.
 | `RUSTFS_OBS_TRACE_ENDPOINT`           | _(empty)_ | Dedicated trace endpoint (overrides root + `/v1/traces`)   |
 | `RUSTFS_OBS_METRIC_ENDPOINT`          | _(empty)_ | Dedicated metrics endpoint                                 |
 | `RUSTFS_OBS_LOG_ENDPOINT`             | _(empty)_ | Dedicated log endpoint                                     |
+| `RUSTFS_OBS_TLS_CA_FILE`              | _(empty)_ | Absolute PEM CA bundle added to system trust for OTLP/HTTP |
 | `RUSTFS_OBS_PROFILING_ENDPOINT`       | _(empty)_ | Dedicated profiling endpoint (e.g. Pyroscope)              |
 | `RUSTFS_OBS_TRACES_EXPORT_ENABLED`    | `true`    | Toggle trace export                                        |
 | `RUSTFS_OBS_METRICS_EXPORT_ENABLED`   | `true`    | Toggle metrics export                                      |
@@ -124,6 +125,8 @@ All configuration is read from environment variables at startup.
 | `RUSTFS_OBS_SAMPLE_RATIO`             | `0.1`     | Trace sampling ratio `0.0`–`1.0`                           |
 | `RUSTFS_OBS_METER_INTERVAL`           | `30`      | Metrics export interval (seconds)                          |

+`RUSTFS_OBS_TLS_CA_FILE` is read only while OTLP exporters are initialized. Its PEM certificates are added to, not substituted for, the platform trust roots, and TLS chain, expiry, and hostname checks remain enabled. Restart RustFS after rotating the bundle. A missing, unreadable, non-absolute, empty, or invalid bundle fails OTLP initialization.
+
 The export interval is separate from application metric collection. Without interval overrides, node/disk metrics refresh every 60 seconds (`RUSTFS_METRICS_NODE_INTERVAL_SEC`), cluster metrics every 60 seconds (`RUSTFS_METRICS_CLUSTER_INTERVAL_SEC`), and per-bucket metrics every 300 seconds (`RUSTFS_METRICS_BUCKET_INTERVAL_SEC`). See the [collection defaults](src/metrics/config.rs) and [interval configuration](src/metrics/scheduler.rs) for collector-specific and global overrides.

 With OTLP → Collector → Prometheus, the time from a failure to an alert also includes underlying health detection, collection and export work, transport and Collector batching, Prometheus `scrape_interval` and `evaluation_interval`, and any alert rule `for` duration. A shorter scrape interval cannot refresh a value that RustFS has not yet collected and exported. These periodic intervals do not guarantee a fixed detection or alert latency; see the [Prometheus alert rule documentation](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) for the role of `for`.
@@ -40,7 +40,8 @@ use rustfs_config::observability::{
    ENV_OBS_LOG_STDOUT_ENABLED, ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL, ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, ENV_OBS_LOG_ZSTD_WORKERS,
    ENV_OBS_LOGGER_LEVEL, ENV_OBS_LOGS_EXPORT_ENABLED, ENV_OBS_METER_INTERVAL, ENV_OBS_METRIC_ENDPOINT,
    ENV_OBS_METRICS_EXPORT_ENABLED, ENV_OBS_PROFILING_ENDPOINT, ENV_OBS_PROFILING_EXPORT_ENABLED, ENV_OBS_SAMPLE_RATIO,
-    ENV_OBS_SERVICE_NAME, ENV_OBS_SERVICE_VERSION, ENV_OBS_TRACE_ENDPOINT, ENV_OBS_TRACES_EXPORT_ENABLED, ENV_OBS_USE_STDOUT,
+    ENV_OBS_SERVICE_NAME, ENV_OBS_SERVICE_VERSION, ENV_OBS_TLS_CA_FILE, ENV_OBS_TRACE_ENDPOINT, ENV_OBS_TRACES_EXPORT_ENABLED,
+    ENV_OBS_USE_STDOUT,
 };
 use rustfs_config::{
    APP_NAME, DEFAULT_LOG_KEEP_FILES, DEFAULT_LOG_LEVEL, DEFAULT_LOG_ROTATION_TIME, DEFAULT_OBS_LOG_FILENAME,
@@ -139,6 +140,9 @@ pub struct OtelConfig {
    pub metric_timeout_millis: Option<u64>,
    /// Timeout (milliseconds) for log OTLP HTTP export.
    pub log_timeout_millis: Option<u64>,
+    /// Absolute path to a PEM bundle whose certificates are added to the
+    /// system trust roots used by OTLP/HTTP exporters.
+    pub tls_ca_file: Option<String>,
    /// Dedicated profiling endpoint.
    pub profiling_endpoint: Option<String>,
    /// Whether to export distributed traces (default: `true`).
@@ -292,6 +296,9 @@ impl OtelConfig {
            trace_timeout_millis: get_env_opt_u64(ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS),
            metric_timeout_millis: get_env_opt_u64(ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS),
            log_timeout_millis: get_env_opt_u64(ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS),
+            tls_ca_file: get_env_opt_str(ENV_OBS_TLS_CA_FILE)
+                .map(|path| path.trim().to_string())
+                .filter(|path| !path.is_empty()),
            profiling_endpoint: get_env_opt_str(ENV_OBS_PROFILING_ENDPOINT),
            traces_export_enabled: Some(get_env_bool(ENV_OBS_TRACES_EXPORT_ENABLED, DEFAULT_OBS_TRACES_EXPORT_ENABLED)),
            metrics_export_enabled: Some(get_env_bool(ENV_OBS_METRICS_EXPORT_ENABLED, DEFAULT_OBS_METRICS_EXPORT_ENABLED)),
@@ -509,4 +516,22 @@ mod tests {
            });
        });
    }
+
+    #[test]
+    fn otlp_tls_ca_file_environment_preserves_unset_empty_and_configured_values() {
+        with_profiling_env_lock(|| {
+            temp_env::with_var_unset(ENV_OBS_TLS_CA_FILE, || {
+                assert_eq!(OtelConfig::extract_otel_config_from_env(None).tls_ca_file, None);
+            });
+            temp_env::with_var(ENV_OBS_TLS_CA_FILE, Some(""), || {
+                assert_eq!(OtelConfig::extract_otel_config_from_env(None).tls_ca_file, None);
+            });
+            temp_env::with_var(ENV_OBS_TLS_CA_FILE, Some("/etc/rustfs/otlp-ca.pem"), || {
+                assert_eq!(
+                    OtelConfig::extract_otel_config_from_env(None).tls_ca_file.as_deref(),
+                    Some("/etc/rustfs/otlp-ca.pem")
+                );
+            });
+        });
+    }
 }
@@ -60,6 +60,18 @@ pub enum TelemetryError {
    BuildMetricExporter(String),
    #[error("Log exporter build failed: {0}")]
    BuildLogExporter(String),
+    #[error("RUSTFS_OBS_TLS_CA_FILE must be an absolute path")]
+    OtlpTlsCaPathNotAbsolute,
+    #[error("Failed to read RUSTFS_OBS_TLS_CA_FILE")]
+    OtlpTlsCaRead(#[source] std::io::Error),
+    #[error("RUSTFS_OBS_TLS_CA_FILE does not contain a PEM certificate")]
+    OtlpTlsCaEmpty,
+    #[error("RUSTFS_OBS_TLS_CA_FILE does not contain a valid PEM certificate")]
+    OtlpTlsCaInvalid,
+    #[error("Failed to parse RUSTFS_OBS_TLS_CA_FILE as a PEM certificate bundle")]
+    OtlpTlsCaParse(#[source] reqwest::Error),
+    #[error("Failed to build OTLP HTTP client")]
+    BuildOtlpHttpClient(#[source] reqwest::Error),
    #[error("Install metrics recorder failed: {0}")]
    InstallMetricsRecorder(String),
    #[error("Tracing subscriber init failed: {0}")]
@@ -89,6 +89,18 @@ fn rust_log_requests_verbose(rust_log: &str) -> bool {
    })
 }

+/// Returns `true` when the effective logging configuration selects a verbose
+/// (`debug`/`trace`) level.
+///
+/// Follows the same precedence as [`build_env_filter`]: `RUST_LOG` overrides
+/// `logger_level` when it is set.
+pub(super) fn effective_verbose_logging(logger_level: &str) -> bool {
+    match env::var("RUST_LOG").ok().filter(|value| !value.trim().is_empty()) {
+        Some(rust_log) => rust_log_requests_verbose(&rust_log),
+        None => is_verbose_level(logger_level),
+    }
+}
+
 fn should_suppress_noisy_crates(logger_level: &str, default_level: Option<&str>, rust_log: Option<&str>) -> bool {
    if let Some(level) = default_level {
        return !is_verbose_level(level);
@@ -36,7 +36,7 @@ use crate::cleaner::LogCleaner;
 use crate::cleaner::types::{CompressionAlgorithm, FileMatchMode};
 use crate::config::OtelConfig;
 use crate::global::{METRIC_LOG_CLEANER_RUN_FAILURES_TOTAL, METRIC_LOG_CLEANER_RUNS_TOTAL, set_observability_metric_enabled};
-use crate::telemetry::filter::{build_env_filter, pyroscope_log_filter};
+use crate::telemetry::filter::{build_env_filter, effective_verbose_logging, pyroscope_log_filter};
 use crate::telemetry::rolling::{RollingAppender, Rotation};
 use metrics::counter;
 use rustfs_config::observability::{
@@ -44,10 +44,11 @@ use rustfs_config::observability::{
    DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, DEFAULT_OBS_LOG_DELETE_EMPTY_FILES, DEFAULT_OBS_LOG_DRY_RUN,
    DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_MATCH_MODE, DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
    DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS, DEFAULT_OBS_LOG_PARALLEL_COMPRESS,
-    DEFAULT_OBS_LOG_PARALLEL_WORKERS, DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP,
-    DEFAULT_OBS_LOG_ZSTD_WORKERS,
+    DEFAULT_OBS_LOG_PARALLEL_WORKERS, DEFAULT_OBS_LOG_SPAN_EVENTS, DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL,
+    DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, DEFAULT_OBS_LOG_ZSTD_WORKERS, ENV_OBS_LOG_SPAN_EVENTS,
 };
 use rustfs_config::{APP_NAME, DEFAULT_LOG_KEEP_FILES, DEFAULT_LOG_ROTATION_TIME};
+use rustfs_utils::get_env_str;
 use serde_json::Value as JsonValue;
 use std::sync::Arc;
 use std::{collections::BTreeMap, fmt};
@@ -78,6 +79,37 @@ pub(super) fn resolve_file_stdout_mirror(configured: Option<bool>, is_production
    configured.unwrap_or(!is_production)
 }

+/// Resolves the span lifecycle events emitted by the JSON log layers.
+///
+/// Defaults to [`FmtSpan::NONE`]. A record per span lifecycle multiplies log
+/// volume on object hot paths and, combined with the ancestor span list,
+/// produced multi-megabyte lines that journald truncated (backlog#2642).
+/// Operators opt back in with `RUSTFS_OBS_LOG_SPAN_EVENTS=close|full`.
+pub(super) fn resolve_span_events() -> FmtSpan {
+    match get_env_str(ENV_OBS_LOG_SPAN_EVENTS, DEFAULT_OBS_LOG_SPAN_EVENTS)
+        .trim()
+        .to_ascii_lowercase()
+        .as_str()
+    {
+        "close" => FmtSpan::CLOSE,
+        "full" => FmtSpan::FULL,
+        "none" | "" => FmtSpan::NONE,
+        unknown => {
+            // An unrecognised value must not silently re-enable span events.
+            eprintln!("{STDERR_WARNING_PREFIX} obs: unknown {ENV_OBS_LOG_SPAN_EVENTS}={unknown:?}; falling back to none");
+            FmtSpan::NONE
+        }
+    }
+}
+
+/// Whether JSON log records embed the full ancestor span list.
+///
+/// `with_span_list(true)` re-serializes every ancestor span on each record, so
+/// it is reserved for verbose levels where that context earns its volume.
+pub(super) fn resolve_span_list(logger_level: &str) -> bool {
+    effective_verbose_logging(logger_level)
+}
+
 #[cfg(unix)]
 pub(super) fn validate_stdout_sink(file_appender: &RollingAppender) -> Result<(), TelemetryError> {
    use std::os::fd::AsFd;
@@ -222,7 +254,12 @@ where
    request_id
 }

-pub(super) fn build_json_log_layer<S, W>(writer: W, enable_ansi: bool, span_events: FmtSpan) -> impl tracing_subscriber::Layer<S>
+pub(super) fn build_json_log_layer<S, W>(
+    writer: W,
+    enable_ansi: bool,
+    span_events: FmtSpan,
+    span_list: bool,
+) -> impl tracing_subscriber::Layer<S>
 where
    S: Subscriber + for<'span> LookupSpan<'span>,
    W: for<'writer> tracing_subscriber::fmt::MakeWriter<'writer> + Send + Sync + 'static,
@@ -239,7 +276,7 @@ where
        .json()
        .flatten_event(true)
        .with_current_span(true)
-        .with_span_list(true)
+        .with_span_list(span_list)
        .with_span_events(span_events)
        .map_event_format(RequestIdJsonFormat::new)
 }
@@ -311,11 +348,12 @@ pub(super) fn init_local_logging(
 fn init_stdout_only(_config: &OtelConfig, logger_level: &str, is_production: bool) -> Result<OtelGuard, TelemetryError> {
    let env_filter = build_env_filter(logger_level, None);
    let (nb, guard) = tracing_appender::non_blocking(std::io::stdout());
-    let span_events = if is_production { FmtSpan::CLOSE } else { FmtSpan::FULL };
+    let span_events = resolve_span_events();
+    let span_list = resolve_span_list(logger_level);

    // Keep stdout formatting JSON-shaped even in local-only mode so operators
    // can ship the same log schema to external collectors if needed.
-    let fmt_layer = build_json_log_layer(nb, std::io::stdout().is_terminal(), span_events);
+    let fmt_layer = build_json_log_layer(nb, std::io::stdout().is_terminal(), span_events, span_list);

    tracing_subscriber::registry()
        .with(env_filter)
@@ -413,11 +451,12 @@ fn init_file_logging_internal(

    // ── 4. Build subscriber layers ────────────────────────────────────────────
    let env_filter = build_env_filter(logger_level, None);
-    let span_events = if is_production { FmtSpan::CLOSE } else { FmtSpan::FULL };
+    let span_events = resolve_span_events();
+    let span_list = resolve_span_list(logger_level);

    // File output stays machine-readable and free of ANSI sequences so the
    // resulting files are safe to parse or ship to log processors.
-    let file_layer = build_json_log_layer(non_blocking, false, span_events.clone());
+    let file_layer = build_json_log_layer(non_blocking, false, span_events.clone(), span_list);

    // Optional stdout mirror: enabled explicitly via `log_stdout_enabled`, or
    // unconditionally in non-production environments so developers still see
@@ -425,7 +464,10 @@ fn init_file_logging_internal(
    let (stdout_layer, stdout_guard) = if resolve_file_stdout_mirror(config.log_stdout_enabled, is_production) {
        let (stdout_nb, stdout_guard) = tracing_appender::non_blocking(std::io::stdout());
        let enable_color = std::io::stdout().is_terminal();
-        (Some(build_json_log_layer(stdout_nb, enable_color, span_events)), Some(stdout_guard))
+        (
+            Some(build_json_log_layer(stdout_nb, enable_color, span_events, span_list)),
+            Some(stdout_guard),
+        )
    } else {
        (None, None)
    };
@@ -773,7 +815,7 @@ mod tests {

    fn render_json_log(enable_ansi: bool) -> (String, Value) {
        let writer = SharedWriter::default();
-        let layer = build_json_log_layer(writer.clone(), enable_ansi, FmtSpan::NONE);
+        let layer = build_json_log_layer(writer.clone(), enable_ansi, FmtSpan::NONE, false);
        let subscriber = Registry::default().with(layer);

        tracing::subscriber::with_default(subscriber, || {
@@ -793,9 +835,32 @@ mod tests {
        (raw, parsed)
    }

+    fn render_event_in_nested_spans(depth: usize, span_list: bool) -> String {
+        let writer = SharedWriter::default();
+        let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE, span_list);
+        let subscriber = Registry::default().with(layer);
+
+        tracing::subscriber::with_default(subscriber, || {
+            let spans = (0..depth)
+                .map(|level| tracing::info_span!("layer", depth = level, payload = "x".repeat(1024)))
+                .collect::<Vec<_>>();
+            let _guards = spans.iter().map(tracing::Span::enter).collect::<Vec<_>>();
+            info!(component = "obs_contract_test", "nested event");
+        });
+
+        String::from_utf8(
+            writer
+                .inner
+                .lock()
+                .expect("shared writer lock should not be poisoned")
+                .clone(),
+        )
+        .expect("log output should be valid UTF-8")
+    }
+
    fn render_json_log_with_request_span() -> Value {
        let writer = SharedWriter::default();
-        let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE);
+        let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE, false);
        let subscriber = Registry::default().with(layer);

        tracing::subscriber::with_default(subscriber, || {
@@ -818,7 +883,7 @@ mod tests {

    fn render_json_log_with_recovery_monitor_child_span() -> Value {
        let writer = SharedWriter::default();
-        let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE);
+        let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE, true);
        let subscriber = Registry::default().with(layer);

        tracing::subscriber::with_default(subscriber, || {
@@ -1075,6 +1140,10 @@ mod tests {
        assert_eq!(parsed["request-id"], Value::String("req-123".to_string()));
        assert_eq!(parsed["message"], Value::String("inside request span".to_string()));
        assert_eq!(parsed["span"]["request_id"], Value::String("req-123".to_string()));
+        assert!(
+            parsed.get("spans").is_none(),
+            "ancestor span list must stay out of records when span_list is disabled: {parsed}"
+        );
    }

    #[test]
@@ -1112,4 +1181,65 @@ mod tests {

        assert_eq!(resolve_log_cleanup_file_pattern(&config, "rustfs.log"), "rustfs.log");
    }
+
+    /// Regression for backlog#2642: records must not embed the ancestor span
+    /// list, so their size stays flat as the enclosing span chain deepens.
+    #[test]
+    fn record_size_does_not_grow_with_span_depth_when_span_list_is_disabled() {
+        let shallow = render_event_in_nested_spans(1, false);
+        let deep = render_event_in_nested_spans(32, false);
+
+        assert!(
+            deep.len() <= shallow.len() * 2,
+            "ancestor spans leaked into the record: shallow={} deep={}",
+            shallow.len(),
+            deep.len()
+        );
+
+        // The flag is what keeps it bounded: re-enabling the list must show the
+        // growth the default avoids.
+        let deep_with_list = render_event_in_nested_spans(32, true);
+        assert!(
+            deep_with_list.len() > deep.len(),
+            "enabling the span list must reintroduce per-ancestor growth"
+        );
+    }
+
+    #[test]
+    fn span_events_default_to_none_and_parse_overrides() {
+        temp_env::with_var(ENV_OBS_LOG_SPAN_EVENTS, None::<&str>, || {
+            assert_eq!(resolve_span_events(), FmtSpan::NONE);
+        });
+        temp_env::with_var(ENV_OBS_LOG_SPAN_EVENTS, Some("close"), || {
+            assert_eq!(resolve_span_events(), FmtSpan::CLOSE);
+        });
+        temp_env::with_var(ENV_OBS_LOG_SPAN_EVENTS, Some("FULL"), || {
+            assert_eq!(resolve_span_events(), FmtSpan::FULL);
+        });
+        temp_env::with_var(ENV_OBS_LOG_SPAN_EVENTS, Some("verbose"), || {
+            assert_eq!(
+                resolve_span_events(),
+                FmtSpan::NONE,
+                "an unrecognised value must not re-enable span events"
+            );
+        });
+    }
+
+    #[test]
+    fn span_list_is_reserved_for_verbose_levels() {
+        temp_env::with_var("RUST_LOG", None::<&str>, || {
+            assert!(!resolve_span_list("info"), "info records must not embed the ancestor span list");
+            assert!(!resolve_span_list("warn"));
+            assert!(!resolve_span_list("error"));
+            assert!(resolve_span_list("debug"));
+            assert!(resolve_span_list("trace"));
+        });
+
+        temp_env::with_var("RUST_LOG", Some("debug"), || {
+            assert!(
+                resolve_span_list("info"),
+                "RUST_LOG=debug overrides the base level and should keep span context"
+            );
+        });
+    }
 }
@@ -64,11 +64,11 @@ use rustfs_config::{
    DEFAULT_OBS_METRICS_EXPORT_ENABLED, DEFAULT_OBS_TRACES_EXPORT_ENABLED, METER_INTERVAL, SAMPLE_RATIO,
 };
 use std::collections::HashMap;
-use std::{fs, io::IsTerminal, time::Duration};
+use std::{fs, io::IsTerminal, path::Path, time::Duration};
 use tracing::{info, warn};
 use tracing_error::ErrorLayer;
 use tracing_opentelemetry::{MetricsLayer, OpenTelemetryLayer};
-use tracing_subscriber::{fmt::format::FmtSpan, layer::SubscriberExt, util::SubscriberInitExt};
+use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

 const GET_OBJECT_DURATION_HISTOGRAM_METRICS: &[&str] = &[
    "rustfs_io_get_object_request_duration_seconds",
@@ -83,6 +83,12 @@ const GET_OBJECT_DURATION_HISTOGRAM_BUCKETS: &[f64] = &[
    0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0,
 ];

+const DEFAULT_OTLP_HTTP_TIMEOUT: Duration = Duration::from_secs(10);
+const OTEL_EXPORTER_OTLP_TIMEOUT: &str = "OTEL_EXPORTER_OTLP_TIMEOUT";
+const OTEL_EXPORTER_OTLP_TRACES_TIMEOUT: &str = "OTEL_EXPORTER_OTLP_TRACES_TIMEOUT";
+const OTEL_EXPORTER_OTLP_METRICS_TIMEOUT: &str = "OTEL_EXPORTER_OTLP_METRICS_TIMEOUT";
+const OTEL_EXPORTER_OTLP_LOGS_TIMEOUT: &str = "OTEL_EXPORTER_OTLP_LOGS_TIMEOUT";
+
 #[cfg(all(
    feature = "pyroscope",
    any(target_os = "macos", all(target_os = "linux", target_env = "gnu", target_arch = "x86_64"))
@@ -130,6 +136,8 @@ pub(super) fn init_observability_http(
    logger_level: &str,
    is_production: bool,
 ) -> Result<OtelGuard, TelemetryError> {
+    let otlp_tls_ca_bundle = OtlpTlsCaBundle::load(config)?;
+
    // ── Resource & sampling ──────────────────────────────────────────────────
    // Build the common resource once so all enabled signals report the same
    // service identity and deployment metadata.
@@ -153,10 +161,12 @@ pub(super) fn init_observability_http(
    let log_ep = resolve_signal_endpoint(config.log_endpoint.as_deref(), root_ep, "/v1/logs");

    // ── Tracer provider (HTTP) ────────────────────────────────────────────────
-    let tracer_provider = build_tracer_provider(&trace_ep, config, res.clone(), sampler, use_stdout)?;
+    let tracer_provider =
+        build_tracer_provider(&trace_ep, config, otlp_tls_ca_bundle.as_ref(), res.clone(), sampler, use_stdout)?;

    // ── Meter provider (HTTP) ─────────────────────────────────────────────────
-    let meter_provider = build_meter_provider(&metric_ep, config, res.clone(), &service_name, use_stdout)?;
+    let meter_provider =
+        build_meter_provider(&metric_ep, config, otlp_tls_ca_bundle.as_ref(), res.clone(), &service_name, use_stdout)?;

    // ── Logger Logic ──────────────────────────────────────────────────────────
    // Logging is the only signal that may intentionally route to either OTLP
@@ -175,7 +185,7 @@ pub(super) fn init_observability_http(
        // Init OTLP logger logic.
        // We initialize the OTLP collector and honor the configured stdout setting
        // (e.g. via RUSTFS_OBS_USE_STDOUT / config.use_stdout) when building the provider.
-        logger_provider = build_logger_provider(&log_ep, config, res, use_stdout)?;
+        logger_provider = build_logger_provider(&log_ep, config, otlp_tls_ca_bundle.as_ref(), res, use_stdout)?;

        // Build bridge to capture `tracing` events.
        otel_bridge = logger_provider.as_ref().map(OpenTelemetryTracingBridge::new);
@@ -184,7 +194,8 @@ pub(super) fn init_observability_http(
        // active, the OpenTelemetry bridge is the authoritative sink for
        // `tracing` events unless local file logging is needed as a fallback.
    }
-    let span_events = if is_production { FmtSpan::CLOSE } else { FmtSpan::FULL };
+    let span_events = crate::telemetry::local::resolve_span_events();
+    let span_list = crate::telemetry::local::resolve_span_list(logger_level);
    // ── Case 2: File Logging
    // If a log directory is configured and OTLP log export is unavailable, use
    // the same rolling-file behavior as the local-only telemetry backend.
@@ -221,7 +232,7 @@ pub(super) fn init_observability_http(
            crate::telemetry::local::validate_stdout_sink(&file_appender)?;

            let (non_blocking, guard) = tracing_appender::non_blocking(file_appender);
-            let file_layer = build_json_log_layer(non_blocking, false, span_events.clone());
+            let file_layer = build_json_log_layer(non_blocking, false, span_events.clone(), span_list);
            let cleanup_handle = spawn_cleanup_task(config, log_directory, log_filename, keep_files);
            Ok((file_layer, guard, cleanup_handle, rotation_str))
        })();
@@ -267,7 +278,7 @@ pub(super) fn init_observability_http(
    if force_stdout_logging || crate::telemetry::local::resolve_file_stdout_mirror(config.log_stdout_enabled, is_production) {
        let (stdout_nb, stdout_g) = tracing_appender::non_blocking(std::io::stdout());
        stdout_guard = Some(stdout_g);
-        stdout_layer_opt = Some(build_json_log_layer(stdout_nb, std::io::stdout().is_terminal(), span_events));
+        stdout_layer_opt = Some(build_json_log_layer(stdout_nb, std::io::stdout().is_terminal(), span_events, span_list));
    }
    let local_file_fallback_enabled = file_layer_opt.is_some();
    let stdout_mirror_enabled = stdout_guard.is_some();
@@ -320,6 +331,7 @@ pub(super) fn init_observability_http(
 fn build_tracer_provider(
    trace_ep: &str,
    config: &OtelConfig,
+    otlp_tls_ca_bundle: Option<&OtlpTlsCaBundle>,
    res: opentelemetry_sdk::Resource,
    sampler: Sampler,
    use_stdout: bool,
@@ -337,9 +349,13 @@ fn build_tracer_provider(
    if !trace_headers.is_empty() {
        exporter_builder = exporter_builder.with_headers(trace_headers);
    }
-    if let Some(timeout) = resolve_signal_timeout(config.endpoint_timeout_millis, config.trace_timeout_millis) {
+    let timeout = resolve_signal_timeout(config.endpoint_timeout_millis, config.trace_timeout_millis);
+    if let Some(timeout) = timeout {
        exporter_builder = exporter_builder.with_timeout(timeout);
    }
+    if let Some(http_client) = build_otlp_http_client(otlp_tls_ca_bundle, timeout, OTEL_EXPORTER_OTLP_TRACES_TIMEOUT)? {
+        exporter_builder = exporter_builder.with_http_client(http_client);
+    }
    let exporter = exporter_builder
        .build()
        .map_err(|e| TelemetryError::BuildSpanExporter(e.to_string()))?;
@@ -408,6 +424,7 @@ fn resolve_signal_endpoint(signal_endpoint: Option<&str>, root_ep: &str, suffix:
 fn build_meter_provider(
    metric_ep: &str,
    config: &OtelConfig,
+    otlp_tls_ca_bundle: Option<&OtlpTlsCaBundle>,
    res: opentelemetry_sdk::Resource,
    service_name: &str,
    use_stdout: bool,
@@ -426,9 +443,13 @@ fn build_meter_provider(
    if !metric_headers.is_empty() {
        exporter_builder = exporter_builder.with_headers(metric_headers);
    }
-    if let Some(timeout) = resolve_signal_timeout(config.endpoint_timeout_millis, config.metric_timeout_millis) {
+    let timeout = resolve_signal_timeout(config.endpoint_timeout_millis, config.metric_timeout_millis);
+    if let Some(timeout) = timeout {
        exporter_builder = exporter_builder.with_timeout(timeout);
    }
+    if let Some(http_client) = build_otlp_http_client(otlp_tls_ca_bundle, timeout, OTEL_EXPORTER_OTLP_METRICS_TIMEOUT)? {
+        exporter_builder = exporter_builder.with_http_client(http_client);
+    }
    let exporter = exporter_builder
        .build()
        .map_err(|e| TelemetryError::BuildMetricExporter(e.to_string()))?;
@@ -484,6 +505,7 @@ fn is_get_object_duration_histogram_metric(name: &str) -> bool {
 fn build_logger_provider(
    log_ep: &str,
    config: &OtelConfig,
+    otlp_tls_ca_bundle: Option<&OtlpTlsCaBundle>,
    res: opentelemetry_sdk::Resource,
    use_stdout: bool,
 ) -> Result<Option<SdkLoggerProvider>, TelemetryError> {
@@ -500,9 +522,13 @@ fn build_logger_provider(
    if !log_headers.is_empty() {
        exporter_builder = exporter_builder.with_headers(log_headers);
    }
-    if let Some(timeout) = resolve_signal_timeout(config.endpoint_timeout_millis, config.log_timeout_millis) {
+    let timeout = resolve_signal_timeout(config.endpoint_timeout_millis, config.log_timeout_millis);
+    if let Some(timeout) = timeout {
        exporter_builder = exporter_builder.with_timeout(timeout);
    }
+    if let Some(http_client) = build_otlp_http_client(otlp_tls_ca_bundle, timeout, OTEL_EXPORTER_OTLP_LOGS_TIMEOUT)? {
+        exporter_builder = exporter_builder.with_http_client(http_client);
+    }
    let exporter = exporter_builder
        .build()
        .map_err(|e| TelemetryError::BuildLogExporter(e.to_string()))?;
@@ -659,16 +685,239 @@ fn resolve_signal_timeout(common_timeout_millis: Option<u64>, signal_timeout_mil
        .map(Duration::from_millis)
 }

+struct OtlpTlsCaBundle {
+    certificates: Vec<reqwest::Certificate>,
+}
+
+impl OtlpTlsCaBundle {
+    fn load(config: &OtelConfig) -> Result<Option<Self>, TelemetryError> {
+        let Some(ca_file) = config.tls_ca_file.as_deref() else {
+            return Ok(None);
+        };
+        let path = Path::new(ca_file);
+        if !path.is_absolute() {
+            return Err(TelemetryError::OtlpTlsCaPathNotAbsolute);
+        }
+
+        let pem_bundle = fs::read(path).map_err(TelemetryError::OtlpTlsCaRead)?;
+        if pem_bundle.iter().all(u8::is_ascii_whitespace) {
+            return Err(TelemetryError::OtlpTlsCaEmpty);
+        }
+        let certificates = reqwest::Certificate::from_pem_bundle(&pem_bundle).map_err(TelemetryError::OtlpTlsCaParse)?;
+        if certificates.is_empty() {
+            return Err(TelemetryError::OtlpTlsCaInvalid);
+        }
+        Ok(Some(Self { certificates }))
+    }
+
+    fn build_http_client(
+        &self,
+        configured_timeout: Option<Duration>,
+        otel_signal_timeout_env: &str,
+    ) -> Result<reqwest::Client, TelemetryError> {
+        let timeout = configured_timeout.unwrap_or_else(|| resolve_otlp_http_timeout(otel_signal_timeout_env));
+        reqwest::Client::builder()
+            .timeout(timeout)
+            .tls_certs_merge(self.certificates.clone())
+            .build()
+            .map_err(TelemetryError::BuildOtlpHttpClient)
+    }
+}
+
+fn build_otlp_http_client(
+    otlp_tls_ca_bundle: Option<&OtlpTlsCaBundle>,
+    configured_timeout: Option<Duration>,
+    otel_signal_timeout_env: &str,
+) -> Result<Option<reqwest::Client>, TelemetryError> {
+    otlp_tls_ca_bundle
+        .map(|bundle| bundle.build_http_client(configured_timeout, otel_signal_timeout_env))
+        .transpose()
+}
+
+fn resolve_otlp_http_timeout(signal_timeout_env: &str) -> Duration {
+    [signal_timeout_env, OTEL_EXPORTER_OTLP_TIMEOUT]
+        .into_iter()
+        .find_map(|name| std::env::var(name).ok().and_then(|value| value.parse::<u64>().ok()))
+        .map(Duration::from_millis)
+        .unwrap_or(DEFAULT_OTLP_HTTP_TIMEOUT)
+}
+
 #[cfg(test)]
 mod tests {
    use super::*;
    use std::env;
    use std::io::{self, Write};
+    use std::path::Path;
    use std::process::Command;
    use std::sync::{Arc, Mutex};
+    use tokio::io::{AsyncReadExt, AsyncWriteExt};
+    use tokio::net::TcpListener;

    const LOG_TRACER_CHILD_ENV: &str = "RUSTFS_OBS_LOG_TRACER_CHILD";

+    fn config_with_tls_ca_file(path: &Path) -> OtelConfig {
+        OtelConfig {
+            tls_ca_file: Some(path.display().to_string()),
+            ..OtelConfig::default()
+        }
+    }
+
+    fn load_tls_ca_bundle(path: &Path) -> OtlpTlsCaBundle {
+        OtlpTlsCaBundle::load(&config_with_tls_ca_file(path))
+            .expect("load CA bundle")
+            .expect("configured CA file should produce a bundle")
+    }
+
+    async fn spawn_test_tls_server() -> (String, String, tokio::task::JoinHandle<bool>) {
+        let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
+        let certified =
+            rcgen::generate_simple_self_signed(vec!["127.0.0.1".to_string()]).expect("generate TLS server certificate");
+        let ca_pem = certified.cert.pem();
+        let private_key = rustls_pki_types::PrivateKeyDer::try_from(certified.signing_key.serialize_der())
+            .expect("convert TLS server private key");
+        let config = rustls::ServerConfig::builder()
+            .with_no_client_auth()
+            .with_single_cert(vec![certified.cert.der().clone()], private_key)
+            .expect("build TLS server config");
+        let acceptor = tokio_rustls::TlsAcceptor::from(Arc::new(config));
+        let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind TLS test server");
+        let endpoint = format!("https://{}", listener.local_addr().expect("read TLS test server address"));
+        let server = tokio::spawn(async move {
+            let Ok((stream, _)) = listener.accept().await else {
+                return false;
+            };
+            let Ok(mut stream) = acceptor.accept(stream).await else {
+                return false;
+            };
+            let mut request = Vec::new();
+            let mut buffer = [0_u8; 1_024];
+            loop {
+                let Ok(read) = stream.read(&mut buffer).await else {
+                    return false;
+                };
+                if read == 0 {
+                    return false;
+                }
+                request.extend_from_slice(&buffer[..read]);
+                if request.windows(4).any(|window| window == b"\r\n\r\n") {
+                    break;
+                }
+            }
+            stream
+                .write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 2\r\nconnection: close\r\n\r\nok")
+                .await
+                .is_ok()
+                && stream.shutdown().await.is_ok()
+        });
+        (endpoint, ca_pem, server)
+    }
+
+    #[tokio::test]
+    async fn otlp_http_client_trusts_every_certificate_in_a_pem_bundle() {
+        let (first_endpoint, first_ca_pem, first_server) = spawn_test_tls_server().await;
+        let (second_endpoint, second_ca_pem, second_server) = spawn_test_tls_server().await;
+        let file = tempfile::NamedTempFile::new().expect("create CA bundle file");
+        std::fs::write(file.path(), format!("{first_ca_pem}\n{second_ca_pem}")).expect("write CA bundle");
+
+        let bundle = load_tls_ca_bundle(file.path());
+        let client = build_otlp_http_client(Some(&bundle), Some(Duration::from_millis(250)), OTEL_EXPORTER_OTLP_TRACES_TIMEOUT)
+            .expect("build client with CA bundle");
+        let client = client.expect("custom CA should build a client");
+        assert_eq!(
+            client
+                .get(first_endpoint)
+                .send()
+                .await
+                .expect("first bundled CA should complete TLS handshake")
+                .status(),
+            reqwest::StatusCode::OK
+        );
+        assert_eq!(
+            client
+                .get(second_endpoint)
+                .send()
+                .await
+                .expect("second bundled CA should complete TLS handshake")
+                .status(),
+            reqwest::StatusCode::OK
+        );
+        assert!(first_server.await.expect("first bundled TLS server task"));
+        assert!(second_server.await.expect("second bundled TLS server task"));
+    }
+
+    #[test]
+    fn otlp_http_client_rejects_relative_missing_empty_and_invalid_ca_files() {
+        let relative = OtelConfig {
+            tls_ca_file: Some("otlp-ca.pem".to_string()),
+            ..OtelConfig::default()
+        };
+        assert!(matches!(OtlpTlsCaBundle::load(&relative), Err(TelemetryError::OtlpTlsCaPathNotAbsolute)));
+
+        let missing_directory = tempfile::tempdir().expect("create missing CA directory");
+        let missing = OtelConfig {
+            tls_ca_file: Some(missing_directory.path().join("otlp-ca.pem").display().to_string()),
+            ..OtelConfig::default()
+        };
+        assert!(matches!(OtlpTlsCaBundle::load(&missing), Err(TelemetryError::OtlpTlsCaRead(_))));
+
+        let empty = tempfile::NamedTempFile::new().expect("create empty CA file");
+        assert!(matches!(
+            OtlpTlsCaBundle::load(&config_with_tls_ca_file(empty.path())),
+            Err(TelemetryError::OtlpTlsCaEmpty)
+        ));
+
+        let invalid = tempfile::NamedTempFile::new().expect("create invalid CA file");
+        std::fs::write(invalid.path(), b"not a PEM certificate").expect("write invalid CA file");
+        assert!(matches!(
+            OtlpTlsCaBundle::load(&config_with_tls_ca_file(invalid.path())),
+            Err(TelemetryError::OtlpTlsCaInvalid)
+        ));
+    }
+
+    #[tokio::test]
+    async fn otlp_http_client_trusts_the_configured_ca_and_rejects_another_ca() {
+        let (endpoint, ca_pem, trusted_server) = spawn_test_tls_server().await;
+        let trusted_file = tempfile::NamedTempFile::new().expect("create trusted CA file");
+        std::fs::write(trusted_file.path(), ca_pem).expect("write trusted CA file");
+        let trusted_bundle = load_tls_ca_bundle(trusted_file.path());
+        let trusted_client =
+            build_otlp_http_client(Some(&trusted_bundle), Some(Duration::from_secs(1)), OTEL_EXPORTER_OTLP_TRACES_TIMEOUT)
+                .expect("build trusted client")
+                .expect("custom CA should build a client");
+        assert_eq!(
+            trusted_client
+                .get(&endpoint)
+                .send()
+                .await
+                .expect("configured CA should complete TLS handshake")
+                .status(),
+            reqwest::StatusCode::OK
+        );
+        assert!(trusted_server.await.expect("trusted TLS server task"));
+
+        let (endpoint, _ca_pem, untrusted_server) = spawn_test_tls_server().await;
+        let wrong_ca =
+            rcgen::generate_simple_self_signed(vec!["wrong.test".to_string()]).expect("generate unrelated CA certificate");
+        let untrusted_file = tempfile::NamedTempFile::new().expect("create unrelated CA file");
+        std::fs::write(untrusted_file.path(), wrong_ca.cert.pem()).expect("write unrelated CA file");
+        let untrusted_bundle = load_tls_ca_bundle(untrusted_file.path());
+        let untrusted_client =
+            build_otlp_http_client(Some(&untrusted_bundle), Some(Duration::from_secs(1)), OTEL_EXPORTER_OTLP_TRACES_TIMEOUT)
+                .expect("build untrusted client")
+                .expect("custom CA should build a client");
+        assert!(untrusted_client.get(&endpoint).send().await.is_err());
+        assert!(!untrusted_server.await.expect("untrusted TLS server task"));
+    }
+
+    #[test]
+    fn otlp_http_client_is_not_built_without_a_custom_ca_file() {
+        assert!(
+            build_otlp_http_client(None, None, OTEL_EXPORTER_OTLP_TRACES_TIMEOUT)
+                .expect("no CA file keeps the default client")
+                .is_none()
+        );
+    }
+
    #[derive(Clone)]
    struct TestWriter(Arc<Mutex<Vec<u8>>>);

@@ -133,6 +133,8 @@ pub struct DeleteObjectCall {
 }

 struct Inner {
+    capture_upload_bodies: bool,
+    upload_bodies: Vec<Vec<u8>>,
    // Response queues. Each method pops from its own queue. Empty queue
    // plus no default means a configured-miss error.
    get_object: VecDeque<Result<GetObjectOutput, DummyError>>,
@@ -190,6 +192,8 @@ struct Inner {
 impl Inner {
    fn new() -> Self {
        Self {
+            capture_upload_bodies: false,
+            upload_bodies: Vec::new(),
            get_object: VecDeque::new(),
            get_object_range: VecDeque::new(),
            put_object: VecDeque::new(),
@@ -265,6 +269,30 @@ impl DummyBackend {
        }
    }

+    /// Opt in to consuming upload bodies for byte-for-byte protocol tests.
+    pub fn capture_upload_bodies(&self) {
+        self.inner.lock().expect("lock").capture_upload_bodies = true;
+    }
+
+    pub fn upload_bodies(&self) -> Vec<Vec<u8>> {
+        self.inner.lock().expect("lock").upload_bodies.clone()
+    }
+
+    async fn record_upload_body(&self, body: &mut Option<StreamingBlob>) -> Result<(), DummyError> {
+        use futures_util::TryStreamExt;
+        if !self.inner.lock().expect("lock").capture_upload_bodies {
+            return Ok(());
+        }
+        let mut bytes = Vec::new();
+        if let Some(mut stream) = body.take() {
+            while let Some(chunk) = stream.try_next().await.map_err(|e| DummyError::Injected(e.to_string()))? {
+                bytes.extend_from_slice(&chunk);
+            }
+        }
+        self.inner.lock().expect("lock").upload_bodies.push(bytes);
+        Ok(())
+    }
+
    // Queue-configuration helpers. Each test stages the responses it
    // expects in order. The method pops in FIFO order.

@@ -629,7 +657,8 @@ impl StorageBackend for DummyBackend {
        }
    }

-    async fn put_object(&self, input: PutObjectInput, _credentials: &Credentials) -> Result<PutObjectOutput, Self::Error> {
+    async fn put_object(&self, mut input: PutObjectInput, _credentials: &Credentials) -> Result<PutObjectOutput, Self::Error> {
+        self.record_upload_body(&mut input.body).await?;
        // Decide control flow while holding the lock. Release before
        // awaiting so the stall path does not hold the Mutex across
        // an await point.
@@ -789,7 +818,8 @@ impl StorageBackend for DummyBackend {
        }
    }

-    async fn upload_part(&self, input: UploadPartInput, _credentials: &Credentials) -> Result<UploadPartOutput, Self::Error> {
+    async fn upload_part(&self, mut input: UploadPartInput, _credentials: &Credentials) -> Result<UploadPartOutput, Self::Error> {
+        self.record_upload_body(&mut input.body).await?;
        // Record the call and decide the control flow while holding the
        // lock. Release the lock before awaiting so the stall path does
        // not hold the Mutex across an await point.
@@ -150,10 +150,10 @@ pub fn is_operation_supported(protocol: super::session::Protocol, action: &S3Act
            S3Action::HeadObject => true,   // SIZE command

            // Multipart operations
-            S3Action::CreateMultipartUpload => false,
-            S3Action::UploadPart => false,
-            S3Action::CompleteMultipartUpload => false,
-            S3Action::AbortMultipartUpload => false,
+            S3Action::CreateMultipartUpload => true,
+            S3Action::UploadPart => true,
+            S3Action::CompleteMultipartUpload => true,
+            S3Action::AbortMultipartUpload => true,
            S3Action::ListMultipartUploads => false,
            S3Action::ListParts => false,

@@ -16,12 +16,12 @@ use crate::common::client::s3::StorageBackend as S3StorageBackend;
 use crate::common::gateway::S3Action;
 use crate::common::gateway::authorize_operation;
 use async_trait::async_trait;
-use futures_util::stream;
 use rustfs_utils::MaskedAccessKey;
 use rustfs_utils::path;
 use s3s::dto::*;
 use std::fmt::Debug;
 use std::path::{Path, PathBuf};
+use std::sync::Arc;
 use tokio::io::AsyncRead;
 use tracing::{debug, error};
 use unftp_core::storage::{Error, ErrorKind, Fileinfo, Metadata, Result, StorageBackend};
@@ -94,12 +94,12 @@ impl Metadata for FtpsMetadata {
 /// FTPS storage driver implementation
 pub struct FtpsDriver<S> {
    /// Storage backend for S3 operations
-    storage: S,
+    storage: Arc<S>,
 }

 impl<S> Debug for FtpsDriver<S>
 where
-    S: S3StorageBackend + Debug,
+    S: S3StorageBackend + Debug + 'static,
 {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        f.debug_struct("FtpsDriver").field("storage", &"StorageBackend").finish()
@@ -108,11 +108,13 @@ where

 impl<S> FtpsDriver<S>
 where
-    S: S3StorageBackend + Debug,
+    S: S3StorageBackend + Debug + 'static,
 {
    /// Create a new FTPS driver with the given storage backend
    pub fn new(storage: S) -> Self {
-        Self { storage }
+        Self {
+            storage: Arc::new(storage),
+        }
    }

    /// List all buckets (for root path)
@@ -232,7 +234,7 @@ where
 #[async_trait]
 impl<S> StorageBackend<super::server::FtpsUser> for FtpsDriver<S>
 where
-    S: S3StorageBackend + Debug,
+    S: S3StorageBackend + Debug + 'static,
 {
    type Metadata = FtpsMetadata;

@@ -544,55 +546,18 @@ where
            .await
            .map_err(|_| Error::new(ErrorKind::PermanentFileNotAvailable, "Access denied"))?;

-        // Convert AsyncRead to bytes
-        let bytes_vec = {
-            let mut buffer = Vec::new();
-            let mut reader = bytes;
-            tokio::io::copy(&mut reader, &mut buffer)
-                .await
-                .map_err(|e| Error::new(ErrorKind::TransientFileNotAvailable, e.to_string()))?;
-            buffer
-        };
-
-        let file_size = bytes_vec.len();
-
-        let mut put_builder = PutObjectInput::builder();
-        put_builder.set_bucket(bucket.clone());
-        put_builder.set_key(key.clone());
-        put_builder.set_content_length(Some(file_size as i64));
-
-        // Create StreamingBlob with known size
-        let data_bytes = bytes::Bytes::from(bytes_vec);
-        let stream = stream::once(async move { Ok::<bytes::Bytes, std::io::Error>(data_bytes) });
-        let streaming_blob = s3s::dto::StreamingBlob::wrap(stream);
-        put_builder.set_body(Some(streaming_blob));
-        let put_input = put_builder
-            .build()
-            .map_err(|_| Error::new(ErrorKind::PermanentFileNotAvailable, "Failed to build PutObjectInput"))?;
-
-        match self.storage.put_object(put_input, session_context.credentials()).await {
-            Ok(_output) => {
-                Ok(file_size as u64) // Return the size of the uploaded object
-            }
-            Err(e) => {
+        upload::upload(Arc::clone(&self.storage), session_context, bytes, &bucket, &key)
+            .await
+            .inspect_err(|e| {
                error!(
                    event = EVENT_FTPS_OBJECT_PUT_FAILED,
                    component = LOG_COMPONENT_PROTOCOLS,
                    subsystem = LOG_SUBSYSTEM_FTPS_DRIVER,
                    username = %masked_username,
-                    path = %path_str,
-                    bucket = %bucket,
-                    object = %key,
-                    file_size,
-                    error = ?e,
+                    error = %e,
                    "ftps object put failed"
                );
-                Err(Error::new(
-                    ErrorKind::PermanentFileNotAvailable,
-                    format!("Failed to upload object: {:?}", e),
-                ))
-            }
-        }
+            })
    }

    async fn del<P: AsRef<Path> + Send>(&self, user: &super::server::FtpsUser, path: P) -> Result<()> {
@@ -927,3 +892,517 @@ mod tests {
        assert!(parse_s3_path("/bucket/__XLDIR__").is_err());
    }
 }
+
+mod upload {
+    use crate::common::{
+        client::s3::StorageBackend,
+        gateway::{S3Action, authorize_operation},
+        session::SessionContext,
+    };
+    use bytes::Bytes;
+    use s3s::dto::*;
+    use std::{
+        sync::{Arc, LazyLock},
+        time::Duration,
+    };
+    use tokio::{
+        io::{AsyncRead, AsyncReadExt},
+        sync::Semaphore,
+    };
+    use unftp_core::storage::{Error, ErrorKind, Result};
+
+    // One in-flight part per upload; never buffer the next part while the backend consumes this one.
+    const PART_SIZE: usize = 16 * 1024 * 1024;
+    const MAX_PARTS: i32 = 10_000;
+    const ABORT_TIMEOUT: Duration = Duration::from_secs(30);
+    static ABORT_PERMITS: LazyLock<Arc<Semaphore>> = LazyLock::new(|| Arc::new(Semaphore::new(32)));
+    const EVENT_FTPS_UPLOAD_CLEANUP: &str = "ftps_upload_cleanup";
+    const LOG_COMPONENT_PROTOCOLS: &str = "protocols";
+    const LOG_SUBSYSTEM_FTPS_UPLOAD: &str = "ftps_upload";
+
+    fn upload_error(error: impl std::fmt::Display) -> Error {
+        Error::new(ErrorKind::TransientFileNotAvailable, error.to_string())
+    }
+
+    async fn authorize(session: &SessionContext, action: S3Action, bucket: &str, key: &str) -> Result<()> {
+        authorize_operation(session, &action, bucket, Some(key))
+            .await
+            .map_err(|_| Error::new(ErrorKind::PermanentFileNotAvailable, "Access denied"))
+    }
+
+    fn next_part_number(completed: usize) -> Result<i32> {
+        let completed = i32::try_from(completed).map_err(upload_error)?;
+        if completed >= MAX_PARTS {
+            return Err(Error::new(
+                ErrorKind::ExceededStorageAllocationError,
+                "FTPS multipart upload exceeds 10000 parts",
+            ));
+        }
+        Ok(completed + 1)
+    }
+
+    async fn read_part(reader: &mut (impl AsyncRead + Unpin)) -> Result<Vec<u8>> {
+        let mut part = Vec::with_capacity(PART_SIZE);
+        reader
+            .take(u64::try_from(PART_SIZE).map_err(upload_error)?)
+            .read_to_end(&mut part)
+            .await
+            .map_err(upload_error)?;
+        Ok(part)
+    }
+
+    fn body(part: Vec<u8>) -> StreamingBlob {
+        StreamingBlob::from_bytes(Bytes::from(part))
+    }
+
+    // Keep the upload ID alive across all cancellable awaits after CreateMultipartUpload.
+    // A process/runtime crash still requires an AbortIncompleteMultipartUpload lifecycle rule.
+    struct PendingUpload<S: StorageBackend + 'static> {
+        storage: Arc<S>,
+        session: SessionContext,
+        input: Option<AbortMultipartUploadInput>,
+    }
+
+    impl<S: StorageBackend + 'static> PendingUpload<S> {
+        async fn abort(&mut self) {
+            if let Some(input) = self.input.as_ref() {
+                abort_upload(&*self.storage, &self.session, input.clone()).await;
+                self.input = None;
+            }
+        }
+    }
+
+    async fn abort_upload<S: StorageBackend>(storage: &S, session: &SessionContext, input: AbortMultipartUploadInput) {
+        let cleanup = async {
+            authorize(session, S3Action::AbortMultipartUpload, &input.bucket, &input.key).await?;
+            storage
+                .abort_multipart_upload(input, session.credentials())
+                .await
+                .map_err(upload_error)?;
+            Ok::<_, Error>(())
+        };
+        if !matches!(tokio::time::timeout(ABORT_TIMEOUT, cleanup).await, Ok(Ok(()))) {
+            tracing::warn!(
+                event = EVENT_FTPS_UPLOAD_CLEANUP,
+                component = LOG_COMPONENT_PROTOCOLS,
+                subsystem = LOG_SUBSYSTEM_FTPS_UPLOAD,
+                result = "abort_failed",
+                "FTPS incomplete upload requires lifecycle cleanup"
+            );
+        }
+    }
+
+    impl<S: StorageBackend + 'static> Drop for PendingUpload<S> {
+        fn drop(&mut self) {
+            let Some(input) = self.input.take() else {
+                return;
+            };
+            let (Ok(runtime), Ok(permit)) =
+                (tokio::runtime::Handle::try_current(), Arc::clone(&ABORT_PERMITS).try_acquire_owned())
+            else {
+                tracing::warn!(
+                    event = EVENT_FTPS_UPLOAD_CLEANUP,
+                    component = LOG_COMPONENT_PROTOCOLS,
+                    subsystem = LOG_SUBSYSTEM_FTPS_UPLOAD,
+                    result = "abort_unavailable",
+                    "FTPS incomplete upload requires lifecycle cleanup"
+                );
+                return;
+            };
+            let storage = Arc::clone(&self.storage);
+            let session = self.session.clone();
+            runtime.spawn(async move {
+                let _permit = permit;
+                abort_upload(&*storage, &session, input).await;
+            });
+        }
+    }
+
+    pub(super) async fn upload<S: StorageBackend + 'static>(
+        storage: Arc<S>,
+        session: &SessionContext,
+        mut reader: impl AsyncRead + Unpin,
+        bucket: &str,
+        key: &str,
+    ) -> Result<u64> {
+        let mut part = read_part(&mut reader).await?;
+        if part.len() < PART_SIZE {
+            let size = u64::try_from(part.len()).map_err(upload_error)?;
+            authorize(session, S3Action::PutObject, bucket, key).await?;
+            let input = PutObjectInput::builder()
+                .bucket(bucket.to_owned())
+                .key(key.to_owned())
+                .content_length(Some(i64::try_from(size).map_err(upload_error)?))
+                .body(Some(body(part)))
+                .build()
+                .map_err(upload_error)?;
+            storage.put_object(input, session.credentials()).await.map_err(upload_error)?;
+            return Ok(size);
+        }
+
+        authorize(session, S3Action::CreateMultipartUpload, bucket, key).await?;
+        let input = CreateMultipartUploadInput::builder()
+            .bucket(bucket.to_owned())
+            .key(key.to_owned())
+            .build()
+            .map_err(upload_error)?;
+        let output = storage
+            .create_multipart_upload(input, session.credentials())
+            .await
+            .map_err(upload_error)?;
+        let upload_id = output
+            .upload_id
+            .filter(|id| !id.is_empty())
+            .ok_or_else(|| upload_error("CreateMultipartUpload returned no upload ID"))?;
+        let abort = AbortMultipartUploadInput::builder()
+            .bucket(bucket.to_owned())
+            .key(key.to_owned())
+            .upload_id(upload_id.clone())
+            .build()
+            .map_err(upload_error)?;
+        let mut pending = PendingUpload {
+            storage: Arc::clone(&storage),
+            session: session.clone(),
+            input: Some(abort),
+        };
+
+        let result = async {
+            let mut parts = Vec::new();
+            let mut total = 0u64;
+            while !part.is_empty() {
+                let number = next_part_number(parts.len())?;
+                let length = i64::try_from(part.len()).map_err(upload_error)?;
+                total = total
+                    .checked_add(u64::try_from(part.len()).map_err(upload_error)?)
+                    .ok_or_else(|| upload_error("FTPS upload size overflow"))?;
+                authorize(session, S3Action::UploadPart, bucket, key).await?;
+                let input = UploadPartInput::builder()
+                    .bucket(bucket.to_owned())
+                    .key(key.to_owned())
+                    .upload_id(upload_id.clone())
+                    .part_number(number)
+                    .content_length(Some(length))
+                    .body(Some(body(part)))
+                    .build()
+                    .map_err(upload_error)?;
+                let output = storage
+                    .upload_part(input, session.credentials())
+                    .await
+                    .map_err(upload_error)?;
+                let etag = output.e_tag.ok_or_else(|| upload_error("UploadPart returned no ETag"))?;
+                parts.push(CompletedPart {
+                    e_tag: Some(etag),
+                    part_number: Some(number),
+                    ..Default::default()
+                });
+                part = read_part(&mut reader).await?;
+            }
+            authorize(session, S3Action::CompleteMultipartUpload, bucket, key).await?;
+            let input = CompleteMultipartUploadInput::builder()
+                .bucket(bucket.to_owned())
+                .key(key.to_owned())
+                .upload_id(upload_id)
+                .multipart_upload(Some(CompletedMultipartUpload { parts: Some(parts) }))
+                .build()
+                .map_err(upload_error)?;
+            storage
+                .complete_multipart_upload(input, session.credentials())
+                .await
+                .map_err(upload_error)?;
+            // No await between a successful commit and disarming cancellation cleanup.
+            pending.input = None;
+            Ok(total)
+        }
+        .await;
+        if result.is_err() {
+            pending.abort().await;
+        }
+        result
+    }
+
+    #[cfg(test)]
+    mod tests {
+        use super::*;
+        use crate::{
+            common::{
+                dummy_storage::{DummyBackend, DummyError},
+                gateway::{is_operation_supported, with_test_auth_override},
+                session::{Protocol, test_session},
+            },
+            ftps::{driver::FtpsDriver, server::FtpsUser},
+        };
+        use std::{
+            pin::Pin,
+            task::{Context, Poll},
+        };
+        use tokio::{io::ReadBuf, sync::Notify};
+        use unftp_core::storage::StorageBackend as _;
+
+        fn user() -> FtpsUser {
+            FtpsUser {
+                username: "upload-test".into(),
+                name: None,
+                session_context: test_session(Protocol::Ftps),
+            }
+        }
+
+        fn multipart_backend() -> DummyBackend {
+            let backend = DummyBackend::new();
+            backend.queue_create_multipart_upload_ok("upload-1");
+            backend.queue_complete_multipart_upload_ok();
+            backend
+        }
+
+        // Refuse to produce part N+1 before part N has reached the backend. The old
+        // read-to-EOF implementation fails this without allocating a huge fixture.
+        struct PacedReader {
+            backend: DummyBackend,
+            position: usize,
+            length: usize,
+            fail_at_end: bool,
+        }
+        impl AsyncRead for PacedReader {
+            fn poll_read(mut self: Pin<&mut Self>, _: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
+                if self.position / PART_SIZE > self.backend.upload_part_calls().len() {
+                    return Poll::Ready(Err(std::io::Error::other("read ahead of uploaded part")));
+                }
+                if self.position == self.length && self.fail_at_end {
+                    return Poll::Ready(Err(std::io::Error::other("interrupted source")));
+                }
+                let count = buf.remaining().min(self.length - self.position).min(8191);
+                let value = u8::try_from(self.position / PART_SIZE).expect("small test part number");
+                // Do not cross a part boundary in a single read, so each part has a distinct byte.
+                let count = count.min(PART_SIZE - self.position % PART_SIZE);
+                buf.put_slice(&vec![value; count]);
+                self.position += count;
+                Poll::Ready(Ok(()))
+            }
+        }
+
+        #[test]
+        fn ftps_part_numbers_stop_at_s3_limit() {
+            assert_eq!(next_part_number(0).expect("first part"), 1);
+            assert_eq!(next_part_number(9_999).expect("last part"), 10_000);
+            assert!(next_part_number(10_000).is_err());
+            assert!(next_part_number(usize::MAX).is_err());
+        }
+
+        #[tokio::test]
+        async fn ftps_part_buffer_does_not_grow_to_detect_eof() {
+            let mut reader = tokio::io::repeat(0).take(u64::try_from(PART_SIZE).expect("size"));
+            let part = read_part(&mut reader).await.expect("read full part");
+            assert_eq!(part.len(), PART_SIZE);
+            assert_eq!(part.capacity(), PART_SIZE);
+            assert!(read_part(&mut reader).await.expect("EOF").is_empty());
+        }
+
+        #[tokio::test]
+        async fn ftps_small_and_empty_uploads_preserve_bytes() {
+            for payload in [Vec::new(), b"small upload\x00\xff".to_vec()] {
+                let backend = DummyBackend::new();
+                backend.capture_upload_bodies();
+                let driver = FtpsDriver::new(backend.clone());
+                let result = with_test_auth_override(
+                    |_, _, _| true,
+                    driver.put(&user(), std::io::Cursor::new(payload.clone()), "/bucket/key", 0),
+                )
+                .await
+                .expect("STOR");
+                assert_eq!(result, u64::try_from(payload.len()).expect("size"));
+                assert_eq!(backend.upload_bodies(), vec![payload]);
+                assert!(backend.create_multipart_calls().is_empty());
+            }
+        }
+
+        #[tokio::test]
+        async fn ftps_upload_streams_before_eof_and_preserves_part_bytes() {
+            for length in [PART_SIZE - 1, PART_SIZE, PART_SIZE + 1, 2 * PART_SIZE, 2 * PART_SIZE + 7] {
+                let backend = multipart_backend();
+                backend.capture_upload_bodies();
+                for n in 1..=3 {
+                    backend.queue_upload_part_ok(format!("etag-{n}"));
+                }
+                let driver = FtpsDriver::new(backend.clone());
+                let reader = PacedReader {
+                    backend: backend.clone(),
+                    position: 0,
+                    length,
+                    fail_at_end: false,
+                };
+                let size = with_test_auth_override(|_, _, _| true, driver.put(&user(), reader, "/bucket/key", 0))
+                    .await
+                    .expect("bounded STOR");
+                assert_eq!(size, u64::try_from(length).expect("size"));
+                let bodies = backend.upload_bodies();
+                assert_eq!(bodies.iter().map(Vec::len).sum::<usize>(), length);
+                for (index, part) in bodies.iter().enumerate() {
+                    assert!(part.len() <= PART_SIZE);
+                    assert!(part.iter().all(|b| *b == u8::try_from(index).expect("index")));
+                }
+                if length >= PART_SIZE {
+                    assert_eq!(backend.complete_multipart_calls()[0].part_count, length.div_ceil(PART_SIZE));
+                    for (index, call) in backend.upload_part_calls().iter().enumerate() {
+                        assert_eq!(call.part_number, i32::try_from(index + 1).expect("part"));
+                        assert_eq!(call.content_length, Some(i64::try_from(bodies[index].len()).expect("length")));
+                    }
+                }
+                assert!(backend.abort_multipart_calls().is_empty());
+            }
+        }
+
+        #[tokio::test]
+        async fn ftps_upload_aborts_on_source_part_or_complete_failure() {
+            for failure in ["source", "part", "etag", "complete"] {
+                let backend = DummyBackend::new();
+                backend.queue_create_multipart_upload_ok("upload-1");
+                match failure {
+                    "part" => backend.queue_upload_part_err(DummyError::Injected("part failed".into())),
+                    "etag" => backend.queue_upload_part_ok_without_etag(),
+                    _ => backend.queue_upload_part_ok("etag-1"),
+                }
+                backend.queue_complete_multipart_upload_err(DummyError::Injected("complete failed".into()));
+                let driver = FtpsDriver::new(backend.clone());
+                let reader = PacedReader {
+                    backend: backend.clone(),
+                    position: 0,
+                    length: PART_SIZE,
+                    fail_at_end: failure == "source",
+                };
+                let result = with_test_auth_override(|_, _, _| true, driver.put(&user(), reader, "/bucket/key", 0)).await;
+                assert!(result.is_err(), "{failure} must not report a committed object");
+                assert_eq!(backend.abort_multipart_calls().len(), 1, "{failure}");
+                assert_eq!(backend.abort_multipart_calls()[0].upload_id, "upload-1");
+                assert_eq!(backend.complete_multipart_calls().len(), usize::from(failure == "complete"));
+            }
+        }
+
+        #[tokio::test]
+        async fn ftps_upload_enforces_authorization_at_each_mutation() {
+            for denied in [
+                S3Action::PutObject,
+                S3Action::CreateMultipartUpload,
+                S3Action::UploadPart,
+                S3Action::CompleteMultipartUpload,
+            ] {
+                assert!(is_operation_supported(Protocol::Ftps, &denied));
+                let backend = multipart_backend();
+                backend.queue_upload_part_ok("etag-1");
+                let driver = FtpsDriver::new(backend.clone());
+                let reader = tokio::io::repeat(0).take(u64::try_from(PART_SIZE).expect("size"));
+                let denied_action = denied.clone();
+                let result = with_test_auth_override(
+                    move |action, _, _| action != &denied_action,
+                    driver.put(&user(), reader, "/bucket/key", 0),
+                )
+                .await;
+                assert!(result.is_err());
+                match denied {
+                    S3Action::PutObject | S3Action::CreateMultipartUpload => assert!(backend.create_multipart_calls().is_empty()),
+                    S3Action::UploadPart => assert!(backend.upload_part_calls().is_empty()),
+                    _ => assert!(backend.complete_multipart_calls().is_empty()),
+                }
+            }
+        }
+
+        #[tokio::test]
+        async fn ftps_upload_cleanup_does_not_bypass_abort_permission() {
+            let backend = multipart_backend();
+            backend.queue_upload_part_err(DummyError::Injected("part failed".into()));
+            let driver = FtpsDriver::new(backend.clone());
+            let reader = tokio::io::repeat(0).take(u64::try_from(PART_SIZE).expect("size"));
+            let result = with_test_auth_override(
+                |action, _, _| action != &S3Action::AbortMultipartUpload,
+                driver.put(&user(), reader, "/bucket/key", 0),
+            )
+            .await;
+            assert!(result.is_err());
+            assert!(backend.abort_multipart_calls().is_empty());
+        }
+
+        #[tokio::test]
+        async fn ftps_cancelled_upload_aborts_pending_parts() {
+            let backend = multipart_backend();
+            let entered = Arc::new(Notify::new());
+            backend.stall_upload_part(Arc::clone(&entered));
+            let driver = FtpsDriver::new(backend.clone());
+            let user = user();
+            with_test_auth_override(|_, _, _| true, async {
+                let reader = tokio::io::repeat(0).take(u64::try_from(PART_SIZE * 2).expect("size"));
+                let mut put = Box::pin(driver.put(&user, reader, "/bucket/key", 0));
+                tokio::select! {
+                    _ = entered.notified() => {},
+                    result = &mut put => panic!("upload should stall: {result:?}"),
+                }
+                drop(put);
+                tokio::time::timeout(Duration::from_secs(5), async {
+                    while backend.abort_multipart_calls().is_empty() {
+                        tokio::task::yield_now().await;
+                    }
+                })
+                .await
+                .expect("cancellation cleanup");
+            })
+            .await;
+            assert_eq!(backend.abort_multipart_calls().len(), 1);
+            assert!(backend.complete_multipart_calls().is_empty());
+        }
+        // Yield on each read so the memory probe keeps all upload buffers live at once.
+        struct YieldingReader {
+            remaining: u64,
+            yielded: bool,
+        }
+
+        impl AsyncRead for YieldingReader {
+            fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
+                if self.remaining == 0 {
+                    return Poll::Ready(Ok(()));
+                }
+                if !self.yielded {
+                    self.yielded = true;
+                    cx.waker().wake_by_ref();
+                    return Poll::Pending;
+                }
+                self.yielded = false;
+                let count = buf
+                    .remaining()
+                    .min(1024 * 1024)
+                    .min(usize::try_from(self.remaining).unwrap_or(usize::MAX));
+                buf.initialize_unfilled_to(count).fill(0x5a);
+                buf.advance(count);
+                self.remaining -= u64::try_from(count).expect("read size");
+                Poll::Ready(Ok(()))
+            }
+        }
+
+        #[tokio::test]
+        #[ignore = "manual synthetic concurrent-upload memory measurement"]
+        async fn ftps_large_upload_memory_probe() {
+            let bytes: u64 = std::env::var("RUSTFS_FTPS_TEST_BYTES")
+                .unwrap_or_else(|_| "67108864".into())
+                .parse()
+                .expect("byte count");
+            let concurrency: usize = std::env::var("RUSTFS_FTPS_TEST_CONCURRENCY")
+                .unwrap_or_else(|_| "10".into())
+                .parse()
+                .expect("concurrency");
+            assert!(bytes > 0 && concurrency > 0);
+            let operations = (0..concurrency).map(|_| async {
+                let backend = multipart_backend();
+                for n in 0..bytes.div_ceil(u64::try_from(PART_SIZE).expect("part size")) {
+                    backend.queue_upload_part_ok(format!("part-{n}"));
+                }
+                let driver = FtpsDriver::new(backend);
+                let reader = YieldingReader {
+                    remaining: bytes,
+                    yielded: false,
+                };
+                driver.put(&user(), reader, "/bucket/key", 0).await
+            });
+            let results = with_test_auth_override(|_, _, _| true, futures_util::future::join_all(operations)).await;
+            for result in results {
+                assert_eq!(result.expect("upload"), bytes);
+            }
+            println!("synthetic_upload_bytes={bytes} concurrent_uploads={concurrency}");
+        }
+    }
+}
@@ -57,6 +57,7 @@ const EXCESSIVE_EMPTY_CHUNKS_ERROR: &str = "HTTP body returned too many empty ch
 pub const INTERNODE_DISK_ERROR_HEADER: &str = "x-rustfs-disk-error";
 pub const INTERNODE_FILE_NOT_FOUND: &str = "file-not-found";
 pub const INTERNODE_VOLUME_NOT_FOUND: &str = "volume-not-found";
+pub const INTERNODE_FILE_CORRUPT: &str = "file-corrupt";

 #[derive(Debug, Clone, Copy, Eq, PartialEq)]
 pub enum InternodeHttpErrorKind {
@@ -185,6 +186,7 @@ pub struct InternodeHttpError {
 enum RemoteDiskErrorKind {
    FileNotFound,
    VolumeNotFound,
+    FileCorrupt,
 }

 impl std::fmt::Debug for InternodeHttpError {
@@ -215,6 +217,10 @@ impl InternodeHttpError {
        self.remote_disk_error == Some(RemoteDiskErrorKind::VolumeNotFound)
    }

+    pub fn is_remote_file_corrupt(&self) -> bool {
+        self.remote_disk_error == Some(RemoteDiskErrorKind::FileCorrupt)
+    }
+
    fn new(kind: InternodeHttpErrorKind, context: InternodeHttpRequestContext) -> Self {
        Self {
            kind,
@@ -328,6 +334,20 @@ pub fn new_test_remote_volume_not_found_http_io_error() -> io::Error {
    .into_io_error()
 }

+#[doc(hidden)]
+pub fn new_test_remote_file_corrupt_http_io_error() -> io::Error {
+    InternodeHttpError::with_remote_disk_error(
+        InternodeHttpErrorKind::HttpStatus(reqwest::StatusCode::INTERNAL_SERVER_ERROR),
+        InternodeHttpRequestContext {
+            method: "GET".to_string(),
+            target: READ_FILE_STREAM_PATH.to_string(),
+            operation: Some(INTERNODE_OPERATION_READ_FILE_STREAM),
+        },
+        RemoteDiskErrorKind::FileCorrupt,
+    )
+    .into_io_error()
+}
+
 fn add_root_certificates_from_der(builder: reqwest::ClientBuilder, certs_der: &[Vec<u8>]) -> reqwest::ClientBuilder {
    let mut b = builder;
    for der in certs_der {
@@ -850,6 +870,7 @@ fn classify_http_response(
    let remote_disk_error = match headers.get(INTERNODE_DISK_ERROR_HEADER).and_then(|value| value.to_str().ok()) {
        Some(INTERNODE_FILE_NOT_FOUND) => Some(RemoteDiskErrorKind::FileNotFound),
        Some(INTERNODE_VOLUME_NOT_FOUND) => Some(RemoteDiskErrorKind::VolumeNotFound),
+        Some(INTERNODE_FILE_CORRUPT) => Some(RemoteDiskErrorKind::FileCorrupt),
        _ => None,
    };
    ClassifiedHttpResponse { kind, remote_disk_error }
@@ -2047,6 +2068,7 @@ mod tests {
        assert_eq!(INTERNODE_DISK_ERROR_HEADER, "x-rustfs-disk-error");
        assert_eq!(INTERNODE_FILE_NOT_FOUND, "file-not-found");
        assert_eq!(INTERNODE_VOLUME_NOT_FOUND, "volume-not-found");
+        assert_eq!(INTERNODE_FILE_CORRUPT, "file-corrupt");
    }

    #[test]
@@ -2090,6 +2112,48 @@ mod tests {
        assert!(wrong_operation.remote_disk_error.is_none());
    }

+    #[test]
+    fn classify_http_response_scopes_corrupt_disk_error_to_read_failures() {
+        let mut headers = HeaderMap::new();
+        headers.insert(INTERNODE_DISK_ERROR_HEADER, INTERNODE_FILE_CORRUPT.parse().expect("valid error token"));
+        let classified = classify_http_response(
+            reqwest::StatusCode::INTERNAL_SERVER_ERROR,
+            &headers,
+            Some(INTERNODE_OPERATION_READ_FILE_STREAM),
+        );
+        assert_eq!(classified.remote_disk_error, Some(RemoteDiskErrorKind::FileCorrupt));
+        for status in [reqwest::StatusCode::NOT_FOUND, reqwest::StatusCode::SERVICE_UNAVAILABLE] {
+            assert!(
+                classify_http_response(status, &headers, Some(INTERNODE_OPERATION_READ_FILE_STREAM))
+                    .remote_disk_error
+                    .is_none()
+            );
+        }
+        for operation in [
+            None,
+            Some(INTERNODE_OPERATION_WALK_DIR),
+            Some(INTERNODE_OPERATION_PUT_FILE_STREAM),
+        ] {
+            assert!(
+                classify_http_response(reqwest::StatusCode::INTERNAL_SERVER_ERROR, &headers, operation)
+                    .remote_disk_error
+                    .is_none()
+            );
+        }
+        headers.insert(INTERNODE_DISK_ERROR_HEADER, "unknown-disk-error".parse().expect("valid unknown token"));
+        for headers in [&headers, &HeaderMap::new()] {
+            assert!(
+                classify_http_response(
+                    reqwest::StatusCode::INTERNAL_SERVER_ERROR,
+                    headers,
+                    Some(INTERNODE_OPERATION_READ_FILE_STREAM),
+                )
+                .remote_disk_error
+                .is_none()
+            );
+        }
+    }
+
    #[derive(Clone, Default)]
    struct TestState {
        head_count: Arc<AtomicUsize>,
@@ -2158,6 +2222,16 @@ mod tests {
        let addr = listener.local_addr().expect("listener local address should be available");
        let app = Router::new()
            .route("/stream", get(get_stream).head(reject_head).put(accept_put))
+            .route(
+                READ_FILE_STREAM_PATH,
+                get(|| async {
+                    (
+                        StatusCode::INTERNAL_SERVER_ERROR,
+                        [(INTERNODE_DISK_ERROR_HEADER, INTERNODE_FILE_CORRUPT)],
+                        "read file err file corrupt",
+                    )
+                }),
+            )
            .route(WALK_DIR_PATH, get(get_stream))
            .route("/reject-put", get(get_stream).put(reject_put))
            .route("/stall", get(get_stalling_stream))
@@ -2172,6 +2246,44 @@ mod tests {
        Some((format!("http://{addr}/stream"), handle))
    }

+    #[tokio::test]
+    async fn http_readers_preserve_remote_file_corruption() {
+        let (url, server) = start_test_server(TestState::default())
+            .await
+            .expect("corruption regression server must bind");
+        let url = format!("{}{READ_FILE_STREAM_PATH}", url.strip_suffix("/stream").expect("test server URL suffix"));
+        let byte_error = HttpReader::new(url.clone(), Method::GET, HeaderMap::new(), None)
+            .await
+            .err()
+            .expect("byte reader must reject corrupt shard response");
+        let chunk_error = HttpChunkReader::new_with_stall_timeout(url, Method::GET, HeaderMap::new(), None, None)
+            .await
+            .err()
+            .expect("chunk reader must reject corrupt shard response");
+        for error in [byte_error, chunk_error] {
+            let source = error
+                .get_ref()
+                .and_then(|source| source.downcast_ref::<InternodeHttpError>())
+                .expect("HTTP error must retain typed internode source");
+            assert!(source.is_remote_file_corrupt());
+            assert!(!source.is_remote_file_not_found());
+            assert!(!source.is_remote_volume_not_found());
+            assert_eq!(
+                source.kind(),
+                InternodeHttpErrorKind::HttpStatus(reqwest::StatusCode::INTERNAL_SERVER_ERROR)
+            );
+            let cloned = clone_internode_http_io_error(&error).expect("typed transport error must be cloneable");
+            assert!(
+                cloned
+                    .get_ref()
+                    .and_then(|source| source.downcast_ref::<InternodeHttpError>())
+                    .expect("clone must retain typed internode source")
+                    .is_remote_file_corrupt()
+            );
+        }
+        server.abort();
+    }
+
    struct BlockedH2Server {
        url: String,
        shutdown: tokio::sync::oneshot::Sender<()>,
@@ -30,8 +30,8 @@ use crate::{
    store::{Key, Store},
    target::{
        ChannelTargetType, EntityTarget, QueuedPayload, QueuedPayloadMeta, TargetDeliveryCounters, TargetDeliverySnapshot,
-        TargetTlsState, TargetType, build_queued_payload_with_records, build_target_tls_fingerprint, is_connectivity_error,
-        open_target_queue_store, persist_queued_payload_to_store,
+        TargetTlsState, TargetType, build_queued_payload, build_queued_payload_with_records, build_target_tls_fingerprint,
+        is_connectivity_error, open_target_queue_store, persist_queued_payload_to_store,
    },
 };
 use async_trait::async_trait;
@@ -404,7 +404,10 @@ where
    }

    fn build_queued_payload(&self, event: &EntityTarget<E>) -> Result<QueuedPayload, TargetError> {
-        build_queued_payload_with_records(event, vec![event.clone()])
+        match self.args.target_type {
+            TargetType::NotifyEvent => build_queued_payload(event),
+            TargetType::AuditLog => build_queued_payload_with_records(event, vec![event.clone()]),
+        }
    }

    async fn get_or_connect(&self) -> Result<Arc<AMQPConnection>, TargetError> {
@@ -676,6 +679,7 @@ mod tests {
    use serde_json::json;
    use std::path::PathBuf;
    use std::sync::Arc;
+    use std::time::{SystemTime, UNIX_EPOCH};
    use uuid::Uuid;

    fn valid_args() -> AMQPArgs {
@@ -713,6 +717,87 @@ mod tests {
        })
    }

+    fn notification_event() -> EntityTarget<serde_json::Value> {
+        EntityTarget {
+            object_name: "incoming%2Fclip+%252F.mp4".to_string(),
+            bucket_name: "example-bucket".to_string(),
+            event_name: EventName::ObjectCreatedPut,
+            data: json!({
+                "eventVersion": "2.0",
+                "eventSource": "aws:s3",
+                "eventName": "s3:ObjectCreated:Put",
+                "s3": {
+                    "bucket": {"name": "example-bucket"},
+                    "object": {"key": "incoming%2Fclip+%252F.mp4", "size": 42,
+                               "eTag": "example-etag", "versionId": "example-version"}
+                }
+            }),
+        }
+    }
+
+    #[test]
+    fn notification_records_contain_the_event_directly() {
+        let target = AMQPTarget::new("notification".to_string(), valid_args()).unwrap();
+        let event = notification_event();
+        let queued = target.build_queued_payload(&event).unwrap();
+        let payload: serde_json::Value = serde_json::from_slice(&queued.body).unwrap();
+
+        assert_eq!(payload["Records"], json!([event.data]));
+        assert_eq!(payload["Key"], "example-bucket/incoming/clip %2F.mp4");
+        assert_eq!(payload["EventName"], "s3:ObjectCreated:Put");
+        assert_eq!(payload["Records"][0]["s3"]["object"]["key"], event.object_name);
+    }
+
+    #[test]
+    fn audit_records_preserve_the_entity_envelope() {
+        let mut args = valid_args();
+        args.target_type = TargetType::AuditLog;
+        let target = AMQPTarget::new("audit".to_string(), args).unwrap();
+        let event = test_event();
+        let queued = target.build_queued_payload(&event).unwrap();
+        let payload: serde_json::Value = serde_json::from_slice(&queued.body).unwrap();
+
+        assert_eq!(payload["Records"], json!([event.as_ref()]));
+    }
+
+    #[tokio::test]
+    async fn queued_notification_preserves_the_flat_record_and_metadata() {
+        let mut args = unreachable_args();
+        args.queue_dir = temp_store_dir("record-shape").to_string_lossy().to_string();
+        let target = AMQPTarget::new("notification".to_string(), args.clone()).unwrap();
+        let event = notification_event();
+        let expected = target.build_queued_payload(&event).unwrap();
+        let unix_time_ms = || {
+            u64::try_from(
+                SystemTime::now()
+                    .duration_since(UNIX_EPOCH)
+                    .expect("system clock should be after the Unix epoch")
+                    .as_millis(),
+            )
+            .expect("current Unix timestamp should fit in u64")
+        };
+        let before_save = unix_time_ms();
+        target.save(Arc::new(event.clone())).await.unwrap();
+        let after_save = unix_time_ms();
+        let store = target.store().unwrap();
+        let keys = store.list();
+        assert_eq!(keys.len(), 1);
+        let raw = store.get_raw(&keys[0]).unwrap();
+        let queued = QueuedPayload::decode(&raw).unwrap();
+        assert_eq!(queued.body, expected.body);
+        assert_eq!(queued.meta.event_name, expected.meta.event_name);
+        assert_eq!(queued.meta.bucket_name, expected.meta.bucket_name);
+        assert_eq!(queued.meta.object_name, expected.meta.object_name);
+        assert_eq!(queued.meta.content_type, expected.meta.content_type);
+        assert_eq!(queued.meta.payload_len, expected.meta.payload_len);
+        assert_eq!(queued.meta.dedup_id, expected.meta.dedup_id);
+        assert_eq!(queued.meta.failure, expected.meta.failure);
+        assert!((before_save..=after_save).contains(&queued.meta.queued_at_unix_ms));
+        let payload: serde_json::Value = serde_json::from_slice(&queued.body).unwrap();
+        assert_eq!(payload["Records"], json!([event.data]));
+        std::fs::remove_dir_all(args.queue_dir).unwrap();
+    }
+
    fn temp_store_dir(name: &str) -> PathBuf {
        std::env::temp_dir().join(format!("rustfs-amqp-target-{name}-{}", Uuid::new_v4()))
    }
@@ -32,9 +32,9 @@ use lapin::{
 use rustfs_s3_types::EventName;
 use rustfs_targets::Target;
 use rustfs_targets::check_amqp_broker_available;
-use rustfs_targets::target::EntityTarget;
 use rustfs_targets::target::TargetType;
 use rustfs_targets::target::amqp::{AMQPArgs, AMQPTarget};
+use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
 use serde_json::Value;
 use std::sync::Arc;
 use uuid::Uuid;
@@ -67,7 +67,12 @@ fn entity_for(bucket: &str, object: &str) -> Arc<EntityTarget<serde_json::Value>
        bucket_name: bucket.to_string(),
        object_name: object.to_string(),
        event_name: EventName::ObjectCreatedPut,
-        data: serde_json::json!({"bucket": bucket, "object": object}),
+        data: serde_json::json!({
+            "eventVersion": "2.0",
+            "eventSource": "aws:s3",
+            "eventName": "s3:ObjectCreated:Put",
+            "s3": {"bucket": {"name": bucket}, "object": {"key": object, "size": 42}}
+        }),
    })
 }

@@ -102,7 +107,7 @@ async fn bind_queue(queue: &str, routing_key: &str) -> lapin::Channel {
    channel
 }

-async fn read_one(channel: &lapin::Channel, queue: &str) -> (Value, BasicProperties) {
+async fn read_one_raw(channel: &lapin::Channel, queue: &str) -> (Vec<u8>, BasicProperties) {
    let msg = tokio::time::timeout(std::time::Duration::from_secs(5), async {
        loop {
            if let Some(msg) = channel
@@ -120,8 +125,12 @@ async fn read_one(channel: &lapin::Channel, queue: &str) -> (Value, BasicPropert
    .expect("message should arrive");

    let properties = msg.properties.clone();
-    let payload = serde_json::from_slice(&msg.data).expect("message payload should be JSON");
-    (payload, properties)
+    (msg.data.clone(), properties)
+}
+
+async fn read_one(channel: &lapin::Channel, queue: &str) -> (Value, BasicProperties) {
+    let (body, properties) = read_one_raw(channel, queue).await;
+    (serde_json::from_slice(&body).expect("message payload should be JSON"), properties)
 }

 #[tokio::test]
@@ -147,7 +156,7 @@ async fn test_direct_publish_delivers_json_payload() {

    let (payload, properties) = read_one(&channel, &queue).await;
    assert_eq!(payload["Key"], "bucket1/object-A");
-    assert_eq!(payload["Records"][0]["data"]["bucket"], "bucket1");
+    assert_eq!(payload["Records"], serde_json::json!([entity_for("bucket1", "object-A").data]));
    assert_eq!(properties.content_type().as_ref().map(|s| s.as_str()), Some("application/json"));
    assert_eq!(*properties.delivery_mode(), Some(2));

@@ -209,6 +218,7 @@ async fn test_queue_replay_delivers_and_removes_stored_payload() {

    let (payload, properties) = read_one(&channel, &queue).await;
    assert_eq!(payload["Key"], "bucket1/object-B");
+    assert_eq!(payload["Records"], serde_json::json!([entity_for("bucket1", "object-B").data]));
    assert_eq!(properties.content_type().as_ref().map(|s| s.as_str()), Some("application/json"));
    assert_eq!(*properties.delivery_mode(), Some(2));
    assert_eq!(target.delivery_snapshot().queue_length, 0);
@@ -219,3 +229,42 @@ async fn test_queue_replay_delivers_and_removes_stored_payload() {
        .expect("delete queue");
    let _ = std::fs::remove_dir_all(args.queue_dir);
 }
+
+#[tokio::test]
+#[ignore = "requires running RabbitMQ-compatible AMQP broker"]
+async fn test_legacy_queued_notification_replays_original_bytes() {
+    let routing_key = format!("rustfs.legacy.{}", Uuid::new_v4().simple());
+    let queue = format!("rustfs-test-{}", Uuid::new_v4().simple());
+    let channel = bind_queue(&queue, &routing_key).await;
+    let queue_dir = std::env::temp_dir().join(format!("rustfs-amqp-legacy-{}", Uuid::new_v4()));
+    let mut args = test_args(&routing_key);
+    args.queue_dir = queue_dir.to_string_lossy().to_string();
+    let target = AMQPTarget::<Value>::new("legacy".to_string(), args).expect("construct AMQP target");
+    let event = entity_for("example-bucket", "old-object");
+    // Model the pre-upgrade envelope, including whitespace that reserialization would change.
+    let body = serde_json::to_vec_pretty(&serde_json::json!({
+        "EventName": event.event_name,
+        "Key": "example-bucket/old-object",
+        "Records": [event.as_ref()]
+    }))
+    .unwrap();
+    let meta = QueuedPayloadMeta::new(
+        event.event_name,
+        event.bucket_name.clone(),
+        event.object_name.clone(),
+        "application/json",
+        body.len(),
+    );
+    let encoded = QueuedPayload::new(meta, body.clone()).encode().unwrap();
+    let store = target.store().expect("store configured");
+    let key = store.put_raw(&encoded).expect("persist pre-upgrade payload");
+    target.send_from_store(key).await.expect("replay legacy payload");
+    let (received, _) = read_one_raw(&channel, &queue).await;
+    assert_eq!(received, body);
+    assert!(store.list().is_empty());
+    channel
+        .queue_delete(queue.into(), QueueDeleteOptions::default())
+        .await
+        .expect("delete queue");
+    std::fs::remove_dir_all(queue_dir).unwrap();
+}
@@ -39,7 +39,7 @@ allow-git = [
    # Official s3s repository. Temporarily pinned to the merged generic REST
    # SigV4 payload-checksum fix until it is available in a crates.io release.
    # owner: marshawcoco  review: 2026-10
-    # "https://github.com/s3s-project/s3s.git",
+    "https://github.com/s3s-project/s3s.git",
    # RustFS fork carrying presigned expiry and constant-time authentication fixes.
    # owner: rustfs-maintainers  review: 2026-10
    # "https://github.com/rustfs/s3s.git",
@@ -8,6 +8,8 @@ collection:

 ## Operations

+For legacy protection assessment and bounded protected copies, see [Object integrity inventory, audit, and migration](operations/shard-integrity-audit.md).
+
 Operational runbooks live under [`operations/`](operations/). Replication
 operators should start with:

@@ -92,7 +92,21 @@ When decommission metadata is present, `decommissionInfo` includes:
 - progress counters: `objectsDecommissioned`, `objectsDecommissionedFailed`, `bytesDecommissioned`, and `bytesDecommissionedFailed`;
 - current location: `bucket`, `prefix`, and `object`;
 - queue/history lists: `queuedBuckets` and `decommissionedBuckets`;
- `waitingReason`: `queued` for queued entries and `waiting_for_worker` when metadata exists but no worker has started.
+- `waitingReason`: `capacity` while the pool is paused on target capacity,
+  `queued` for queued entries, and `waiting_for_worker` when metadata exists but
+  no worker has started;
+- `capacityBlockedReason`: the persisted detail for the active capacity pause,
+  absent once the pause clears.
+
+A capacity pause is reported ahead of the worker states so operators can tell
+"contending but progressing" from "genuinely out of target capacity". The existing `pool.bin` layout is unchanged;
+cumulative pause history requires a separately versioned persistence contract.
+
+The object-attempt metrics count entry passes and repeated version-copy attempts
+within a listing and its deferred replay. Inline gate waits use the gate-retry
+counter. A new listing after a durable pause starts a new per-object observation;
+the maximum is the highest observation in the process, not a persisted lifetime
+attempt count.

 This makes queued pools and stalled metadata visible without requiring operators to inspect pool metadata files directly.

@@ -12,7 +12,7 @@ The broad `rustfs_ecstore::api` facade is a compatibility boundary, not an archi
 | `storage`, `layout`, `error`, `runtime`, `cluster`, `rpc` | Compatibility spine for storage, topology, runtime handles, cluster control, and internode calls. | Keep until replacement contracts compile in downstream boundary files. |
 | `bucket` | Domain facade consumed through owner-local `storage_api` boundaries; explicit submodules and symbol lists, never whole bucket owner modules. | Keep lists aligned with boundary consumers; never restore whole-module passthroughs. |
 | `config`, `disk`, `tier` | Compatibility paths with explicit nested submodules and symbol lists. | Same as `bucket`. |
-| `data_usage`, `capacity`, `notification`, `metrics`, `rebalance` | Domain and service facades consumed through owner-local boundaries. | Narrow one group at a time after explicit aliases or wrappers exist. |
+| `data_usage`, `capacity`, `notification`, `metrics`, `rebalance`, `integrity` | Domain and service facades consumed through owner-local boundaries. | Narrow one group at a time after explicit aliases or wrappers exist. |
 | `set_disk`, `object`, `object_api_utils`, `rio`, `bitrot`, `erasure`, `compression`, `cache`, `store_list` | Low-level object IO, reader, erasure, cache, and migration helper compatibility. | Keep stable while `SetDisks` remains the shared state carrier. |
 | `admin`, `event`, `global` | Admin, event hook, and bootstrap-global compatibility. | `global` is limited to bootstrap writes and lifecycle controls; read-only runtime access goes through `runtime`. |

@@ -28,7 +28,7 @@ External `rustfs_ecstore::api` imports stay in these local boundary files:

 | Boundary file | Facade families consumed |
 |---|---|
-| `rustfs/src/storage/storage_api.rs` | Broad storage-owner bridge: admin, bucket submodules, capacity, compression, cluster, config, data usage, disk, error, event, global bootstrap controls, runtime getters, layout, metrics, notification, rebalance, rio, rpc, set disk, storage, tier. Replication pool/stat handles are projected into RustFS-local wrapper types here. |
+| `rustfs/src/storage/storage_api.rs` | Broad storage-owner bridge: admin, bucket submodules, capacity, compression, cluster, config, data usage, disk, error, event, global bootstrap controls, runtime getters, layout, metrics, notification, rebalance, integrity, rio, rpc, set disk, storage, tier. Replication pool/stat handles are projected into RustFS-local wrapper types here. |
 | `rustfs/src/storage_api.rs`, `rustfs/src/admin/storage_api.rs`, `rustfs/src/app/storage_api.rs` | Root, admin, and app owner boundaries: explicit aliases only, no `metadata`, `metadata_sys`, `quota`, `com`, or bare `init` module passthroughs; object and error aliases anchor on storage-api associated types and a local `StorageError`. |
 | `crates/scanner/src/storage_api.rs` | Bucket lifecycle, replication, metadata, capacity, config, data usage, disk, error, runtime, set disk, storage, tier. Replication queue config, admission, and heal object DTOs are projected into scanner-local types. |
 | `crates/obs/src/metrics/storage_api.rs` | Bucket bandwidth, lifecycle, replication, quota, capacity, data usage, error, runtime, storage; data usage is consumed as a local DTO projection. |
@@ -41,7 +41,7 @@ Every route asserted in `rustfs/src/admin/route_registration_test.rs` is registe
 | Pools decommission / cancel / clear | 部分兼容 | `rustfs/src/admin/handlers/pools.rs` returns `NotImplemented` when endpoints are not initialized (single-pool or uninitialized clusters). |
 | `/v3/top/drives`, `/v3/top/net` | 缺失 | Only `/v3/top/locks` is registered (`rustfs/src/admin/handlers/diagnostics.rs`). |
 | Bucket / site replication per-object diff | 缺失 | `/v3/replicationmetrics` and site-replication status exist; no diff endpoint. |
-| MRF (most-recent-failures) replication metrics breakdown | 缺失 | Only the generic `/v3/metrics` stream and replication metrics wire (`rustfs/src/admin/replication_metrics_wire.rs`). |
+| MRF (most-recent-failures) replication metrics breakdown | 缺失 | Only the generic `/v3/realtime` stream and replication metrics wire (`rustfs/src/admin/replication_metrics_wire.rs`). |

 Formerly-missing families that are now registered and therefore not exceptions: `/v3/healthinfo`, `/v3/obdinfo`, `/v3/force-unlock`, `/v3/top/locks`, `/v3/speedtest*`, `/v3/log`, `/v3/trace`, `/v3/profile`, `/v3/profiling/*`, `/v3/idp/{ldap|openid}/*`, `/v3/idp-config/*`.

@@ -0,0 +1,51 @@
+# AMQP notification record layout
+
+New AMQP bucket notifications place the S3 event directly in `Records`, matching
+Kafka and webhook notification targets. For example, a consumer reads the bucket
+from `Records[0].s3.bucket.name` and the object key from
+`Records[0].s3.object.key`. Previously these fields were nested under
+`Records[0].data` in an internal target envelope.
+
+This follows the record layout described in the
+[AWS S3 notification message structure](https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html).
+It changes the record envelope only; the event's existing fields and values are
+preserved. The outer `EventName` and `Key` fields are retained. The outer `Key`
+decodes the object name once, while `Records[0].s3.object.key` retains the event's
+encoded key. AMQP audit log payloads retain their existing envelope.
+
+## Upgrading consumers
+
+Consumers can read event fields at the standard S3 record paths. Consumers built
+around the previous AMQP envelope must be updated before upgrading producers. During rolling upgrades or queue draining, consumers
+may receive both layouts. A temporary compatibility path can unwrap `record.data`
+when present and otherwise use `record` directly.
+
+Existing disk-queued messages are replayed byte for byte. Upgrading does not
+rewrite their bodies. Newly queued notifications use the same flat record as
+direct publishing. Retain dual-layout handling until old producers are upgraded
+and old queued messages have drained. If rolling back a producer, retain that
+handling while any flat messages remain queued or in the broker.
+
+## Verification
+
+The regression `notification_records_contain_the_event_directly` fails against
+the old serializer because `Records[0]` contains the target envelope. It passes
+with the notification serializer using the shared event-record builder. Unit
+tests also cover encoded keys, preserved metadata in the queue, and the unchanged
+audit envelope:
+
+```bash
+cargo test -p rustfs-targets --lib target::amqp::tests
+```
+
+Broker integration tests exercise direct publishing, disk-queue replay, reconnect,
+and byte-for-byte replay of a pre-upgrade envelope. They use synthetic events and
+require an isolated RabbitMQ-compatible broker:
+
+```bash
+RUSTFS_TEST_AMQP_URL='amqp://guest:guest@127.0.0.1:5672/%2f' \
+  cargo test -p rustfs-targets --test amqp_integration -- --ignored
+```
+
+These checks cover the target's serialization and delivery boundary. They do not
+exercise the full bucket-notification pipeline or every downstream S3 client.
@@ -0,0 +1,25 @@
+# FTPS uploads
+
+FTPS `STOR` buffers at most one 16 MiB payload chunk per active upload in the
+protocol driver. It waits for the storage backend to consume each chunk before
+reading the next. Backend, TLS, connection, and allocator overhead are additional;
+this is not a process-wide memory limit. Concurrent uploads each have their own
+buffer.
+
+Files smaller than 16 MiB, including empty files, use `PutObject`. Files at or
+above that threshold use sequential S3 multipart uploads, so they can exceed the
+5 GiB single-`PutObject` limit. Multipart ETags differ from single-PUT ETags and
+must not be interpreted as a whole-file MD5 checksum. An upload is successful only
+after the multipart completion succeeds. The fixed part size and S3's 10,000-part
+limit allow up to 156.25 GiB per FTPS upload; a larger input fails and cleanup is
+attempted. Resuming or appending with a nonzero offset remains unsupported.
+
+Each write operation requires `s3:PutObject`. Failed or cancelled transfers also
+attempt `s3:AbortMultipartUpload` using the authenticated user's permissions;
+cleanup does not bypass IAM. Grant that permission on the upload prefix to allow
+immediate cleanup. Cancellation cleanup has a bounded task count and timeout.
+Configure an `AbortIncompleteMultipartUpload` bucket lifecycle rule as a fallback
+for denied/failed cleanup, process crashes, or losing the upload ID while upload
+initiation is in flight. Before completion, received parts do not replace an existing completed object.
+A lost or failed completion response may have an ambiguous outcome; clients
+should verify the destination before retrying.
@@ -11,7 +11,7 @@ Every stage is env-gated, so "before" and "after" run the *same binary* with dif
 | --- | --- |
 | RPC secret | Internode RPC fails closed: remote endpoints with default credentials and no `RUSTFS_RPC_SECRET` abort startup with `store init aborted: endpoints include remote nodes but ...` (`crates/ecstore/src/store/init.rs`). Set a non-default `RUSTFS_RPC_SECRET`, identical on every node. |
 | systemd start timeout | `deploy/build/rustfs.service` is `Type=notify` and ships `TimeoutStartSec=120s`; READY fires only after quorum. If freshly purged disks need longer, raise it in a drop-in rather than lowering it. |
-| Metrics export | RustFS has no Prometheus pull endpoint (`/admin/v3/metrics` is NDJSON, see `rustfs/src/admin/handlers/metrics.rs`); it pushes OTLP. Run an otel-collector (OTLP receiver → Prometheus exporter) and set `RUSTFS_OBS_ENDPOINT`, `RUSTFS_OBS_METRICS_EXPORT_ENABLED=true`, `RUSTFS_OBS_METER_INTERVAL=5`. For lock p99 also set `RUSTFS_OBJECT_LOCK_DIAG_ENABLE=true` (default off). |
+| Metrics export | RustFS has no Prometheus pull endpoint (`/admin/v3/realtime` is NDJSON, see `rustfs/src/admin/handlers/realtime.rs`); it pushes OTLP. Run an otel-collector (OTLP receiver → Prometheus exporter) and set `RUSTFS_OBS_ENDPOINT`, `RUSTFS_OBS_METRICS_EXPORT_ENABLED=true`, `RUSTFS_OBS_METER_INTERVAL=5`. For lock p99 also set `RUSTFS_OBJECT_LOCK_DIAG_ENABLE=true` (default off). |
 | Server env | `RUSTFS_INTERNODE_*` are **server** env. For p0/p1/p2, source the emitted `server-env.sh` on every node and restart before the run; the driver cannot mutate a running server. |

 ## Driver
@@ -20,7 +20,7 @@ sum by (instance) (rate(rustfs_s3_http_requests_total{method="PUT",outcome=~"[1-

 Inspect `service_error` and `cancelled` separately; neither implies a received HTTP status. A zero denominator or absent series means no observed traffic, not proof of health. Use `rate` or reset-aware deltas because counters restart with the process. The older `rustfs_s3_operations_total` counter measures handler entries and excludes pre-dispatch rejections; it is not this HTTP denominator.

-The authenticated admin metrics endpoint exposes the same counters through the optional `http` field in `aggregated` and `by_host`. Request `/rustfs/admin/v3/metrics?types=512&by-host=true&n=1` on each node for HTTP-only data. The default type selection also includes HTTP outcomes. This endpoint remains an NDJSON stream and does not become a cluster-wide peer fanout. `http.requests` contains `method`, `operation`, `outcome`, and `total`; `http.collected` timestamps collection. Compare consecutive samples from the same host. Concurrent snapshots are not atomic across series.
+The authenticated admin metrics endpoint exposes the same counters through the optional `http` field in `aggregated` and `by_host`. Request `/rustfs/admin/v3/realtime?types=512&by-host=true&n=1` on each node for HTTP-only data. The default type selection also includes HTTP outcomes. This endpoint remains an NDJSON stream and does not become a cluster-wide peer fanout. `http.requests` contains `method`, `operation`, `outcome`, and `total`; `http.collected` timestamps collection. Compare consecutive samples from the same host. Concurrent snapshots are not atomic across series.

 A missing `http` field means an older or non-reporting node, not zero failures. Old map-encoded RPC readers ignore the additive field; new readers accept older snapshots. In mixed-version deployments, check reporting coverage before aggregating a fleet-wide ratio.

@@ -317,7 +317,7 @@ Use the harness around a post-start bucket creation workload to cover the timing
 1. Start RustFS from an empty data path.
 2. Start the harness before creating buckets.
 3. Create a bucket, upload objects, and keep the harness running until at least one usage save is observed.
-4. Compare `scanner-summary.csv` with `/rustfs/admin/v3/metrics?types=1&n=1` bucket metrics.
+4. Compare `scanner-summary.csv` with `/rustfs/admin/v3/realtime?types=1&n=1` bucket metrics.

 Expected evidence: dirty usage is marked, `life_time_scan_cycle` or `life_time_scan_bucket_drive` advances, `life_time_scan_object` advances for object workloads, and `life_time_save_usage` plus `usage_last_save_result=success` appear before non-zero bucket usage metrics are accepted as fresh.

@@ -481,7 +481,7 @@ Read this snapshot before changing scanner controls: `blocked_source` with `life

 ## Reading Distributed Metrics

-`/rustfs/admin/v3/scanner/status` and `/rustfs/admin/v3/metrics` report the node that handles the HTTP request; the metrics endpoint does not fan out to peers. In distributed deployments, query every node explicitly and keep `by-host=true` so each response includes that node's host view:
+`/rustfs/admin/v3/scanner/status` and `/rustfs/admin/v3/realtime` report the node that handles the HTTP request; the metrics endpoint does not fan out to peers. In distributed deployments, query every node explicitly and keep `by-host=true` so each response includes that node's host view:

 ```bash
 for endpoint in http://node-a:9000 http://node-b:9000 http://node-c:9000; do
@@ -493,7 +493,7 @@ for endpoint in http://node-a:9000 http://node-b:9000 http://node-c:9000; do
    --access_key "$RUSTFS_ACCESS_KEY" \
    --secret_key "$RUSTFS_SECRET_KEY" \
    --request GET \
-    "${endpoint}/rustfs/admin/v3/metrics?types=1&by-host=true&n=1" \
+    "${endpoint}/rustfs/admin/v3/realtime?types=1&by-host=true&n=1" \
    > "artifacts/scanner-metrics.${node}.$(date -u +%Y%m%dT%H%M%SZ).ndjson"
 done
 ```
@@ -0,0 +1,72 @@
+# Object integrity inventory, audit, and migration
+
+**Use this when:** assessing legacy object protection, checking a bounded manifest against an existing SHA256, or creating protected copies without overwriting the source.
+
+## Protection and activation
+
+Legacy shard-local checksums can detect accidental corruption, but do not independently bind a complete valid donor shard to its original object. A successful legacy read or Heal is not proof of original content. Inventory reports the stored protection declaration; it does not verify payload bytes. `independent_commitment` requires consistent object and part declarations. `unknown` means metadata could not be observed reliably.
+
+Both `RUSTFS_SHARD_INTEGRITY_WRITE` and `RUSTFS_SHARD_INTEGRITY_FLEET_CONFIRMED` remain false by default. Enabling both permits protected new writes on that process. The second flag is an operator attestation, not a peer capability probe or a storage fencing mechanism. Before enabling, qualify all readers, writers, repair processes, rollback binaries, and replacement nodes against the protected format. Prevent old processes from accessing those drives. Measure full GET, Range GET, PUT, degraded read, and repair latency and throughput on the deployment's erasure layout. The readiness endpoint reports these requirements as unverified; it does not make the deployment safe automatically.
+
+Disabling the write flags stops protection for new writes without an inherited mode; it does not remove existing commitments or make an old binary safe. Keep compatible readers for already migrated objects. Normal S3 requests and existing legacy reads do not depend on the integrity job worker being available.
+
+## Administrative API
+
+Use an authenticated, signed admin client. Paths use `/rustfs/admin/v3/integrity` and normal RustFS admin request authentication and body conventions.
+
+| Method and suffix | Permission | Behavior |
+|---|---|---|
+| `GET /readiness` | `admin:ServerInfo` | Local support, local flags, operator attestation, and unverified fleet requirements |
+| `GET /{bucket}/inventory` | `admin:InspectData` | One page of version observations, no payload scan |
+| `POST /{bucket}/jobs` | `admin:StartBatchJob` | Persist a paused manifest and return its ID |
+| `GET /{bucket}/jobs/{job_id}` | `admin:DescribeBatchJob` | Durable status and per-item outcomes |
+| `POST /{bucket}/jobs/{job_id}/control` | `admin:StartBatchJob` | Explicit `resume`, `pause`, or `cancel` |
+
+Inventory accepts `prefix`, `key-marker`, `version-marker`, and `limit` (1–100). Continue with both returned markers only after storing the entire page. Each inventory item returns a reusable string version selector, including `"null"` for the null version. Pages are observations, not a bucket snapshot: concurrent creates and deletes can affect enumeration. A recreated bucket has a different incarnation; do not combine observations across incarnations. Inventory and audit suppress their own read-repair submissions and bypass the body cache for content verification; they do not stop unrelated normal reads or background maintenance.
+
+Example job body:
+
+```json
+{
+  "mode": "audit",
+  "items": [
+    {
+      "key": "archive/object.bin",
+      "version_id": null,
+      "expected_sha256": "ungWv48Bz+pBQUDeXa4iI7ADYaOWF3qctBD/YfIAFa0="
+    }
+  ],
+  "bytes_per_second": 10485760,
+  "max_object_bytes": 1073741824
+}
+```
+
+The example digest is illustrative; supply the expected digest of your own object from an independent trusted record. `version_id` omitted or JSON null selects the current version; the string `"null"` selects the null version. A non-null version must be a UUID. The job records and rechecks the observed source identity. For `mode: "migrate"`, each item also requires a distinct `target_key` in the same bucket. No target may equal any source key in the manifest.
+
+`expected_sha256` is canonical Base64 of 32 bytes, not hex or an ETag. When omitted, only a canonical stored single-part SHA256 is eligible. A stored checksum verifies consistency with stored metadata; it does not establish an external historical provenance. The service never derives a new expected digest from an unchecked read and calls that historical evidence.
+
+Start the persisted job with:
+
+```json
+{"operation":"resume"}
+```
+
+Use the same control endpoint with `pause` or `cancel`. Poll status by ID and retain the ID externally; there is no job-list endpoint.
+
+## Supported scope and resource limits
+
+Audit accepts local, untransformed single-part objects with an expected SHA256. Multipart, encrypted, compressed, tiered objects, delete markers, and invalid protection declarations are reported as unsupported. Migration additionally requires a plain unversioned bucket without configured encryption, Object Lock, replication, notification, quota, lifecycle, table namespace, ACL, access logging, or on-demand migration. Disk commits continue to use the bucket’s existing durability policy. These checks hold the bucket configuration fence through publication so the direct storage writer cannot bypass those policies during a race.
+
+Each manifest contains 1–64 items. `max_object_bytes` is 1 byte–5 GiB and limits the anonymous local temporary file used for one object. Ensure that amount of space is available in the process temporary directory. One cluster-wide worker runs at a time. `bytes_per_second` is 64 KiB/s–1 GiB/s and bounds each sequential source read, destination PUT stream, and verification read. It is not a physical disk or network aggregate ceiling: erasure redundancy and internal buffering still apply. Metadata requests are not byte throttled. Lower rates extend the time a source read or destination publication holds its normal object and bucket configuration locks; choose the rate and per-object budget with concurrent traffic in mind.
+
+Migration uploads the same staged bytes that passed SHA256 verification, rechecks that checksum during PUT, and publishes with a create-only precondition. It preserves user metadata, content type/encoding/language, disposition, cache control, expiration, storage class, and tags. The new key has a new modification time and storage identity; consumers switch keys separately. The source is neither overwritten nor deleted. No original-object metadata is backfilled in place.
+
+## Recovery and interpretation
+
+`complete` means every item reached a final outcome, not that every item passed. Inspect `verified`, `migrated`, `mismatch`, `unsupported`, `stale`, and `conflict` individually. A source that is temporarily unavailable leaves the job `failed`; `resume` retries unavailable or prepared items while preserving completed results. A mismatch never publishes a target.
+
+Pause and cancel first persist intent. An in-flight conditional target publication may finish; terminal acknowledgement follows after the worker stops. Cancellation does not delete a successfully published target. After a coordinator restart, `running` is the last durable state and does not prove a worker is alive. Explicitly resume to reacquire the distributed worker lock and recover. A persisted cancel intent is completed rather than restarted. A busy response means another worker still owns the lock; retry later.
+
+A prepared checkpoint precedes destination publication. If the PUT acknowledgement or completion checkpoint is lost, resume first checks the destination's internal operation identity, protection declarations, size, and uncached full-content SHA256. Only a matching receipt is accepted as this job's completed copy. An unrelated destination is a conflict and is never overwritten. Source changes before verification finishes produce `stale`; create a new manifest after investigating the change. Bucket deletion/recreation invalidates the old job incarnation.
+
+This workflow does not automatically repair damaged historical content, certify all historical shards, enable protected writes fleet-wide, clean MRF queues, or authorize rolling back to an incompatible binary.
@@ -108,7 +108,7 @@ Post-merge and tag-driven; not a substitute for a PR gate.
 |---|---|---|
 | Push to `main`, weekly schedule, dispatch | `build.yml` `build-rustfs` (a development build on a main push restricts the matrix to the Linux targets) | build artifacts; no release publication |
 | Valid release or preview tag | `build.yml` `build-rustfs`, `create-release`, `upload-release-assets`, `publish-release` | draft release, checksummed assets, publish |
-| Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `build-docker`, `scan-docker-image` | multi-architecture images and vulnerability report |
+| Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `build-docker`, `scan-docker-image`, `sync-dockerhub-description` | multi-architecture images, vulnerability report, and the Docker Hub overview republished from `README.md` |
 | Successful release-tag build (`workflow_run`) | `package.yml` `package` | DEB/RPM packages and checksums uploaded to the release |
 | Successful non-preview release-tag build (`workflow_run`) | `helm-package.yml` `build-helm-package`, `publish-helm-package` | versioned chart and repository index |
 | Final tag's release published | `build.yml` `cleanup-preview-releases` | deletes every `<target>-preview.<N>` Release for that target; the tags are kept |
@@ -0,0 +1,71 @@
+# FTPS upload memory regression
+
+The old `FtpsDriver::put` copied the entire input into a `Vec` before calling
+`PutObject`. Wrapping that completed allocation in `StreamingBlob` did not make
+input consumption streaming. It also sent large files through the 5 GiB
+single-PUT path.
+
+## Automated regression
+
+```sh
+cargo test -p rustfs-protocols --no-default-features --features ftps --lib
+```
+
+`ftps_upload_streams_before_eof_and_preserves_part_bytes` uses a reader that
+refuses to supply the next part until the previous part reaches the backend.
+Restoring the driver from commit `9c30cc88513c5e1b5443b6ee86bf4bc7965e79e9`
+while retaining the new test harness makes this test fail with
+`read ahead of uploaded part`. The multipart implementation passes. Tests also
+check exact bytes, empty files, part boundaries, part numbering, errors,
+permissions, cancellation cleanup, and buffer capacity.
+
+## Manual memory probe
+
+The ignored test `ftps_large_upload_memory_probe` creates concurrent synthetic
+readers. Each reader yields between reads so all uploads are active together.
+It calls the real FTPS storage driver with a scripted storage backend that
+returns successful S3 responses and discards the payload. It neither stores
+files nor opens network connections. No large fixture is required.
+
+Build the test executable first; measuring Cargo would include compiler memory:
+
+```sh
+export CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0
+cargo test -p rustfs-protocols --no-default-features --features ftps --lib --no-run
+```
+
+Use the executable path printed by that command as `test_binary`, then on macOS:
+
+```sh
+/usr/bin/time -l "$test_binary" --ignored --exact \
+  ftps::driver::upload::tests::ftps_large_upload_memory_probe --nocapture
+
+RUSTFS_FTPS_TEST_BYTES=10737418240 RUSTFS_FTPS_TEST_CONCURRENCY=10 \
+  /usr/bin/time -l "$test_binary" --ignored --exact \
+  ftps::driver::upload::tests::ftps_large_upload_memory_probe --nocapture
+```
+
+On Linux use `/usr/bin/time -v` (its maximum RSS is reported in KiB). The default
+probe uses 64 MiB per upload and 10 uploads. These environment variables are
+**test-only**, not server configuration.
+
+Observed on macOS arm64 with Rust 1.98.1, debug information disabled, one run per
+case (maximum resident set size reported in bytes):
+
+| Driver | Bytes per upload | Concurrent uploads | Maximum RSS |
+| --- | ---: | ---: | ---: |
+| Original driver at `9c30cc8`, same probe | 67,108,864 | 10 | 710,033,408 (677.1 MiB) |
+| Bounded multipart driver | 67,108,864 | 10 | 179,781,632 (171.5 MiB) |
+| Bounded multipart driver | 10,737,418,240 | 10 | 187,465,728 (178.8 MiB) |
+
+For the original-driver comparison, only `ftps/driver.rs` was replaced with its
+base-commit version in the test checkout; the identical probe and dummy backend
+were retained, the executable rebuilt, and the 64 MiB case rerun. The original
+driver was not subjected to the 100 GiB aggregate case.
+
+These measurements isolate protocol-driver buffering, not total server RSS,
+persisted-data correctness, TLS behavior, or storage throughput. The large case
+supplies 100 GiB of generated bytes without retaining them. A real server also
+uses memory for TLS, storage, caches, and allocator overhead. Memory still scales
+with active upload count. See [FTPS operations](../operations/ftps.md) for limits
+and cleanup behavior.
@@ -2,11 +2,11 @@
  "nodes": {
    "nixpkgs": {
      "locked": {
-        "lastModified": 1789073787,
-        "narHash": "sha256-xfX/toC2QV707s06GbP4II/TxYF0fNQj7s5/LClNDKc=",
+        "lastModified": 1789632929,
+        "narHash": "sha256-RjR8AyvGlWuw16XRj7C1YDEw4E27ciuyPeB2nDMzgTU=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "aff8a0b28396750446e5537a96461bc4facdb287",
+        "rev": "a32edd7654519351e48e80372a928df336394670",
        "type": "github"
      },
      "original": {
@@ -29,11 +29,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1789196581,
-        "narHash": "sha256-yJr1Bt4fKkKIpPYbsKGqJ0VFdoDnURgRwuffmBQ2WzY=",
+        "lastModified": 1789802016,
+        "narHash": "sha256-QVFfArbYaQMGK4dFV+5RUL72Ci+hNHVhaF/R9qJfDhs=",
        "owner": "oxalica",
        "repo": "rust-overlay",
-        "rev": "228ecefb6329d5a531b77b46b581a2f0c26ee056",
+        "rev": "26a71e661c47bd21a05d06fec749f3f7c75e9d12",
        "type": "github"
      },
      "original": {
@@ -189,6 +189,7 @@ mod tests {
    use crate::admin::handlers::target_descriptor::admin_target_spec_from_builtin;
    use crate::admin::runtime_sources::{IamInterface, KmsInterface};
    use crate::admin::storage_api::config::save_admin_server_config;
+    use crate::admin::storage_api::error::StorageError;
    use rustfs_config::audit::AUDIT_WEBHOOK_SUB_SYS;
    use rustfs_config::server_config::KVS;
    use rustfs_config::{ENABLE_KEY, EnableState, SCANNER_CYCLE, SCANNER_SUB_SYS, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR};
@@ -231,11 +232,17 @@ mod tests {
            let mut poll = tokio::time::interval(Duration::from_millis(10));
            loop {
                poll.tick().await;
-                let config = read_admin_config_without_migrate(store.clone())
-                    .await
-                    .expect("read persisted server config");
-                if config.0.get(subsystem).is_some_and(|targets| targets.contains_key(target)) {
-                    return;
+                match read_admin_config_without_migrate(store.clone()).await {
+                    Ok(config) => {
+                        if config.0.get(subsystem).is_some_and(|targets| targets.contains_key(target)) {
+                            return;
+                        }
+                    }
+                    Err(StorageError::Lock(rustfs_lock::LockError::Timeout { .. })) => {
+                        // The writer may still be committing the config snapshot. Retry after
+                        // the object lock is released instead of failing the polling helper.
+                    }
+                    Err(error) => panic!("read persisted server config: {error}"),
                }
            }
        })
@@ -0,0 +1,207 @@
+// Copyright 2024 RustFS Team
+// Licensed under the Apache License, Version 2.0.
+
+//! Administrator-directed integrity inspection and bounded, recoverable jobs.
+//! These routes do not change the MinIO batch-job compatibility endpoints.
+
+use crate::admin::auth::authorize_admin_request;
+use crate::admin::router::{AdminOperation, Operation, S3Router};
+use crate::admin::runtime_sources::object_store_from_req;
+use crate::admin::storage_api::integrity as service;
+use crate::admin::storage_api::s3::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, error as admin_s3_error};
+use crate::admin::utils::{extract_query_params, json_response, read_compatible_admin_body};
+use crate::server::ADMIN_PREFIX;
+use hyper::{Method, StatusCode};
+use matchit::Params;
+use rustfs_policy::policy::action::{Action, AdminAction};
+use serde::Deserialize;
+use uuid::Uuid;
+
+#[derive(Clone, Copy)]
+enum Route {
+    Readiness,
+    Inventory,
+    Create,
+    Status,
+    Control,
+}
+
+struct Handler(Route);
+
+fn permission(route: Route) -> AdminAction {
+    match route {
+        Route::Readiness => AdminAction::ServerInfoAdminAction,
+        Route::Inventory => AdminAction::InspectDataAction,
+        Route::Status => AdminAction::DescribeBatchJobAction,
+        Route::Create | Route::Control => AdminAction::StartBatchJobAction,
+    }
+}
+
+pub fn register_integrity_routes(router: &mut S3Router<AdminOperation>) -> std::io::Result<()> {
+    for (method, path, handler) in [
+        (Method::GET, "/v3/integrity/readiness", &Handler(Route::Readiness)),
+        (Method::GET, "/v3/integrity/{bucket}/inventory", &Handler(Route::Inventory)),
+        (Method::POST, "/v3/integrity/{bucket}/jobs", &Handler(Route::Create)),
+        (Method::GET, "/v3/integrity/{bucket}/jobs/{job_id}", &Handler(Route::Status)),
+        (Method::POST, "/v3/integrity/{bucket}/jobs/{job_id}/control", &Handler(Route::Control)),
+    ] {
+        router.insert(method, &format!("{ADMIN_PREFIX}{path}"), AdminOperation(handler))?;
+    }
+    Ok(())
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(deny_unknown_fields)]
+struct ControlRequest {
+    operation: Control,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "snake_case")]
+enum Control {
+    Pause,
+    Resume,
+    Cancel,
+}
+
+fn map_error(error: service::IntegrityError) -> S3Error {
+    match error {
+        service::IntegrityError::Invalid(message) => admin_s3_error(S3ErrorCode::InvalidArgument, message),
+        service::IntegrityError::Conflict => admin_s3_error(S3ErrorCode::PreconditionFailed, "integrity job or target changed"),
+        service::IntegrityError::Busy => {
+            admin_s3_error(S3ErrorCode::SlowDown, "an integrity worker is already active; retry resume later")
+        }
+        service::IntegrityError::NotFound => admin_s3_error(S3ErrorCode::NoSuchKey, "integrity job not found"),
+        service::IntegrityError::NotActivated => admin_s3_error(
+            S3ErrorCode::InvalidRequest,
+            "protected writes must be explicitly activated before migration",
+        ),
+        service::IntegrityError::UnsupportedBucket => admin_s3_error(
+            S3ErrorCode::InvalidRequest,
+            "migration requires a plain unversioned bucket without configured automation or retention",
+        ),
+        _ => admin_s3_error(
+            S3ErrorCode::InternalError,
+            "integrity operation failed; inspect the durable job before retrying",
+        ),
+    }
+}
+
+#[async_trait::async_trait]
+impl Operation for Handler {
+    async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
+        let credentials = authorize_admin_request(&req, vec![Action::AdminAction(permission(self.0))]).await?;
+        if matches!(self.0, Route::Readiness) {
+            return json_response(StatusCode::OK, &service::readiness());
+        }
+        let store = object_store_from_req(&req)
+            .ok_or_else(|| admin_s3_error(S3ErrorCode::InternalError, "object store is not initialized"))?;
+        let bucket = params
+            .get("bucket")
+            .ok_or_else(|| admin_s3_error(S3ErrorCode::InvalidArgument, "bucket is required"))?;
+        match self.0 {
+            Route::Inventory => {
+                let query = extract_query_params(&req.uri);
+                if query
+                    .keys()
+                    .any(|key| !["prefix", "key-marker", "version-marker", "limit"].contains(&key.as_str()))
+                {
+                    return Err(admin_s3_error(S3ErrorCode::InvalidArgument, "unknown inventory parameter"));
+                }
+                let limit = query
+                    .get("limit")
+                    .map_or(Ok(100), |v| v.parse::<i32>())
+                    .map_err(|_| admin_s3_error(S3ErrorCode::InvalidArgument, "invalid limit"))?;
+                let page = service::inventory(
+                    store,
+                    bucket,
+                    query.get("prefix").map_or("", String::as_str),
+                    query.get("key-marker").cloned(),
+                    query.get("version-marker").cloned(),
+                    limit,
+                )
+                .await
+                .map_err(map_error)?;
+                json_response(StatusCode::OK, &page)
+            }
+            Route::Create => {
+                let body = read_compatible_admin_body(req.input, 128 * 1024, req.uri.path(), &credentials.secret_key).await?;
+                let request: service::JobRequest = serde_json::from_slice(&body)
+                    .map_err(|_| admin_s3_error(S3ErrorCode::InvalidArgument, "invalid integrity job request"))?;
+                let job = service::create_job(store, bucket, request).await.map_err(map_error)?;
+                json_response(StatusCode::CREATED, &job)
+            }
+            Route::Status | Route::Control => {
+                let id = params
+                    .get("job_id")
+                    .and_then(|s| Uuid::parse_str(s).ok())
+                    .filter(|id| !id.is_nil())
+                    .ok_or_else(|| admin_s3_error(S3ErrorCode::InvalidArgument, "invalid job id"))?;
+                let job = if matches!(self.0, Route::Status) {
+                    service::get_job(store, bucket, id).await.map_err(map_error)?
+                } else {
+                    let body = read_compatible_admin_body(req.input, 1024, req.uri.path(), &credentials.secret_key).await?;
+                    let request: ControlRequest = serde_json::from_slice(&body)
+                        .map_err(|_| admin_s3_error(S3ErrorCode::InvalidArgument, "expected pause, resume or cancel"))?;
+                    match request.operation {
+                        Control::Resume => service::resume_job(store, bucket, id).await,
+                        Control::Pause => service::control_job(store, bucket, id, false).await,
+                        Control::Cancel => service::control_job(store, bucket, id, true).await,
+                    }
+                    .map_err(map_error)?
+                };
+                json_response(StatusCode::OK, &job)
+            }
+            Route::Readiness => json_response(StatusCode::OK, &service::readiness()),
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    #[test]
+    fn integrity_read_permissions_cannot_start_or_resume_payload_writes() {
+        assert_eq!(permission(Route::Create), AdminAction::StartBatchJobAction);
+        assert_eq!(permission(Route::Control), AdminAction::StartBatchJobAction);
+        assert_eq!(permission(Route::Inventory), AdminAction::InspectDataAction);
+        assert_eq!(permission(Route::Status), AdminAction::DescribeBatchJobAction);
+    }
+    #[test]
+    fn control_rejects_unknown_fields_and_operations() {
+        assert!(serde_json::from_str::<ControlRequest>(r#"{"operation":"resume","force":true}"#).is_err());
+        assert!(serde_json::from_str::<ControlRequest>(r#"{"operation":"overwrite"}"#).is_err());
+    }
+}
+
+#[cfg(test)]
+#[tokio::test]
+async fn integrity_routes_reject_unauthenticated_requests_before_storage_access() {
+    let mut router = matchit::Router::new();
+    router.insert("/{bucket}", ()).expect("test route");
+    for route in [
+        Route::Readiness,
+        Route::Inventory,
+        Route::Create,
+        Route::Status,
+        Route::Control,
+    ] {
+        let req = S3Request {
+            input: Body::from(String::new()),
+            method: Method::POST,
+            uri: http::Uri::from_static("/rustfs/admin/v3/integrity/example/jobs"),
+            headers: http::HeaderMap::new(),
+            extensions: http::Extensions::new(),
+            credentials: None,
+            region: None,
+            service: None,
+            trailing_headers: None,
+        };
+        let error = Handler(route)
+            .call(req, router.at("/example").expect("params").params)
+            .await
+            .expect_err("credentials required");
+        assert_eq!(error.code(), &S3ErrorCode::InvalidRequest);
+        assert_eq!(error.message(), Some("get cred failed"));
+    }
+}
@@ -33,6 +33,7 @@ pub(crate) mod iam_error;
 pub mod idp_compat;
 pub mod ilm_transition;
 pub mod inspect_archive;
+pub mod integrity;
 pub mod is_admin;
 pub mod kms;
 pub mod kms_audit;
@@ -43,7 +44,6 @@ pub mod kms_key_metadata;
 pub mod kms_keys;
 pub mod kms_management;
 pub mod kms_rekey;
-pub mod metrics;
 pub mod mfa;
 pub mod module_switch;
 mod notify_runtime_access;
@@ -58,6 +58,7 @@ pub mod pools;
 pub mod profile;
 pub mod profile_admin;
 pub mod quota;
+pub mod realtime;
 pub mod rebalance;
 pub mod replication;
 pub mod scanner;
@@ -145,7 +146,7 @@ mod tests {
        let _inspect_data_handler = system::InspectDataHandler {};
        let _storage_info_handler = system::StorageInfoHandler {};
        let _data_usage_handler = system::DataUsageInfoHandler {};
-        let _metrics_handler = metrics::MetricsHandler {};
+        let _metrics_handler = realtime::MetricsHandler {};
        let _profile_handler = profile_admin::ProfileHandler {};
        let _profile_status_handler = profile_admin::ProfileStatusHandler {};
        let _tls_status_handler = tls_debug::TlsStatusHandler {};
@@ -14,7 +14,7 @@

 //! Console realtime metrics API.
 //!
-//! This preserves the console's fixed `/admin/v3/metrics` contract while
+//! This preserves the console's fixed `/admin/v3/realtime` contract while
 //! keeping the response format explicitly NDJSON. It is not a Prometheus text
 //! exposition endpoint.

@@ -302,7 +302,7 @@ mod tests {

    #[test]
    fn metrics_params_default_to_single_sample() {
-        let uri: Uri = "/rustfs/admin/v3/metrics".parse().unwrap();
+        let uri: Uri = "/rustfs/admin/v3/realtime".parse().unwrap();
        let mp = extract_metrics_init_params(&uri);

        assert_eq!(resolve_sample_count(&mp), DEFAULT_METRICS_SAMPLES);
@@ -310,7 +310,7 @@ mod tests {

    #[test]
    fn metrics_params_treat_zero_as_single_sample() {
-        let uri: Uri = "/rustfs/admin/v3/metrics?n=0".parse().unwrap();
+        let uri: Uri = "/rustfs/admin/v3/realtime?n=0".parse().unwrap();
        let mp = extract_metrics_init_params(&uri);

        assert_eq!(resolve_sample_count(&mp), DEFAULT_METRICS_SAMPLES);
@@ -318,7 +318,7 @@ mod tests {

    #[test]
    fn metrics_params_cap_samples_to_safety_limit() {
-        let uri: Uri = "/rustfs/admin/v3/metrics?n=9999".parse().unwrap();
+        let uri: Uri = "/rustfs/admin/v3/realtime?n=9999".parse().unwrap();
        let mp = extract_metrics_init_params(&uri);

        assert_eq!(resolve_sample_count(&mp), MAX_METRICS_SAMPLES);
@@ -332,7 +332,7 @@ mod tests {
    #[tokio::test]
    async fn metrics_handler_rejects_missing_credentials() {
        let result = MetricsHandler {}
-            .call(build_metrics_request("/rustfs/admin/v3/metrics"), Params::new())
+            .call(build_metrics_request("/rustfs/admin/v3/realtime"), Params::new())
            .await;
        let err = match result {
            Ok(_) => panic!("metrics handler must reject unauthenticated requests"),
@@ -43,13 +43,21 @@ use s3s::{
    s3_error,
 };
 use serde::{Deserialize, Serialize};
-use std::{sync::Arc, time::Duration};
+use std::{
+    future::Future,
+    sync::Arc,
+    time::{Duration, Instant},
+};
 use time::OffsetDateTime;
 use tracing::{error, info, warn};

 const LOG_COMPONENT_ADMIN: &str = "admin";
 const LOG_SUBSYSTEM_REBALANCE: &str = "rebalance";
 const EVENT_ADMIN_REBALANCE_STATE: &str = "admin_rebalance_state";
+const REBALANCE_START_FLEET_PROOF_MARKER: &str = "pool activation requires a live fleet capability proof";
+const REBALANCE_START_FLEET_PROOF_RETRY_BUDGET: Duration = Duration::from_secs(30);
+const REBALANCE_START_FLEET_PROOF_RETRY_DELAY: Duration = Duration::from_secs(5);
+const REBALANCE_START_RETRY_AFTER_SECS: &str = "5";

 fn admin_request_id(headers: &HeaderMap) -> Option<&str> {
    headers
@@ -105,6 +113,80 @@ fn rebalance_internal_error(message: impl Into<String>) -> S3Error {
    S3Error::with_message(S3ErrorCode::InternalError, message.into())
 }

+fn rebalance_fleet_proof_unavailable_error() -> S3Error {
+    let mut err = S3Error::with_message(
+        S3ErrorCode::ServiceUnavailable,
+        "rebalance start is waiting for cluster capability readiness; retry later",
+    );
+    let mut headers = HeaderMap::new();
+    headers.insert(http::header::RETRY_AFTER, HeaderValue::from_static(REBALANCE_START_RETRY_AFTER_SECS));
+    err.set_headers(headers);
+    err
+}
+
+fn is_rebalance_start_fleet_proof_retryable(err: &StorageError) -> bool {
+    crate::storage_api::capacity::is_pool_activation_fleet_proof_error(err)
+        && err.to_string().contains(REBALANCE_START_FLEET_PROOF_MARKER)
+}
+
+fn rebalance_start_failure_error(start_err: &StorageError, rollback_result: &Result<(), String>) -> S3Error {
+    if crate::storage_api::capacity::is_pool_activation_fleet_proof_error(start_err) {
+        if rollback_result.is_ok() {
+            return rebalance_fleet_proof_unavailable_error();
+        }
+        return rebalance_internal_error("failed to roll back rebalance start after fleet capability readiness failure");
+    }
+
+    rebalance_internal_error(rebalance_start_rollback_error(&start_err.to_string(), rollback_result))
+}
+
+async fn retry_rebalance_start_operation<T, F, Fut>(
+    operation: &'static str,
+    rebalance_id: &str,
+    request_id: &str,
+    actor: &str,
+    remote_addr: &str,
+    deadline: Instant,
+    mut operation_fn: F,
+) -> Result<T, StorageError>
+where
+    F: FnMut() -> Fut,
+    Fut: Future<Output = Result<T, StorageError>>,
+{
+    let mut attempt = 1_u32;
+    loop {
+        match operation_fn().await {
+            Ok(value) => return Ok(value),
+            Err(err) if is_rebalance_start_fleet_proof_retryable(&err) => {
+                let now = Instant::now();
+                if now >= deadline {
+                    return Err(err);
+                }
+                let delay = REBALANCE_START_FLEET_PROOF_RETRY_DELAY.min(deadline.saturating_duration_since(now));
+                warn!(
+                    event = EVENT_ADMIN_REBALANCE_STATE,
+                    component = LOG_COMPONENT_ADMIN,
+                    subsystem = LOG_SUBSYSTEM_REBALANCE,
+                    action = "start",
+                    state = "fleet_proof_retry_scheduled",
+                    result = "retrying",
+                    operation,
+                    attempt,
+                    retry_delay_ms = delay.as_millis(),
+                    request_id = %request_id,
+                    actor = %actor,
+                    remote_addr = %remote_addr,
+                    rebalance_id = %rebalance_id,
+                    "admin rebalance state"
+                );
+                tokio::time::sleep(delay).await;
+                attempt = attempt.saturating_add(1);
+            }
+            Err(err) => return Err(err),
+        }
+    }
+}
+
 fn rebalance_rollback_stop_failure_message(rebalance_id: &str, failures: &[String]) -> String {
    format!("cluster stop_rebalance rollback for {rebalance_id} partial: {}", failures.join("; "))
 }
@@ -209,7 +291,7 @@ async fn rollback_rebalance_start_for_admin(
    store: &Arc<ECStore>,
    notification_sys: Option<&NotificationSys>,
    rebalance_id: &str,
-    start_err: &str,
+    start_err: &StorageError,
    request_id: &str,
    actor: &str,
    remote_addr: &str,
@@ -246,7 +328,7 @@ async fn rollback_rebalance_start_for_admin(
        ),
    }

-    Err(rebalance_internal_error(rebalance_start_rollback_error(start_err, &rollback_result)))
+    Err(rebalance_start_failure_error(start_err, &rollback_result))
 }

 pub fn register_rebalance_route(r: &mut S3Router<AdminOperation>) -> std::io::Result<()> {
@@ -562,6 +644,7 @@ impl Operation for RebalanceStart {
            "admin rebalance state"
        );
        let notification_sys = current_notification_system();
+        let fleet_proof_deadline = Instant::now() + REBALANCE_START_FLEET_PROOF_RETRY_BUDGET;
        for step in rebalance_start_steps(notification_sys.is_some()) {
            match step {
                RebalanceStartStep::PropagateFence => {
@@ -592,13 +675,11 @@ impl Operation for RebalanceStart {
                                error = %err,
                                "admin rebalance state"
                            );
-
-                            let start_err = err.to_string();
                            rollback_rebalance_start_for_admin(
                                &store,
                                Some(notification_sys),
                                &id,
-                                &start_err,
+                                &err,
                                &request_id,
                                &actor,
                                &remote_addr,
@@ -608,7 +689,17 @@ impl Operation for RebalanceStart {
                    }
                }
                RebalanceStartStep::StartLocal => {
-                    if let Err(err) = store.start_rebalance_for_id(&id).await {
+                    let start_result = retry_rebalance_start_operation(
+                        "start_rebalance_for_id",
+                        &id,
+                        &request_id,
+                        &actor,
+                        &remote_addr,
+                        fleet_proof_deadline,
+                        || store.start_rebalance_for_id(&id),
+                    )
+                    .await;
+                    if let Err(err) = start_result {
                        error!(
                            event = EVENT_ADMIN_REBALANCE_STATE,
                            component = LOG_COMPONENT_ADMIN,
@@ -682,6 +773,9 @@ impl Operation for RebalanceStart {
                                )));
                            }
                        }
+                        if crate::storage_api::capacity::is_pool_activation_fleet_proof_error(&err) {
+                            return Err(rebalance_fleet_proof_unavailable_error());
+                        }
                        return Err(rebalance_internal_error(format!(
                            "failed to start rebalance after metadata initialized for {id}; local metadata was finalized as failed: {start_err}"
                        )));
@@ -701,7 +795,17 @@ impl Operation for RebalanceStart {
                            rebalance_id = %id,
                            "admin rebalance state"
                        );
-                        if let Err(err) = notification_sys.load_rebalance_meta(true).await {
+                        let worker_result = retry_rebalance_start_operation(
+                            "load_rebalance_meta(start=true)",
+                            &id,
+                            &request_id,
+                            &actor,
+                            &remote_addr,
+                            fleet_proof_deadline,
+                            || notification_sys.load_rebalance_meta(true),
+                        )
+                        .await;
+                        if let Err(err) = worker_result {
                            error!(
                                event = EVENT_ADMIN_REBALANCE_STATE,
                                component = LOG_COMPONENT_ADMIN,
@@ -715,13 +819,11 @@ impl Operation for RebalanceStart {
                                error = %err,
                                "admin rebalance state"
                            );
-
-                            let start_err = err.to_string();
                            rollback_rebalance_start_for_admin(
                                &store,
                                Some(notification_sys),
                                &id,
-                                &start_err,
+                                &err,
                                &request_id,
                                &actor,
                                &remote_addr,
@@ -1059,12 +1161,14 @@ mod rebalance_handler_tests {
    use super::calculate_rebalance_progress;
    use super::{
        Body, HeaderMap, Method, Operation, Params, RebalPoolProgress, RebalanceAdminStatus, RebalancePoolStatus, RebalanceStart,
-        RebalanceStartStep, RebalanceStatus, RebalanceStop, RebalanceStopPropagationStatus, S3ErrorCode, S3Request, Uri,
-        build_rebalance_admin_status, build_rebalance_pool_statuses, build_rebalance_stop_propagation_status,
-        rebalance_pool_used, rebalance_query_present, rebalance_remaining_buckets, rebalance_rollback_failure_message,
-        rebalance_rollback_stop_failure_message, rebalance_start_rollback_error, rebalance_start_steps, rebalance_stop_target_id,
-        rebalance_used_pct, rollback_result_label, stop_rebalance_admission_first,
+        RebalanceStartStep, RebalanceStatus, RebalanceStop, RebalanceStopPropagationStatus, S3ErrorCode, S3Request, StatusCode,
+        Uri, build_rebalance_admin_status, build_rebalance_pool_statuses, build_rebalance_stop_propagation_status,
+        is_rebalance_start_fleet_proof_retryable, rebalance_pool_used, rebalance_query_present, rebalance_remaining_buckets,
+        rebalance_rollback_failure_message, rebalance_rollback_stop_failure_message, rebalance_start_failure_error,
+        rebalance_start_rollback_error, rebalance_start_steps, rebalance_stop_target_id, rebalance_used_pct,
+        retry_rebalance_start_operation, rollback_result_label, stop_rebalance_admission_first,
    };
+    use crate::admin::storage_api::error::StorageError;
    use crate::admin::storage_api::rebalance::{
        DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo,
        RebalanceMeta, RebalanceStats, RebalanceStopPropagationRecord, encode_rebalance_stop_propagation_record,
@@ -1328,6 +1432,108 @@ mod rebalance_handler_tests {
        assert!(message.contains("rollback error: peer b stop_rebalance failed: timeout"));
    }

+    #[test]
+    fn test_rebalance_start_fleet_proof_maps_to_retryable_503() {
+        let start_err = StorageError::other(super::REBALANCE_START_FLEET_PROOF_MARKER);
+
+        let err = rebalance_start_failure_error(&start_err, &Ok(()));
+
+        assert_eq!(err.code(), &S3ErrorCode::ServiceUnavailable);
+        assert_eq!(err.status_code(), Some(StatusCode::SERVICE_UNAVAILABLE));
+        assert_eq!(
+            err.headers()
+                .and_then(|headers| headers.get(http::header::RETRY_AFTER))
+                .and_then(|value| value.to_str().ok()),
+            Some(super::REBALANCE_START_RETRY_AFTER_SECS)
+        );
+        assert!(
+            !err.message()
+                .expect("retryable rebalance start error should have a message")
+                .contains(super::REBALANCE_START_FLEET_PROOF_MARKER)
+        );
+    }
+
+    #[test]
+    fn test_rebalance_start_fleet_proof_rollback_failure_is_generic_internal_error() {
+        let start_err = StorageError::other(super::REBALANCE_START_FLEET_PROOF_MARKER);
+        let rollback_result = Err("peer rollback failed".to_string());
+
+        let err = rebalance_start_failure_error(&start_err, &rollback_result);
+
+        assert_eq!(err.code(), &S3ErrorCode::InternalError);
+        assert!(
+            !err.message()
+                .expect("internal rebalance start error should have a message")
+                .contains(super::REBALANCE_START_FLEET_PROOF_MARKER)
+        );
+    }
+
+    #[test]
+    fn test_rebalance_start_unrelated_failure_remains_internal_error() {
+        let start_err = StorageError::other("disk read failed");
+
+        let err = rebalance_start_failure_error(&start_err, &Ok(()));
+
+        assert_eq!(err.code(), &S3ErrorCode::InternalError);
+        assert!(err.message().is_some_and(|message| message.contains("disk read failed")));
+    }
+
+    #[test]
+    fn test_rebalance_start_fleet_proof_retry_only_matches_missing_live_proof() {
+        let missing = StorageError::other(super::REBALANCE_START_FLEET_PROOF_MARKER);
+        let expired = StorageError::other(format!(
+            "rebalance meta save failed during start_rebalance: {}",
+            "pool activation fleet capability proof expired before commit"
+        ));
+
+        assert!(is_rebalance_start_fleet_proof_retryable(&missing));
+        assert!(!is_rebalance_start_fleet_proof_retryable(&expired));
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_rebalance_start_retry_waits_for_fleet_proof() {
+        use std::sync::{
+            Arc,
+            atomic::{AtomicUsize, Ordering},
+        };
+
+        let attempts = Arc::new(AtomicUsize::new(0));
+        let task = tokio::spawn({
+            let attempts = Arc::clone(&attempts);
+            async move {
+                retry_rebalance_start_operation(
+                    "load_rebalance_meta(start=true)",
+                    "rebalance-id",
+                    "request-id",
+                    "actor",
+                    "remote",
+                    std::time::Instant::now() + super::REBALANCE_START_FLEET_PROOF_RETRY_BUDGET,
+                    move || {
+                        let attempts = Arc::clone(&attempts);
+                        async move {
+                            let attempt = attempts.fetch_add(1, Ordering::SeqCst);
+                            if attempt == 0 {
+                                Err(StorageError::other(super::REBALANCE_START_FLEET_PROOF_MARKER))
+                            } else {
+                                Ok(())
+                            }
+                        }
+                    },
+                )
+                .await
+            }
+        });
+
+        tokio::task::yield_now().await;
+        assert_eq!(attempts.load(Ordering::SeqCst), 1);
+
+        tokio::time::advance(super::REBALANCE_START_FLEET_PROOF_RETRY_DELAY).await;
+        task.await
+            .expect("retry task should not panic")
+            .expect("fleet proof retry should eventually succeed");
+        assert_eq!(attempts.load(Ordering::SeqCst), 2);
+    }
+
    #[test]
    fn test_rebalance_rollback_stop_failure_message_lists_failures() {
        let failures = vec![
@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-use super::{cluster_snapshot, metrics};
+use super::{cluster_snapshot, realtime};
 use crate::admin::auth::authorize_admin_request;
 use crate::admin::handlers::account::{ACCOUNT_INFO_ROUTE, ACCOUNT_PASSWORD_ROUTE};
 use crate::admin::handlers::mfa::{ACCOUNT_MFA_ROUTE, MFA_CHALLENGE_ROUTE, USER_MFA_ROUTE};
@@ -167,8 +167,8 @@ pub fn register_system_route(r: &mut S3Router<AdminOperation>) -> std::io::Resul

    r.insert(
        Method::GET,
-        format!("{}{}", ADMIN_PREFIX, "/v3/metrics").as_str(),
-        AdminOperation(&metrics::MetricsHandler {}),
+        format!("{}{}", ADMIN_PREFIX, "/v3/realtime").as_str(),
+        AdminOperation(&realtime::MetricsHandler {}),
    )?;

    r.insert(
@@ -35,10 +35,10 @@ mod route_registration_test;

 use handlers::{
    account, audit, batch_job, bucket_meta, cluster_snapshot, config_admin, diagnostics, durability as durability_handler,
-    extensions, gateway_key_inventory, heal, health, idp_compat, ilm_transition, inspect_archive, kms, mfa, module_switch,
-    object_data_cache, object_zip_download, oidc, on_demand_migration, plugins_catalog, plugins_instances, pools, profile_admin,
-    quota as quota_handler, rebalance, replication as replication_handler, scanner, site_replication, sts, system, table_catalog,
-    tier, tls_debug, usage_prefix, user,
+    extensions, gateway_key_inventory, heal, health, idp_compat, ilm_transition, inspect_archive, integrity, kms, mfa,
+    module_switch, object_data_cache, object_zip_download, oidc, on_demand_migration, plugins_catalog, plugins_instances, pools,
+    profile_admin, quota as quota_handler, rebalance, replication as replication_handler, scanner, site_replication, sts, system,
+    table_catalog, tier, tls_debug, usage_prefix, user,
 };
 use router::{AdminOperation, S3Router};
 use s3s::route::S3Route;
@@ -94,6 +94,7 @@ fn register_admin_routes(r: &mut S3Router<AdminOperation>) -> std::io::Result<()

    replication_handler::register_replication_route(r)?;
    batch_job::register_batch_job_route(r)?;
+    integrity::register_integrity_routes(r)?;
    site_replication::register_site_replication_route(r)?;
    profile_admin::register_profiling_route(r)?;
    diagnostics::register_diagnostics_route(r)?;
@@ -320,7 +320,7 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
    ),
    admin(HttpMethod::Get, "/rustfs/admin/v3/info", SERVER_INFO, RouteRiskLevel::Sensitive),
    admin(HttpMethod::Get, "/rustfs/admin/v3/storageinfo", STORAGE_INFO, RouteRiskLevel::Sensitive),
-    admin(HttpMethod::Get, "/rustfs/admin/v3/metrics", GET_METRICS, RouteRiskLevel::Sensitive),
+    admin(HttpMethod::Get, "/rustfs/admin/v3/realtime", GET_METRICS, RouteRiskLevel::Sensitive),
    admin(
        HttpMethod::Get,
        "/rustfs/admin/v3/object-data-cache/stats",
@@ -1585,6 +1585,36 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
        COMMIT_TABLE,
        RouteRiskLevel::High,
    ),
+    admin(
+        HttpMethod::Get,
+        "/rustfs/admin/v3/integrity/readiness",
+        SERVER_INFO,
+        RouteRiskLevel::Sensitive,
+    ),
+    admin(
+        HttpMethod::Get,
+        "/rustfs/admin/v3/integrity/{bucket}/inventory",
+        INSPECT_DATA,
+        RouteRiskLevel::Sensitive,
+    ),
+    admin(
+        HttpMethod::Post,
+        "/rustfs/admin/v3/integrity/{bucket}/jobs",
+        START_BATCH_JOB,
+        RouteRiskLevel::High,
+    ),
+    admin(
+        HttpMethod::Get,
+        "/rustfs/admin/v3/integrity/{bucket}/jobs/{job_id}",
+        DESCRIBE_BATCH_JOB,
+        RouteRiskLevel::Sensitive,
+    ),
+    admin(
+        HttpMethod::Post,
+        "/rustfs/admin/v3/integrity/{bucket}/jobs/{job_id}/control",
+        START_BATCH_JOB,
+        RouteRiskLevel::High,
+    ),
    // MinIO admin compat: batch job lifecycle (backlog#613).
    admin(HttpMethod::Post, "/rustfs/admin/v3/start-job", START_BATCH_JOB, RouteRiskLevel::High),
    admin(HttpMethod::Get, "/rustfs/admin/v3/list-jobs", LIST_BATCH_JOBS, RouteRiskLevel::Sensitive),
@@ -2303,7 +2333,7 @@ mod tests {

    #[test]
    fn route_policy_maps_metrics_to_explicit_admin_action() {
-        assert_action(HttpMethod::Get, "/rustfs/admin/v3/metrics", GET_METRICS);
+        assert_action(HttpMethod::Get, "/rustfs/admin/v3/realtime", GET_METRICS);
    }

    #[test]
@@ -190,7 +190,7 @@ fn expected_admin_route_matrix() -> Vec<RouteMatrixEntry> {
        admin_route(Method::GET, "/v3/storageinfo"),
        admin_route(Method::GET, "/v3/datausageinfo"),
        admin_route_sample(Method::GET, "/v3/usage/{bucket}", "/v3/usage/test-bucket"),
-        admin_route(Method::GET, "/v3/metrics"),
+        admin_route(Method::GET, "/v3/realtime"),
        admin_route(Method::GET, "/v3/object-data-cache/stats"),
        admin_route(Method::POST, "/v3/object-data-cache/flush"),
        admin_route(Method::GET, "/v3/pools/list"),
@@ -337,6 +337,19 @@ fn expected_admin_route_matrix() -> Vec<RouteMatrixEntry> {
        admin_route(Method::DELETE, "/v3/remove-remote-target"),
        admin_route(Method::POST, "/v3/replication/diff"),
        admin_route(Method::GET, "/v3/replication/mrf"),
+        admin_route(Method::GET, "/v3/integrity/readiness"),
+        admin_route_sample(Method::GET, "/v3/integrity/{bucket}/inventory", "/v3/integrity/example/inventory"),
+        admin_route_sample(Method::POST, "/v3/integrity/{bucket}/jobs", "/v3/integrity/example/jobs"),
+        admin_route_sample(
+            Method::GET,
+            "/v3/integrity/{bucket}/jobs/{job_id}",
+            "/v3/integrity/example/jobs/11111111-1111-4111-8111-111111111111",
+        ),
+        admin_route_sample(
+            Method::POST,
+            "/v3/integrity/{bucket}/jobs/{job_id}/control",
+            "/v3/integrity/example/jobs/11111111-1111-4111-8111-111111111111/control",
+        ),
        admin_route(Method::POST, "/v3/start-job"),
        admin_route(Method::GET, "/v3/list-jobs"),
        admin_route(Method::GET, "/v3/status-job"),
@@ -1333,7 +1346,7 @@ fn test_register_routes_cover_representative_admin_paths() {
    assert_route(&router, Method::POST, &admin_path("/v3/service"));
    assert_route(&router, Method::GET, &admin_path("/v3/info"));
    assert_route(&router, Method::GET, &admin_path("/v3/storageinfo"));
-    assert_route(&router, Method::GET, &admin_path("/v3/metrics"));
+    assert_route(&router, Method::GET, &admin_path("/v3/realtime"));

    assert_route(&router, Method::GET, &admin_path("/v3/pools/list"));
    assert_route(&router, Method::GET, &admin_path("/v3/decommission/status"));
@@ -16,6 +16,12 @@ use std::ops::Deref;
 use std::sync::Arc;

 use rustfs_storage_api as storage_contracts;
+
+pub(crate) mod integrity {
+    pub(crate) use crate::storage::storage_api::ecstore_integrity::{
+        IntegrityError, JobRequest, control_job, create_job, get_job, inventory, readiness, resume_job,
+    };
+}
 use time::OffsetDateTime;

 mod ecstore_bucket {
@@ -43,6 +43,10 @@ use std::sync::Arc;
 use tracing::{debug, error, info, warn};

 pub type AdminUsecaseResult<T> = Result<T, ApiError>;
+
+/// `waitingReason` value reported while a pool is paused on target capacity.
+const DECOMMISSION_WAITING_REASON_CAPACITY: &str = "capacity";
+
 pub const ADMIN_CLUSTER_SNAPSHOT_ROUTE: &str = "/rustfs/admin/v4/cluster/snapshot";
 pub const ADMIN_EXTENSIONS_CATALOG_ROUTE: &str = "/rustfs/admin/v4/extensions/catalog";
 pub const ADMIN_RUNTIME_CAPABILITIES_ROUTE: &str = "/rustfs/admin/v4/runtime/capabilities";
@@ -108,6 +112,8 @@ pub struct AdminPoolDecommissionInfo {
    pub bytes_failed: usize,
    #[serde(rename = "waitingReason")]
    pub waiting_reason: Option<String>,
+    #[serde(rename = "capacityBlockedReason", skip_serializing_if = "Option::is_none")]
+    pub capacity_blocked_reason: Option<String>,
    #[serde(rename = "unresolvedEntries", skip_serializing_if = "Vec::is_empty")]
    pub unresolved_entries: Vec<DecommissionUnresolvedEntry>,
 }
@@ -622,12 +628,23 @@ impl DefaultAdminUsecase {
            bytes_done: info.bytes_done,
            bytes_failed: info.bytes_failed,
            waiting_reason,
+            capacity_blocked_reason: info.capacity_blocked_reason,
            unresolved_entries: info.unresolved_entries,
        }
    }

+    /// Why a pool is not currently making progress.
+    ///
+    /// A durable capacity pause is reported ahead of the worker states: it is
+    /// the actionable condition, and `capacityBlockedReason` carries the detail.
    fn decommission_waiting_reason(info: &PoolDecommissionInfo) -> Option<&'static str> {
-        if !info.has_decommission_state() || info.complete || info.failed || info.canceled || info.start_time.is_some() {
+        if !info.has_decommission_state() || info.complete || info.failed || info.canceled {
+            return None;
+        }
+        if info.capacity_blocked_reason.is_some() {
+            return Some(DECOMMISSION_WAITING_REASON_CAPACITY);
+        }
+        if info.start_time.is_some() {
            return None;
        }
        if info.queued {
@@ -995,6 +1012,57 @@ mod tests {
        assert_eq!(item.rebalance_status, "started");
    }

+    /// A durable capacity pause must be distinguishable from "slow but
+    /// progressing": the pause surfaces its own waiting reason, the persisted
+    /// detail.
+    #[test]
+    fn admin_pool_list_item_exposes_decommission_capacity_pause() {
+        let item = DefaultAdminUsecase::pool_list_item_from_status(
+            PoolStatus {
+                id: 0,
+                cmd_line: "pool-0".to_string(),
+                last_update: OffsetDateTime::UNIX_EPOCH,
+                decommission: Some(PoolDecommissionInfo {
+                    start_time: Some(OffsetDateTime::UNIX_EPOCH),
+                    total_size: 1_000,
+                    current_size: 500,
+                    capacity_blocked_reason: Some("target pool 1 target capacity mutation gate is busy".to_string()),
+                    ..Default::default()
+                }),
+            },
+            (RebalStatus::None, false),
+        );
+
+        let value = serde_json::to_value(item).expect("paused pool status should serialize");
+        assert_eq!(value["decommissionInfo"]["waitingReason"], "capacity");
+        assert_eq!(
+            value["decommissionInfo"]["capacityBlockedReason"],
+            "target pool 1 target capacity mutation gate is busy"
+        );
+    }
+
+    /// Once the pause clears, `waitingReason` must return to the worker states
+    /// without retaining a stale capacity reason.
+    #[test]
+    fn admin_pool_list_item_clears_capacity_pause() {
+        let item = DefaultAdminUsecase::pool_list_item_from_status(
+            PoolStatus {
+                id: 0,
+                cmd_line: "pool-0".to_string(),
+                last_update: OffsetDateTime::UNIX_EPOCH,
+                decommission: Some(PoolDecommissionInfo {
+                    start_time: Some(OffsetDateTime::UNIX_EPOCH),
+                    ..Default::default()
+                }),
+            },
+            (RebalStatus::None, false),
+        );
+
+        let value = serde_json::to_value(item).expect("resumed pool status should serialize");
+        assert!(value["decommissionInfo"]["waitingReason"].is_null());
+        assert!(value["decommissionInfo"].get("capacityBlockedReason").is_none());
+    }
+
    #[test]
    fn admin_pool_list_item_exposes_queued_decommission_state() {
        let item = DefaultAdminUsecase::pool_list_item_from_status(
@@ -499,6 +499,14 @@ fn get_object_stream_failure_reason(error_class: &'static str) -> &'static str {
    }
 }

+/// A mid-stream reopen can recover only when another representation of the
+/// committed object may exist. A single-disk, no-parity object has no alternate
+/// shard after a bitrot/read-quorum failure, so retrying the same read only
+/// repeats the failure and adds latency.
+fn should_attach_get_object_resume(info: &ObjectInfo) -> bool {
+    info.parity_blocks > 0 || info.data_blocks > 1 || !info.transitioned_object.tier.is_empty()
+}
+
 fn record_get_object_reader_stream_failure(
    reason: &'static str,
    error_class: &'static str,
@@ -4188,16 +4196,18 @@ impl DefaultObjectUsecase {
                part_number,
                lifecycle,
                |info| {
-                    Some(get_object_resume_control(GetObjectResumeContext::new(
-                        store,
-                        &bucket,
-                        &key,
-                        opts,
-                        &req.headers,
-                        info,
-                        resume_range_start,
-                        resume_range_end,
-                    )))
+                    should_attach_get_object_resume(info).then(|| {
+                        get_object_resume_control(GetObjectResumeContext::new(
+                            store,
+                            &bucket,
+                            &key,
+                            opts,
+                            &req.headers,
+                            info,
+                            resume_range_start,
+                            resume_range_end,
+                        ))
+                    })
                },
            )
            .await;
@@ -8365,6 +8375,34 @@ mod tests {
        assert_eq!(reopen_count.load(Ordering::Relaxed), 0);
    }

+    #[test]
+    fn get_object_resume_only_attaches_for_recoverable_layouts() {
+        let single_disk = ObjectInfo {
+            data_blocks: 1,
+            parity_blocks: 0,
+            ..Default::default()
+        };
+        assert!(!should_attach_get_object_resume(&single_disk));
+
+        let erasure = ObjectInfo {
+            data_blocks: 2,
+            parity_blocks: 1,
+            ..Default::default()
+        };
+        assert!(should_attach_get_object_resume(&erasure));
+
+        let transitioned = ObjectInfo {
+            data_blocks: 1,
+            parity_blocks: 0,
+            transitioned_object: crate::storage_api::server::event::contract::lifecycle::TransitionedObject {
+                tier: "remote-tier".to_string(),
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+        assert!(should_attach_get_object_resume(&transitioned));
+    }
+
    #[test]
    fn get_object_resume_identity_requires_same_version() {
        let version_id = Uuid::from_u128(0x1234);
@@ -673,7 +673,10 @@ fn build_put_object_expiration_header(event: &lifecycle::Event) -> Option<String
        return None;
    }

-    let expiry_date = expire_time.format(&Rfc3339).ok()?;
+    // `x-amz-expiration` is a date-valued S3 response header and therefore uses
+    // HTTP-date, not ISO-8601: the AWS SDKs hand this value to an RFC 822
+    // parser, which rejects `2026-10-02T00:00:00Z` and drops the metadata.
+    let expiry_date = format_http_date(&Timestamp::from(expire_time)).ok()?;
    Some(format!("expiry-date=\"{}\", rule-id=\"{}\"", expiry_date, event.rule_id))
 }

@@ -709,10 +712,21 @@ pub(super) fn parse_expires_header(expires: Option<&str>) -> S3Result<Option<Tim
 }

 pub(super) fn format_expires_header(expires: &Timestamp) -> S3Result<String> {
+    format_http_date(expires)
+}
+
+/// Render a date-valued S3 response header (`Expires`, `x-amz-expiration`) as
+/// HTTP-date: RFC 1123 with a literal `GMT` zone, e.g.
+/// `Fri, 23 Dec 2012 00:00:00 GMT`.
+///
+/// Interoperability depends on this shape. `x-amz-expiration` in particular is
+/// parsed with an RFC 822 parser by the AWS SDKs, which rejects an RFC 3339
+/// value such as `2026-10-02T00:00:00Z` and then discards the whole header.
+pub(super) fn format_http_date(timestamp: &Timestamp) -> S3Result<String> {
    let mut formatted = Vec::new();
-    expires
+    timestamp
        .format(TimestampFormat::HttpDate, &mut formatted)
-        .map_err(|e| ApiError::from(StorageError::other(format!("Invalid expires timestamp: {e}"))))?;
+        .map_err(|e| ApiError::from(StorageError::other(format!("Invalid HTTP-date timestamp: {e}"))))?;
    Ok(String::from_utf8_lossy(&formatted).into_owned())
 }

@@ -1873,9 +1887,10 @@ mod tests {
            storage_class: String::new(),
        };

-        let expiry_date = expire_time.format(&Rfc3339).unwrap();
-        let expected = format!("expiry-date=\"{}\", rule-id=\"rule-1\"", expiry_date);
-        assert_eq!(build_put_object_expiration_header(&event), Some(expected));
+        // HTTP-date, not RFC 3339: the AWS SDKs parse this field with an
+        // RFC 822 parser, so `2026-10-02T00:00:00Z` loses the metadata.
+        let expected = "expiry-date=\"Tue, 14 Nov 2023 22:13:20 GMT\", rule-id=\"rule-1\"";
+        assert_eq!(build_put_object_expiration_header(&event), Some(expected.to_string()));
    }

    #[test]
@@ -2538,7 +2538,7 @@ mod tests {
        assert!(futures::poll!(cancelled.as_mut()).is_pending());
        drop(cancelled);
        for path in [
-            "/rustfs/admin/v3/metrics",
+            "/rustfs/admin/v3/realtime",
            "/minio/admin/v3/storageinfo",
            CONSOLE_PREFIX,
            "/rustfs/rpc/test",
@@ -29,6 +29,7 @@ use std::{
 use tokio_util::sync::CancellationToken;

 const EVENT_ON_DEMAND_MIGRATION_RUNTIME_INITIALIZED: &str = "on_demand_migration_runtime_initialized";
+const EVENT_BUCKET_METADATA_STARTUP_RETRY: &str = "bucket_metadata_startup_retry";
 const EVENT_REPLICATION_RESYNC_STARTUP_BACKGROUND_CANCELED: &str = "replication_resync_startup_background_canceled";
 const EVENT_REPLICATION_RESYNC_STARTUP_BACKGROUND_COMPLETED: &str = "replication_resync_startup_background_completed";
 const EVENT_REPLICATION_RESYNC_STARTUP_BACKGROUND_FAILED: &str = "replication_resync_startup_background_failed";
@@ -36,8 +37,9 @@ const EVENT_REPLICATION_RESYNC_STARTUP_BACKGROUND_STARTED: &str = "replication_r
 const LOG_COMPONENT_STARTUP_BUCKET_METADATA: &str = "startup_bucket_metadata";
 const LOG_SUBSYSTEM_ON_DEMAND_MIGRATION: &str = "on_demand_migration";
 const LOG_SUBSYSTEM_REPLICATION: &str = "replication";
-const IAM_MIGRATION_MAX_RETRIES: usize = 15;
-const IAM_MIGRATION_RETRY_INTERVAL: Duration = Duration::from_secs(1);
+const LOG_SUBSYSTEM_METADATA_STARTUP: &str = "metadata_startup";
+const METADATA_STARTUP_MAX_RETRIES: usize = 15;
+const METADATA_STARTUP_RETRY_INTERVAL: Duration = Duration::from_secs(1);
 const METRIC_REPLICATION_RESYNC_STARTUP_BACKGROUND_DURATION_SECONDS: &str =
    "rustfs_replication_resync_startup_background_duration_seconds";
 const METRIC_REPLICATION_RESYNC_STARTUP_BACKGROUND_EVENTS_TOTAL: &str =
@@ -55,39 +57,45 @@ const STARTUP_BACKGROUND_STATUS_RUNNING: f64 = 2.0;
 const STARTUP_BACKGROUND_STATUS_CANCELED: f64 = 3.0;

 pub(crate) async fn init_embedded_bucket_metadata_runtime(store: Arc<ECStore>, ctx: &CancellationToken) -> IoResult<Vec<String>> {
-    let buckets_list = store
-        .list_bucket(&BucketOptions {
-            no_metadata: true,
-            ..Default::default()
-        })
-        .await
-        .map_err(|err| IoError::other(format!("list_bucket: {err}")))?;
+    let buckets_list = retry_metadata_startup(ctx, "list_buckets", || async {
+        store
+            .list_bucket(&BucketOptions {
+                no_metadata: true,
+                ..Default::default()
+            })
+            .await
+            .map_err(IoError::other)
+    })
+    .await?;

    let buckets: Vec<String> = buckets_list.into_iter().map(|v| v.name).collect();

-    try_migrate_bucket_metadata(store.clone()).await?;
+    retry_metadata_startup(ctx, "bucket_migration", || try_migrate_bucket_metadata(store.clone())).await?;
    init_on_demand_migration_runtime();
    init_bucket_metadata_sys(store.clone(), buckets.clone()).await;
-    try_migrate_iam_config(store).await?;
+    retry_metadata_startup(ctx, "iam_migration", || try_migrate_iam_config(store.clone())).await?;
    spawn_bucket_resync_startup_reconcile(buckets.clone(), ctx.clone(), false);

    Ok(buckets)
 }

 pub(crate) async fn init_bucket_metadata_runtime(store: Arc<ECStore>, ctx: CancellationToken) -> IoResult<Vec<String>> {
-    let buckets_list = store
-        .list_bucket(&BucketOptions {
-            no_metadata: true,
-            ..Default::default()
-        })
-        .await
-        .map_err(IoError::other)?;
+    let buckets_list = retry_metadata_startup(&ctx, "list_buckets", || async {
+        store
+            .list_bucket(&BucketOptions {
+                no_metadata: true,
+                ..Default::default()
+            })
+            .await
+            .map_err(IoError::other)
+    })
+    .await?;

    let buckets: Vec<String> = buckets_list.into_iter().map(|v| v.name).collect();

-    try_migrate_bucket_metadata(store.clone()).await?;
+    retry_metadata_startup(&ctx, "bucket_migration", || try_migrate_bucket_metadata(store.clone())).await?;

-    retry_iam_config_migration(|| try_migrate_iam_config(store.clone())).await?;
+    retry_metadata_startup(&ctx, "iam_migration", || try_migrate_iam_config(store.clone())).await?;
    init_on_demand_migration_runtime();
    init_bucket_metadata_sys(store, buckets.clone()).await;
    spawn_bucket_resync_startup_reconcile(buckets.clone(), ctx, true);
@@ -95,45 +103,63 @@ pub(crate) async fn init_bucket_metadata_runtime(store: Arc<ECStore>, ctx: Cance
    Ok(buckets)
 }

-async fn retry_iam_config_migration<Operation, OperationFuture>(mut operation: Operation) -> IoResult<()>
+async fn retry_metadata_startup<T, Operation, OperationFuture>(
+    ctx: &CancellationToken,
+    stage: &'static str,
+    mut operation: Operation,
+) -> IoResult<T>
 where
    Operation: FnMut() -> OperationFuture,
-    OperationFuture: Future<Output = IoResult<()>>,
+    OperationFuture: Future<Output = IoResult<T>>,
 {
-    retry_iam_config_migration_with(&mut operation, IAM_MIGRATION_MAX_RETRIES, IAM_MIGRATION_RETRY_INTERVAL).await
+    retry_metadata_startup_with(ctx, stage, &mut operation, METADATA_STARTUP_MAX_RETRIES, METADATA_STARTUP_RETRY_INTERVAL).await
 }

-async fn retry_iam_config_migration_with<Operation, OperationFuture>(
+async fn retry_metadata_startup_with<T, Operation, OperationFuture>(
+    ctx: &CancellationToken,
+    stage: &'static str,
    operation: &mut Operation,
    max_retries: usize,
    retry_interval: Duration,
-) -> IoResult<()>
+) -> IoResult<T>
 where
    Operation: FnMut() -> OperationFuture,
-    OperationFuture: Future<Output = IoResult<()>>,
+    OperationFuture: Future<Output = IoResult<T>>,
 {
    let mut retries = 0;
    loop {
-        match operation().await {
-            Ok(()) => return Ok(()),
-            Err(error) if iam_migration_error_is_retryable(&error) && retries < max_retries => {
+        let result = tokio::select! {
+            biased;
+            _ = ctx.cancelled() => return Err(IoError::other(StorageError::OperationCanceled)),
+            result = operation() => result,
+        };
+        match result {
+            Ok(value) => return Ok(value),
+            Err(error) if metadata_startup_error_is_retryable(&error) && retries < max_retries => {
                retries += 1;
                tracing::warn!(
+                    event = EVENT_BUCKET_METADATA_STARTUP_RETRY,
                    component = LOG_COMPONENT_STARTUP_BUCKET_METADATA,
-                    subsystem = "iam_migration",
+                    subsystem = LOG_SUBSYSTEM_METADATA_STARTUP,
+                    state = "retrying",
+                    stage,
                    retry_count = retries,
                    max_retries,
                    error = %error,
-                    "IAM config migration hit a transient quorum error; retrying"
+                    "Metadata startup hit a transient quorum error"
                );
-                tokio::time::sleep(retry_interval).await;
+                tokio::select! {
+                    biased;
+                    _ = ctx.cancelled() => return Err(IoError::other(StorageError::OperationCanceled)),
+                    _ = tokio::time::sleep(retry_interval) => {}
+                }
            }
            Err(error) => return Err(error),
        }
    }
 }

-fn iam_migration_error_is_retryable(error: &IoError) -> bool {
+fn metadata_startup_error_is_retryable(error: &IoError) -> bool {
    error
        .get_ref()
        .and_then(|source| source.downcast_ref::<StorageError>())
@@ -331,9 +357,11 @@ mod tests {
    }

    #[tokio::test]
-    async fn iam_migration_retries_only_quorum_errors() {
+    async fn metadata_startup_retries_only_quorum_errors() {
        let mut attempts = 0;
-        retry_iam_config_migration_with(
+        retry_metadata_startup_with(
+            &CancellationToken::new(),
+            "bucket_migration",
            &mut || {
                attempts += 1;
                std::future::ready(if attempts == 1 {
@@ -353,10 +381,12 @@ mod tests {
        assert_eq!(attempts, 2);

        let mut deterministic_attempts = 0;
-        let error = retry_iam_config_migration_with(
+        let error = retry_metadata_startup_with(
+            &CancellationToken::new(),
+            "iam_migration",
            &mut || {
                deterministic_attempts += 1;
-                std::future::ready(Err(IoError::other("incompatible IAM metadata")))
+                std::future::ready(Err::<(), _>(IoError::other("incompatible IAM metadata")))
            },
            1,
            Duration::ZERO,
@@ -366,4 +396,89 @@ mod tests {
        assert_eq!(error.to_string(), "incompatible IAM metadata");
        assert_eq!(deterministic_attempts, 1);
    }
+
+    #[tokio::test(start_paused = true)]
+    async fn metadata_startup_preserves_quorum_error_when_retries_are_exhausted() {
+        let mut attempts = 0;
+        let error = retry_metadata_startup_with(
+            &CancellationToken::new(),
+            "bucket_migration",
+            &mut || {
+                attempts += 1;
+                std::future::ready(Err::<(), _>(IoError::other(StorageError::InsufficientReadQuorum(
+                    ".rustfs.sys".into(),
+                    "buckets/bucket/.metadata.bin".into(),
+                ))))
+            },
+            2,
+            Duration::from_secs(1),
+        )
+        .await
+        .expect_err("an unavailable authority must not become an empty metadata success");
+        assert_eq!(attempts, 3);
+        assert!(metadata_startup_error_is_retryable(&error));
+    }
+
+    #[test]
+    fn metadata_startup_never_retries_corruption_or_string_only_quorum_errors() {
+        for error in [
+            IoError::other(StorageError::FileCorrupt),
+            IoError::other(StorageError::OperationCanceled),
+            IoError::new(std::io::ErrorKind::PermissionDenied, "metadata access denied"),
+            IoError::other("insufficient read quorum"),
+        ] {
+            assert!(!metadata_startup_error_is_retryable(&error));
+        }
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn metadata_startup_cancels_inflight_operations_and_retry_waits() {
+        for pending_operation in [false, true] {
+            let ctx = CancellationToken::new();
+            let cancel = ctx.clone();
+            tokio::spawn(async move {
+                tokio::time::sleep(Duration::from_secs(1)).await;
+                cancel.cancel();
+            });
+            let error = retry_metadata_startup_with(
+                &ctx,
+                "bucket_migration",
+                &mut || async {
+                    if pending_operation {
+                        std::future::pending::<()>().await;
+                    }
+                    Err::<(), _>(IoError::other(StorageError::InsufficientReadQuorum(
+                        ".rustfs.sys".into(),
+                        "buckets/bucket/.metadata.bin".into(),
+                    )))
+                },
+                2,
+                Duration::from_secs(20),
+            )
+            .await
+            .expect_err("shutdown must interrupt both storage I/O and backoff");
+            assert!(matches!(
+                error.get_ref().and_then(|source| source.downcast_ref::<StorageError>()),
+                Some(StorageError::OperationCanceled)
+            ));
+        }
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn metadata_startup_does_not_cap_healthy_migration_duration() {
+        // Large bucket inventories can legitimately take longer than a fixed
+        // wall-clock deadline. Bound retry count, not a healthy migration.
+        retry_metadata_startup_with(
+            &CancellationToken::new(),
+            "bucket_migration",
+            &mut || async {
+                tokio::time::sleep(Duration::from_secs(120)).await;
+                Ok(())
+            },
+            2,
+            Duration::from_secs(1),
+        )
+        .await
+        .expect("healthy migration must not be aborted by a fleet readiness deadline");
+    }
 }
@@ -1712,16 +1712,17 @@ fn response_with_status(status: StatusCode, message: impl Into<String>) -> Respo
 }

 fn response_with_disk_error(error: &DiskError, message: impl Into<String>) -> Response<Body> {
-    let missing = match error {
+    let disk_error = match error {
        DiskError::FileNotFound => Some(rustfs_rio::INTERNODE_FILE_NOT_FOUND),
        DiskError::VolumeNotFound => Some(rustfs_rio::INTERNODE_VOLUME_NOT_FOUND),
+        DiskError::FileCorrupt => Some(rustfs_rio::INTERNODE_FILE_CORRUPT),
        _ => None,
    };
    let mut response = response_with_status(StatusCode::INTERNAL_SERVER_ERROR, message);
-    if let Some(missing) = missing {
+    if let Some(disk_error) = disk_error {
        response
            .headers_mut()
-            .insert(rustfs_rio::INTERNODE_DISK_ERROR_HEADER, HeaderValue::from_static(missing));
+            .insert(rustfs_rio::INTERNODE_DISK_ERROR_HEADER, HeaderValue::from_static(disk_error));
    }
    response
 }
@@ -3087,12 +3088,14 @@ mod tests {
    }

    #[test]
-    fn read_file_error_response_marks_only_missing_disk_errors() {
+    fn read_file_error_response_preserves_typed_disk_errors() {
        for (error, expected) in [
            (DiskError::FileNotFound, rustfs_rio::INTERNODE_FILE_NOT_FOUND),
            (DiskError::VolumeNotFound, rustfs_rio::INTERNODE_VOLUME_NOT_FOUND),
+            (DiskError::FileCorrupt, rustfs_rio::INTERNODE_FILE_CORRUPT),
        ] {
            let response = response_with_disk_error(&error, error.to_string());
+            assert_eq!(response.status(), StatusCode::INTERNAL_SERVER_ERROR);
            assert_eq!(
                response.headers().get(rustfs_rio::INTERNODE_DISK_ERROR_HEADER),
                Some(&HeaderValue::from_static(expected))
@@ -52,9 +52,11 @@ use serde::Deserialize;
 use sha2::{Digest, Sha256};
 use std::{
    collections::HashMap,
+    future::Future,
    io::Cursor,
    pin::Pin,
    sync::{Arc, LazyLock, OnceLock},
+    time::Instant,
 };
 use time::OffsetDateTime;
 use tokio::spawn;
@@ -78,6 +80,9 @@ const EVENT_RPC_REQUEST_FAILED: &str = "rpc_request_failed";
 const EVENT_RPC_RESPONSE_EMITTED: &str = "rpc_response_emitted";
 const EVENT_RPC_BACKGROUND_TASK_SPAWNED: &str = "rpc_background_task_spawned";
 const EVENT_RPC_BACKGROUND_TASK_FAILED: &str = "rpc_background_task_failed";
+const REBALANCE_START_FLEET_PROOF_MARKER: &str = "pool activation requires a live fleet capability proof";
+const REBALANCE_START_FLEET_PROOF_RETRY_BUDGET: Duration = Duration::from_secs(30);
+const REBALANCE_START_FLEET_PROOF_RETRY_DELAY: Duration = Duration::from_secs(5);
 const HEAL_CONTROL_REPLAY_CACHE_MAX_ENTRIES: usize = 4096;
 const TIER_MUTATION_PEER_STATE_UNSPECIFIED_WIRE: i32 = 0;
 const TIER_MUTATION_PEER_STATE_PREPARED_WIRE: i32 = 1;
@@ -501,6 +506,48 @@ fn background_rebalance_start_error_message(result: StorageResult<()>) -> Option
    result.err().map(|err| format!("start_rebalance failed: {err}"))
 }

+fn is_rebalance_start_fleet_proof_retryable(err: &Error) -> bool {
+    crate::storage::storage_api::ecstore_capacity::is_pool_activation_fleet_proof_error(err)
+        && err.to_string().contains(REBALANCE_START_FLEET_PROOF_MARKER)
+}
+
+async fn retry_rebalance_start_fleet_proof<F, Fut>(mut operation: F) -> StorageResult<()>
+where
+    F: FnMut() -> Fut,
+    Fut: Future<Output = StorageResult<()>>,
+{
+    let deadline = Instant::now() + REBALANCE_START_FLEET_PROOF_RETRY_BUDGET;
+    let mut attempt = 1_u32;
+
+    loop {
+        match operation().await {
+            Ok(()) => return Ok(()),
+            Err(err) if is_rebalance_start_fleet_proof_retryable(&err) => {
+                let now = Instant::now();
+                if now >= deadline {
+                    return Err(err);
+                }
+                let delay = REBALANCE_START_FLEET_PROOF_RETRY_DELAY.min(deadline.saturating_duration_since(now));
+                warn!(
+                    event = EVENT_RPC_BACKGROUND_TASK_FAILED,
+                    component = LOG_COMPONENT_STORAGE,
+                    subsystem = LOG_SUBSYSTEM_REBALANCE,
+                    operation = "start_rebalance",
+                    state = "fleet_proof_retry_scheduled",
+                    result = "retrying",
+                    attempt,
+                    retry_delay_ms = delay.as_millis(),
+                    error = %err,
+                    "node rpc background task retry"
+                );
+                tokio::time::sleep(delay).await;
+                attempt = attempt.saturating_add(1);
+            }
+            Err(err) => return Err(err),
+        }
+    }
+}
+
 fn stop_rebalance_response(result: StorageResult<()>) -> StopRebalanceResponse {
    match result {
        Ok(_) => StopRebalanceResponse {
@@ -2800,7 +2847,9 @@ impl Node for NodeService {

        if start_rebalance {
            log_background_rebalance_task_spawned!(start_rebalance);
-            if let Some(message) = background_rebalance_start_error_message(store.start_rebalance().await) {
+            if let Some(message) =
+                background_rebalance_start_error_message(retry_rebalance_start_fleet_proof(|| store.start_rebalance()).await)
+            {
                error!(
                    event = EVENT_RPC_BACKGROUND_TASK_FAILED,
                    component = LOG_COMPONENT_STORAGE,
@@ -2975,9 +3024,10 @@ mod tests {
        SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION, SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION, SCANNER_PUBLICATION_LEASE_TTL_MS,
        SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, STORAGE_CLASS_SUB_SYS, admit_heal_control_replay,
        background_rebalance_start_error_message, execute_heal_control_envelope_with_manager,
-        initialize_heal_topology_fingerprint, initialize_heal_topology_fingerprint_with_probe, legacy_scanner_activity_response,
-        make_heal_control_server, make_heal_control_server_with_cache, make_server, make_server_for_context,
-        make_tier_mutation_control_server_for_context, previous_scanner_activity_response, remove_heal_control_replay,
+        initialize_heal_topology_fingerprint, initialize_heal_topology_fingerprint_with_probe,
+        is_rebalance_start_fleet_proof_retryable, legacy_scanner_activity_response, make_heal_control_server,
+        make_heal_control_server_with_cache, make_server, make_server_for_context, make_tier_mutation_control_server_for_context,
+        previous_scanner_activity_response, remove_heal_control_replay, retry_rebalance_start_fleet_proof,
        scanner_activity_response_v7, start_decommission_failure_response, stop_rebalance_response,
        validate_admin_heal_control_start,
    };
@@ -8121,6 +8171,49 @@ mod tests {
        assert!(message.contains("boom"));
    }

+    #[test]
+    fn test_rebalance_start_retry_ignores_expired_fleet_proof() {
+        let expired = Error::other("pool activation fleet capability proof expired before commit");
+
+        assert!(!is_rebalance_start_fleet_proof_retryable(&expired));
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn test_retry_rebalance_start_waits_for_fleet_proof() {
+        use std::sync::{
+            Arc,
+            atomic::{AtomicUsize, Ordering},
+        };
+
+        let attempts = Arc::new(AtomicUsize::new(0));
+        let task = tokio::spawn({
+            let attempts = Arc::clone(&attempts);
+            async move {
+                retry_rebalance_start_fleet_proof(move || {
+                    let attempts = Arc::clone(&attempts);
+                    async move {
+                        let attempt = attempts.fetch_add(1, Ordering::SeqCst);
+                        if attempt == 0 {
+                            Err(Error::other(super::REBALANCE_START_FLEET_PROOF_MARKER))
+                        } else {
+                            Ok(())
+                        }
+                    }
+                })
+                .await
+            }
+        });
+
+        tokio::task::yield_now().await;
+        assert_eq!(attempts.load(Ordering::SeqCst), 1);
+
+        tokio::time::advance(super::REBALANCE_START_FLEET_PROOF_RETRY_DELAY).await;
+        task.await
+            .expect("retry task should not panic")
+            .expect("fleet proof retry should eventually succeed");
+        assert_eq!(attempts.load(Ordering::SeqCst), 2);
+    }
+
    #[test]
    fn test_stop_rebalance_response_reports_local_stop_error() {
        let response = stop_rebalance_response(Err(Error::other("boom")));
@@ -21,6 +21,12 @@ use std::time::Duration;

 use rand::RngExt as _;
 use rustfs_storage_api as storage_contracts;
+
+pub(crate) mod ecstore_integrity {
+    pub(crate) use rustfs_ecstore::api::integrity::{
+        IntegrityError, JobRequest, control_job, create_job, get_job, inventory, readiness, resume_job,
+    };
+}
 use tokio::sync::{Mutex, OwnedMutexGuard};
 use tokio_util::sync::CancellationToken;

@@ -90,6 +90,7 @@ their issue closes.
 | `test_build_rustfs_options.sh` | dev-tool | Shell test for rustfs build-option wiring | `make test` (script-tests) |
 | `test_entrypoint_credentials.sh` | dev-tool | Container entrypoint credential-handling test | `make test` (script-tests) |
 | `test_helm_chart_version.sh` | dev-tool | Test for `helm_chart_version.sh` | — |
+| `test_package_service_scripts.sh` | ci-gate | Verifies DEB/RPM install, upgrade, removal, and service restart scriptlets | audit.yml `workflow-pin-report` |
 | `test_package_versions.sh` | ci-gate | Exact-output and fail-closed tests for DEB/RPM package version normalization | audit.yml `workflow-pin-report`; package.yml RPM build |
 | `windows-sftp-listener-smoke.sh` | dev-tool | Confirms `rustfs.exe --features sftp` binds an SFTP listener on Windows | — |

@@ -2277,7 +2277,7 @@ fi
    rustfs/src/admin/router.rs \
    rustfs/src/admin/console.rs \
    rustfs/src/admin/handlers/heal.rs \
-    rustfs/src/admin/handlers/metrics.rs \
+    rustfs/src/admin/handlers/realtime.rs \
    rustfs/src/admin/handlers/object_zip_download.rs || true
 ) >"$RUSTFS_APP_ADMIN_STORAGE_API_BYPASS_HITS_FILE"

@@ -50,7 +50,7 @@
 1|crates/ecstore/src/services/rebalance/entry.rs
 8|crates/ecstore/src/services/rebalance/meta.rs
 8|crates/ecstore/src/services/rebalance/runtime.rs
-18|crates/ecstore/src/services/rebalance/worker.rs
+17|crates/ecstore/src/services/rebalance/worker.rs
 33|crates/ecstore/src/services/tier/tier.rs
 1|crates/ecstore/src/services/tier/tier_config.rs
 1|crates/ecstore/src/services/tier/warm_backend.rs
@@ -28,7 +28,12 @@ def current_chain():
    require(sha(chain["workflow_sha"]) and chain["workflow_sha"] == os.environ["GITHUB_SHA"], "chain workflow source mismatch")
    head = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
    require(head == chain["workflow_sha"], "lane checkout differs from chain workflow source")
-    require(chain["testing_sha"] == (ROOT / ".config/functional-script-revision.txt").read_text().strip() and sha(chain["testing_sha"]), "private script pin differs from chain")
+    # testing_sha is either the committed pin or auto-testing main HEAD via
+    # resolve_functional_candidate.py's >24h staleness fallback, so pin
+    # equality is no longer an invariant (the 09-21 chain died on exactly
+    # that check once the fallback finally fired). Lanes check out exactly
+    # this sha, which is what the format check guards.
+    require(sha(chain["testing_sha"]), "private script revision is not a valid commit sha")
    candidate = chain["candidate"]
    require(isinstance(candidate, dict) and set(candidate) == {"manifest", "artifact_id", "artifact_digest", "workflow_sha", "workflow_ref", "build_started_at"}, "invalid candidate envelope")
    manifest = candidate["manifest"]
@@ -32,8 +32,9 @@ def validate_summary(summary, run):
    candidate = chain["candidate"]
    manifest = candidate["manifest"]
    require(resolve(manifest["build_run_id"], manifest["build_run_attempt"]) == candidate, "producer candidate identity changed")
-    config = api(f"repos/{REPOSITORY}/contents/.config/functional-script-revision.txt?ref={run['head_sha']}")
-    require(base64.b64decode(config["content"]).decode().strip() == chain["testing_sha"], "private pin differs from workflow source")
+    # testing_sha may legitimately be auto-testing main HEAD via the prepare
+    # step's >24h staleness fallback (checked for sha format above), so pin
+    # equality is not an invariant; drop the pin-file comparison.
    completed = timestamp(summary["completed_at"])
    require(timestamp(run["run_started_at"]) <= completed <= datetime.now(timezone.utc) + timedelta(minutes=5), "invalid completion timestamp")
    source_ref = manifest.get("source_ref", candidate["workflow_ref"])
@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+case ${1:-} in
+  after-install)
+    cat <<'SCRIPT'
+#!/bin/sh
+set -e
+if ! getent passwd rustfs > /dev/null 2>&1; then
+    useradd -r -s /bin/false -d /opt/rustfs rustfs
+fi
+mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
+chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
+if [ -d /run/systemd/system ]; then
+    systemctl daemon-reload
+    if [ "${1:-}" = configure ]; then
+        marker=/run/rustfs-package-upgrade-was-active
+        if [ -z "${2:-}" ]; then
+            rm -f "$marker"
+        else
+            # Versions published before this fix had no upgrade-aware prerm:
+            # the old prerm stopped the service and left no marker, so the
+            # marker check below cannot see it. Fall back to restarting for
+            # upgrades from those versions. NOTE: these are dpkg control-file
+            # versions - package_versions.sh maps the SemVer prerelease "-"
+            # to "~", so 1.0.0-rc.5 was published as 1.0.0~rc.5. Drop this list once
+            # upgrades from <= 1.0.0 no longer need support.
+            case $2 in
+                1.0.0|1.0.0~rc.[1-6]|1.0.0~rc.[1-6]~preview.*) legacy_upgrade=true ;;
+                *) legacy_upgrade=false ;;
+            esac
+            if [ -f "$marker" ] || [ "$legacy_upgrade" = true ]; then
+                rm -f "$marker"
+                if command -v deb-systemd-invoke > /dev/null 2>&1; then
+                    deb-systemd-invoke restart rustfs.service
+                else
+                    systemctl restart rustfs.service
+                fi
+            fi
+        fi
+    fi
+fi
+SCRIPT
+    ;;
+  before-remove)
+    cat <<'SCRIPT'
+#!/bin/sh
+set -e
+case ${1:-} in
+    upgrade)
+        marker=/run/rustfs-package-upgrade-was-active
+        rm -f "$marker"
+        if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
+            touch "$marker"
+        fi
+        exit 0
+        ;;
+    0|remove|deconfigure) ;;
+    *) exit 0 ;;
+esac
+if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
+    systemctl stop rustfs
+fi
+SCRIPT
+    ;;
+  after-remove)
+    cat <<'SCRIPT'
+#!/bin/sh
+set -e
+if [ -d /run/systemd/system ]; then
+    systemctl daemon-reload
+fi
+if [ "${1:-}" != upgrade ]; then
+    rm -f /run/rustfs-package-upgrade-was-active
+fi
+SCRIPT
+    ;;
+  rpm-before-install)
+    cat <<'SCRIPT'
+#!/bin/sh
+set -e
+rm -f /run/rustfs-package-upgrade-was-active
+SCRIPT
+    ;;
+  rpm-before-upgrade)
+    cat <<'SCRIPT'
+#!/bin/sh
+set -e
+marker=/run/rustfs-package-upgrade-was-active
+rm -f "$marker"
+if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
+    touch "$marker"
+fi
+SCRIPT
+    ;;
+  rpm-posttrans)
+    cat <<'SCRIPT'
+#!/bin/sh
+set -e
+marker=/run/rustfs-package-upgrade-was-active
+if [ -f "$marker" ]; then
+    rm -f "$marker"
+    systemctl restart rustfs
+fi
+SCRIPT
+    ;;
+  *)
+    echo "usage: $0 {after-install|before-remove|after-remove|rpm-before-install|rpm-before-upgrade|rpm-posttrans}" >&2
+    exit 2
+    ;;
+esac
@@ -5,6 +5,7 @@ from __future__ import annotations
 import hashlib
 import io
 import json
+from datetime import datetime, timezone
 import os
 from pathlib import Path
 import re
@@ -112,6 +113,25 @@ def prepare():
    candidate = resolve(run_id, attempt)
    revision = (ROOT / ".config/functional-script-revision.txt").read_text().strip()
    require(sha(revision), "private test script revision must be pinned")
+    # The revision file is refreshed only by release syncs and goes stale
+    # within days, so the nightly chain keeps replaying an old harness while
+    # fixes (conffile prompts, FT --results-file, tier retries, STS-105
+    # signing) sit untested on auto-testing main. When the pinned revision is
+    # older than a day, fall back to auto-testing main HEAD. Any probe
+    # failure keeps the pinned revision - the chain still runs, just on the
+    # old harness.
+    try:
+        head = api("repos/rustfs/auto-testing/git/ref/heads/main")["object"]["sha"]
+        pinned_date = api(f"repos/rustfs/auto-testing/commits/{revision}")["commit"]["committer"]["date"]
+        pinned_at = datetime.fromisoformat(pinned_date.replace("Z", "+00:00"))
+        age_hours = (datetime.now(timezone.utc) - pinned_at).total_seconds() / 3600
+        if age_hours > 24:
+            print(f"functional-script revision {revision[:8]} is {age_hours:.0f}h old; falling back to auto-testing main HEAD {head[:8]}")
+            revision = head
+        else:
+            print(f"functional-script revision {revision[:8]} is {age_hours:.0f}h old; keeping pin")
+    except Exception as error:
+        print(f"staleness probe failed ({error}); keeping pinned revision {revision[:8]}")
    chain = {"schema": 1, "run_id": int(os.environ["GITHUB_RUN_ID"]), "attempt": int(os.environ["GITHUB_RUN_ATTEMPT"]),
             "workflow_sha": os.environ["GITHUB_SHA"], "testing_sha": revision, "candidate": candidate}
    encoded = json.dumps(chain, sort_keys=True, separators=(",", ":"))
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
cxymds	1880b42169	fix(heal): preserve decode repair signals and retry startup quorum (#8067 ) * fix(startup): retry transient bucket metadata quorum failures * fix(ecstore): retain recovered shard damage for read repair * fix(ecstore): reconnect missing read sources before GET * test(heal): prove exact marker replay preserves history * fix(ecstore): fence reconnect retries at the deadline Reject expired reconnect waiters before dispatch and start cooldown on sweep completion, including timeout and cancellation. Cover exact-deadline admission independently of cooldown and preserve the original strict RPC-count regression.	2026-09-22 14:58:31 +00:00
Chris	3f549e26f3	fix(ci): publish version-tagged preview Docker images (#8068 )	2026-09-22 13:06:23 +00:00
Dae-Cheol Noh	3c6c88b2e7	fix(amqp): emit S3 events directly in notification records (#8066 ) * fix(amqp): emit S3 events directly in notification records * test(amqp): avoid queued timestamp equality race	2026-09-22 12:16:40 +00:00
Dae-Cheol Noh	1f04a12abf	fix(ftps): bound upload memory with multipart streaming (#8064 ) * fix(ftps): bound upload memory with multipart streaming * fix(ftps): keep multipart upload within driver module	2026-09-22 12:15:20 +00:00
cxymds	122abfaae2	feat(integrity): add inventory, audit, and protected migration (#8065 ) * feat(integrity): add inventory, audit, and protected migration * refactor(admin): use gateway facade for integrity handlers * fix(integrity): sort fingerprint metadata explicitly	2026-09-22 19:23:41 +08:00
cxymds	d0ce2f758b	fix(ecstore): bound decommission target gate contention (#8061 ) * fix(ecstore): bound decommission target gate contention Target capacity gate contention during pool decommission escalated a per-object transient into a durable bucket pause: each contended object failed its bucket entry, which re-ran the whole bucket listing and amplified attempts on the same objects. - centralize the decommission capacity failure classification so gate contention, benign contention and fatal failures are decided once - retry target-gate contention inline (bounded, jittered) before a mutation is admitted, covering put, part, complete, new-multipart and abort - defer contended entries to the end of the round instead of failing the bucket entry, and require the deferred set to drain before a set completes - treat a missing object or version, an overwrite and a superseded upload id as benign contention that is neither counted as a failure nor escalated - use full-jitter exponential backoff, capped, for decommission retries - expose the capacity pause reason, the waiting reason and a cumulative pause count in the admin pool status, plus gate-retry and per-object attempt metrics Related: rustfs/backlog#2644 * fix(ecstore): correct deferred replay and metadata compatibility	2026-09-22 19:20:05 +08:00
hector	9c30cc8851	feat(ci): on-demand fault-tolerance matrix sweep workflow (#8058 ) Adds rustfs-fault-tolerance-matrix.yml, a manually dispatched workflow that runs the --matrix topology x EC outage sweep from rustfs/auto-testing (38 combinations, 320 cases). The scenario suite hardcodes --all and a 60-minute budget, so the exhaustive sweep had no CI entry point.	2026-09-22 13:22:18 +08:00
Hauser	0ff06757c8	fix(obs): support custom OTLP CA bundles (#8060 ) Add RUSTFS_OBS_TLS_CA_FILE for additive private CA trust in OTLP HTTP exporters. Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-09-22 13:21:44 +08:00
cxymds	e12100b4a1	fix(s3): emit x-amz-expiration as HTTP-date (#8057 ) The expiry-date field of the x-amz-expiration response header was rendered as RFC 3339 (e.g. 2026-10-02T00:00:00Z). AWS SDKs parse this value with an RFC 822 parser; the Java SDK v1 rejects the ISO-8601 shape, drops the whole header, and logs a WARN from ObjectExpirationHeaderHandler. S3 specifies HTTP-date (RFC 1123 with a literal GMT zone), which is also what MinIO emits. Render via the shared HTTP-date formatter (extracted alongside format_expires_header) and pin the exact output in the unit test.	2026-09-22 11:18:59 +08:00
hector	d7c42d4099	fix(ci): accept the testing-sha staleness fallback in chain evidence (#8056 ) The 09-22 nightly chain failed all 12 lanes in seconds at the 'Bind functional candidate' step: ValueError: private script pin differs from chain resolve_functional_candidate.py's >24h staleness fallback (added by #8026, made functional by #8041's token fix) legitimately sets manifest.testing_sha to auto-testing main HEAD, but current_chain() still required it to equal .config/functional-script-revision.txt - a check written for the pre-fallback world where the two could never diverge. Once the fallback finally fired, prepare produced testing_sha 21edcf4 while the pin file still holds 27e9584 and every lane aborted before checking out the test scripts. Drop the pin-file comparison and keep what the lane actually needs to guarantee: testing_sha is a valid commit sha (current_chain), the lane checked out exactly that sha (record: private_head == testing_sha, kept as-is), and the health checker validates the same format instead of re-reading the pin file. Tests updated: a fallback testing_sha that differs from the pin is accepted; a non-sha testing_sha is rejected. Verified: python3 -m unittest test_functional_chain test_functional_chain_health -> 39 tests OK.	2026-09-22 10:25:36 +08:00
hector	4b0118520e	test(ci): align FT workflow contract tests with scenario E (44 cases) (#8054 ) #8052 renamed the fault-tolerance step to 'Run fault-tolerance scenarios (A, B, C, C2, D, E)' and raised the evidence gate to 44 cases, but scripts/test_security_workflow.py still expected the old step name and a 38-case fixture, breaking Quick Checks with three KeyErrors and one assertion failure. Update DIRECT_TESTS and the write_result fixture to the new contract.	2026-09-22 00:18:40 +08:00
cxymds	1afaf4491e	fix(ecstore): bound decommission capacity gate waits (#8044 ) * fix(ecstore): bound decommission capacity gate waits * fix(ecstore): keep rebalance meta save lock errors retryable The rebalance metadata retry policy only retries when the source chain still carries the typed lock error, but resolve_rebalance_meta_save_result collapsed every failure into a plain string. A transient rebalance.bin write-lock timeout therefore bypassed retry_rebalance_metadata_access entirely and surfaced as a hard failure of the periodic and terminal rebalance metadata saves. Wrap the failure with the data-movement stage context instead, which keeps the original lock error reachable through rebalance_error_source so the existing transient lock policy applies. The rendered message is unchanged. Add a regression test that feeds the wrapped meta save lock timeout through the retry helper and asserts the retry engages. * chore(ecstore): refresh error format ratchet baseline Removing the string-wrapped meta save failure drops one `::other(format!` call site in the rebalance worker, so the shrink-only baseline must be regenerated in the same change.	2026-09-21 20:56:26 +08:00
hector	bdea44f332	fix(ci): align FT evidence contract with scenario E (44 cases) (#8052 ) The fault-tolerance suite now registers and runs scenario E (rustfs/auto-testing#100): --all covers A, B, C, C2, D, E with 38 + 6 = 44 expected cases. The chain-evidence assertion still hardcoded [A, B, C, C2, D] and 38 everywhere, which would fail every green FT run's evidence validation once E is registered. Bump the scenario list to include E and all count assertions from 38 to 44; step name updated to match. No other lanes reference 38.	2026-09-21 20:13:56 +08:00
cxymds	8ba3309624	fix(rebalance): retry fleet proof before start activation (#8049 ) Retry the transient missing fleet capability proof in both the admin start flow and node RPC activation. Return a retryable 503 with Retry-After after rollback when readiness does not converge.	2026-09-21 11:27:58 +00:00
cxymds	cd2fa50e50	fix: rename admin metrics endpoint to realtime (#8046 )	2026-09-21 19:23:58 +08:00
cxymds	077b961232	fix(obs): default span events to none and gate span list by level (#8047 ) JSON log layers synthesized span lifecycle events for every span and embedded the full ancestor span list on every record. With INFO spans on object hot paths that produced multi-megabyte lines, which journald truncated at 48 KiB: the payload was silently dropped and the volume itself became the dominant load. Add RUSTFS_OBS_LOG_SPAN_EVENTS (none\|close\|full) with a default of none, and keep with_span_list enabled only for verbose (debug/trace) levels so the ancestor chain stops being re-serialized per record. The request_id promotion path reads the span scope, not the span list, so it is unaffected. Refs rustfs/backlog#2642	2026-09-21 19:15:39 +08:00
cxymds	e1bc731331	fix(lifecycle): reject empty expiration actions (#8045 ) * fix(lifecycle): reject empty expiration actions * style: apply rustfmt to lifecycle validation	2026-09-21 15:43:07 +08:00
cxymds	bb43ef861a	fix(decommission): use owned data for source capacity (#8038 )	2026-09-21 15:29:30 +08:00
Chris	4047ed6c8d	fix(heal): restore bucket metadata before replacement completion (#8042 )	2026-09-21 12:18:06 +08:00
hector	ae6bbaef27	fix(ci): give the chain staleness probe a token that can read auto-testing (#8041 ) resolve_functional_candidate.py probes rustfs/auto-testing (private) to age the pinned functional-script revision and fall back to main HEAD after 24h. The prepare step passed github.token, which cannot see the private repo, so every nightly chain logged staleness probe failed (...exit status 1.); keeping pinned revision and replayed the 09-14 harness. On the 09-20 nightly that harness died on the dpkg conffile prompt in all 12 lanes (see rustfs/auto-testing#97) because the --force-confold and other fixes never reached the chain. Use PF_TESTING_GH_TOKEN - already required by the other steps in this workflow - so the probe can actually run and the >24h fallback works.	2026-09-21 11:08:41 +08:00
cxymds	a7c875b48f	fix(heal): preserve remote corruption and reject incomplete deep scans (#8032 ) * fix(heal): reject truncated encoded shards * test(heal): cover truncated shards across integrity modes (#8035) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(heal): preserve remote corruption and reject incomplete deep scans * test(admin): tolerate config lock timeout while polling * fix(test): use admin storage error facade --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com>	2026-09-21 02:33:03 +00:00
マルコメ	42dd76d3b2	ci(docker): sync the Docker Hub overview from README.md on release (#8034 ) Docker Hub's overview is a separate `full_description` field that `docker push` never touches, so it had drifted into an 8 KB snapshot of an old README.md that still linked to https://docs.rustfs.com/introduction.html (now 404). Add a `sync-dockerhub-description` job to docker.yml that runs after the images are pushed and republishes README.md from the same commit via peter-evans/dockerhub-description (pinned to v5.0.0). It reuses the existing DOCKERHUB_USERNAME / DOCKERHUB_TOKEN credentials, so no new secrets are needed. Relative links (docs/, CONTRIBUTING.md) are rewritten to github.com URLs so they resolve on Docker Hub. Docker Hub caps the field at 25,000 bytes and the action truncates to fit with only a warning; README.md is at 22,879 bytes today. Read the published overview back after the sync and fail the job if it hit the cap, so a truncated overview cannot be published silently. Fixes #7995 Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Hauser <housemecn@gmail.com>	2026-09-21 07:04:47 +08:00
Chris	9aeecad4a9	docs(security): add encrypted ETag advisory lesson (#8040 )	2026-09-20 23:52:47 +08:00
Hauser	3738b32df1	chore(deps): update flake.lock (#8033 ) Flake lock file updates: • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/aff8a0b' (2026-09-10) → 'github:NixOS/nixpkgs/a32edd7' (2026-09-17) • Updated input 'rust-overlay': 'github:oxalica/rust-overlay/228ecef' (2026-09-12) → 'github:oxalica/rust-overlay/26a71e6' (2026-09-19)	2026-09-20 14:54:45 +08:00
Hauser	d997b07505	fix(get): skip futile resume for single-disk objects (#8031 ) * chore: pin s3s to upstream git revision Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4. Allow the official s3s git source in cargo-deny so source checks continue to pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(get): skip futile resume for single-disk objects Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-09-20 13:25:08 +08:00
Chris	0632a64f91	fix(ecstore): allow server-side copy at normal read quorum (#8029 )	2026-09-20 12:36:53 +08:00
hector	cbf6b02905	ci(chain): fall back to auto-testing main HEAD when the pinned harness goes stale (#8026 ) The functional chain pins the test harness to the revision recorded in .config/functional-script-revision.txt, which is only refreshed by release syncs - it went stale on Sep 16 and the nightly chain has been replaying a Sep-17-era harness ever since. Last night that meant: the performance lane died on a dpkg conffile prompt (fix merged as #88 but never reached the chain), the FT lane died on 'unknown option: --results-file' (harness/workflow version mismatch), and the STS-105 signing fix (#94) sat untested. When the pinned revision's commit is older than 24 hours, fall back to auto-testing main HEAD so the chain always runs the current harness. Any staleness-probe failure keeps the pinned revision (fail-safe). The revision file stays in place as the audit trail and the release sync continues to manage it.	2026-09-20 11:27:56 +08:00
hector	ed2c3eaf77	fix(package): preserve service state across upgrades (#8024 ) * fix(package): preserve service state across upgrades * fix(package): match legacy DEB versions in tilde form Published prerelease packages carry ~ in the dpkg control version (package_versions.sh maps the SemVer prerelease - to ~), so the legacy fallback list written with dots never matched 1.0.0~rc.x and upgrades away from those DEBs still left the service stopped (#8011). Fix the legacy glob to the tilde form and update the contract test, which had enshrined the dot form. Verified on Ubuntu 24.04 systemd containers: DEB upgrade 1.0.0~rc.5 -> 1.0.1 now keeps the service running; 1.0.0 -> 1.0.1 and 1.0.1 -> 1.0.2 marker path still pass. * docs(package): expand /etc/default/rustfs example template Document the commonly used RUSTFS_* settings as commented examples in the packaged conffile and point to docs.rustfs.com.	2026-09-20 11:27:41 +08:00