docs(obs): add Grafana server-label dashboard (#5468)

Co-authored-by: heihutu <heihutu@gmail.com>

.agents/skills/arch-checks/SKILL.md

@@ -10,10 +10,16 @@ never weaken a check to get green.
 
 ## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
 
-Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
-upward imports. Known legacy violations live in
+Enforces `composition (server, startup/init) → interface (admin,
+storage/ecfs, storage/s3_api) → app → infra`; no upward imports. Server source
+files are composition roots, while imports of their exported HTTP contracts
+are classified as interface dependencies. Known legacy violations live in
 `scripts/layer-dependency-baseline.txt`.
 
+Dedicated `*_test.rs` and `tests/` modules are outside this production guard.
+Inline `#[cfg(test)]` imports remain checked under their source file's layer;
+move architecture-crossing test scaffolding into a dedicated test module.
+
 - **New violation**: restructure your change so the dependency points
   downward (move the shared type/function to the lower layer).
 - **You legitimately removed a baseline entry**: run

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,19 +1,21 @@
 ---
 name: rustfs-release-publish
-description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
+description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
 ---
 # RustFS Release Publish (preview-validated pipeline)
 
 This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
 
-Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
+Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
 
 Pipeline shape:
 
 ```
-bump version files to <target> (final version, ONE commit) -> merge
+check console main against its latest Release
+  -> if ahead: publish console -> wait for Release asset + latest API
+  -> bump RustFS version files to <target> (final version, ONE commit) -> merge
   -> tag <preview-tag> at that commit -> CI green
-  -> verify release artifacts -> run binary locally + console checks
+  -> verify preview Release assets -> run binary locally + console checks
   -> validate with latest rc client
   -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
 ```
@@ -23,7 +25,7 @@ On validation failure: fix lands on main via normal PR (version files are alread
 ## Required inputs
 
 - Final target version, for example `1.0.0-beta.10`.
-- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
+- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
 
 If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
 
@@ -45,14 +47,18 @@ Rules:
 
 ## Preview tag naming
 
-- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
-- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
+- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
+- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
+- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
 
 ## Hard rules
 
 - Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
+- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
 - Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
 - The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
+- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
+- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
 - Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
 - If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
 - User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
@@ -63,6 +69,61 @@ Rules:
 - `gh auth status` works; confirm you can view `gh release list -L 3`.
 - Confirm the exact final target version with the user if not explicit.
 
+### Console release gate
+
+Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
+
+1. Read the latest published Console tag and compare it with Console `main`:
+
+```bash
+CONSOLE_REPO="rustfs/console"
+CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
+gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
+  --jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
+```
+
+- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
+- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
+- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
+
+2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
+
+```bash
+CONSOLE_SCRATCH=$(mktemp -d)
+gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
+git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
+CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
+git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
+gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
+```
+
+If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
+
+```bash
+git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
+git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
+```
+
+Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
+
+3. Find the exact tag run and wait for completion:
+
+```bash
+gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
+gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
+```
+
+4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
+
+```bash
+gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
+test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
+test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
+  --jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
+```
+
+Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
+
 ## Phase 1 — Version bump to the final target (once)
 
 - If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
@@ -85,16 +146,17 @@ git push origin "<preview-tag>"
 
 Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
 
+The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
+
 On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
 
-## Phase 3 — CI and artifact verification
+## Phase 3 — CI and preview Release verification
 
-- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
-- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
-  - `isPrerelease` must be `true`.
-  - Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
-- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
-- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
+- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
+- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
+- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
+- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
+- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
 
 ## Phase 4 — Run the artifact locally, verify the console
 
@@ -157,13 +219,15 @@ git push origin "<target>"
 ```
 
 - CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
+- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
+- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
 - Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
 
 ## Output contract
 
 Always report:
 
+- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
 - Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
+- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
 - Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -18,7 +18,7 @@ Validated baseline: release pattern used in PR `#2957`.
 
 If target version is missing or ambiguous, stop and ask before editing.
 
-Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
+Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
 
 ## Read before editing
 

.github/workflows/audit.yml

@@ -23,6 +23,8 @@ on:
       - 'deny.toml'
       - '.github/actions/**'
       - '.github/workflows/**'
+      - 'scripts/release/create_or_update_release.sh'
+      - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
@@ -33,6 +35,8 @@ on:
       - 'deny.toml'
       - '.github/actions/**'
       - '.github/workflows/**'
+      - 'scripts/release/create_or_update_release.sh'
+      - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
   schedule:
     - cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
@@ -96,6 +100,9 @@ jobs:
       - name: Report unpinned GitHub Actions
         run: ./scripts/security/check_workflow_pins.sh --enforce
 
+      - name: Check preview release workflow policy
+        run: ./scripts/security/check_preview_release_workflow.sh
+
   dependency-review:
     name: Dependency Review
     runs-on: ubuntu-latest

.github/workflows/build.yml

@@ -107,13 +107,21 @@ jobs:
 
           # Determine build type based on trigger
           if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
-            # Tag push - release or prerelease
+            # Tag push - preview, release, or prerelease
             should_build=true
             tag_name="${GITHUB_REF#refs/tags/}"
             version="${tag_name}"
 
-            # Check if this is a prerelease
-            if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
+            # Preview tags publish a GitHub prerelease for validation, but
+            # must not update any latest channel.
+            if [[ "$tag_name" =~ -preview\.[0-9]+$ ]]; then
+              build_type="preview"
+              is_prerelease=true
+              echo "🔍 Preview build detected: $tag_name"
+            elif [[ "$tag_name" == *"-preview"* ]]; then
+              echo "❌ Invalid preview tag: $tag_name (expected suffix: -preview.<number>)" >&2
+              exit 1
+            elif [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
               build_type="prerelease"
               is_prerelease=true
               echo "🚀 Prerelease build detected: $tag_name"
@@ -714,6 +722,10 @@ jobs:
           echo ""
 
           case "$BUILD_TYPE" in
+            "preview")
+              echo "🔍 Preview artifacts are published in a GitHub prerelease"
+              echo "⏭️  Preview releases do not update latest channels"
+              ;;
             "development")
               echo "🛠️  Development build artifacts have been uploaded to OSS dev directory"
               echo "⚠️  This is a development build - not suitable for production use"
@@ -732,7 +744,9 @@ jobs:
 
           echo ""
           echo "🐳 Docker Images:"
-          if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
+          if [[ "$BUILD_TYPE" == "preview" ]]; then
+            echo "⏭️  Preview tags do not publish Docker images"
+          elif [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
             echo "⏭️  Docker image build was skipped (binary only build)"
           elif [[ "$BUILD_STATUS" == "success" ]]; then
             echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -740,11 +754,11 @@ jobs:
             echo "❌ Docker image build will be skipped due to build failure"
           fi
 
-  # Create GitHub Release (only for tag pushes)
+  # Create GitHub Release for every valid release tag, including previews
   create-release:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
-    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
+    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
     runs-on: ubuntu-latest
     permissions:
       contents: write
@@ -767,9 +781,12 @@ jobs:
           VERSION="${{ needs.build-check.outputs.version }}"
           IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+          TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
 
           # Determine release type for title
-          if [[ "$BUILD_TYPE" == "prerelease" ]]; then
+          if [[ "$BUILD_TYPE" == "preview" ]]; then
+            RELEASE_TYPE="preview"
+          elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
             if [[ "$TAG" == *"alpha"* ]]; then
               RELEASE_TYPE="alpha"
             elif [[ "$TAG" == *"beta"* ]]; then
@@ -783,54 +800,24 @@ jobs:
             RELEASE_TYPE="release"
           fi
 
-          # Check if release already exists
-          if gh release view "$TAG" >/dev/null 2>&1; then
-            echo "Release $TAG already exists"
-            RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
-            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
+          # Create release title
+          if [[ "$IS_PRERELEASE" == "true" ]]; then
+            TITLE="RustFS $VERSION (${RELEASE_TYPE})"
           else
-            # Get release notes from tag message
-            RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
-            if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
-              if [[ "$IS_PRERELEASE" == "true" ]]; then
-                RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
-              else
-                RELEASE_NOTES="Release ${VERSION}"
-              fi
-            fi
-
-            # Create release title
-            if [[ "$IS_PRERELEASE" == "true" ]]; then
-              TITLE="RustFS $VERSION (${RELEASE_TYPE})"
-            else
-              TITLE="RustFS $VERSION"
-            fi
-
-            # Create the release
-            PRERELEASE_FLAG=""
-            if [[ "$IS_PRERELEASE" == "true" ]]; then
-              PRERELEASE_FLAG="--prerelease"
-            fi
-
-            gh release create "$TAG" \
-              --title "$TITLE" \
-              --notes "$RELEASE_NOTES" \
-              $PRERELEASE_FLAG \
-              --draft
-
-            RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
-            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
+            TITLE="RustFS $VERSION"
           fi
 
-          echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
-          echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
-          echo "Created release: $RELEASE_URL"
+          ./scripts/release/create_or_update_release.sh \
+            "$TAG" \
+            "$TARGET_COMMITISH" \
+            "$TITLE" \
+            "$IS_PRERELEASE"
 
   # Prepare and upload release assets
   upload-release-assets:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
-    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
+    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
     runs-on: ubuntu-latest
     permissions:
       contents: write
@@ -920,8 +907,8 @@ jobs:
   # the pointed-to version is a prerelease.
   update-latest-version:
     name: Update Latest Version
-    needs: [ build-check, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/')
+    needs: [ build-check, publish-release ]
+    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
     runs-on: ubuntu-latest
     steps:
       - name: Update latest.json
@@ -980,51 +967,33 @@ jobs:
   publish-release:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
+    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
     runs-on: ubuntu-latest
     permissions:
       contents: write
     steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Update release notes and publish
+      - name: Publish release
         env:
           GH_TOKEN: ${{ github.token }}
         shell: bash
         run: |
           TAG="${{ needs.build-check.outputs.version }}"
-          VERSION="${{ needs.build-check.outputs.version }}"
-          IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
+          RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
 
-          # Determine release type
-          if [[ "$BUILD_TYPE" == "prerelease" ]]; then
-            if [[ "$TAG" == *"alpha"* ]]; then
-              RELEASE_TYPE="alpha"
-            elif [[ "$TAG" == *"beta"* ]]; then
-              RELEASE_TYPE="beta"
-            elif [[ "$TAG" == *"rc"* ]]; then
-              RELEASE_TYPE="rc"
-            else
-              RELEASE_TYPE="prerelease"
-            fi
+          # Publish the release and correct its channel state on retries.
+          # Only a stable final release may become GitHub Latest.
+          if [[ "$BUILD_TYPE" == "release" ]]; then
+            gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
+              -F draft=false \
+              -F prerelease=false \
+              -f make_latest=true >/dev/null
           else
-            RELEASE_TYPE="release"
+            gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
+              -F draft=false \
+              -F prerelease=true \
+              -f make_latest=false >/dev/null
           fi
 
-          # Get original release notes from tag
-          ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
-          if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
-            if [[ "$IS_PRERELEASE" == "true" ]]; then
-              ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
-            else
-              ORIGINAL_NOTES="Release ${VERSION}"
-            fi
-          fi
-
-          # Publish the release (remove draft status)
-          gh release edit "$TAG" --draft=false
-
           echo "🎉 Released $TAG successfully!"
           echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"

.github/workflows/docker.yml

@@ -82,7 +82,8 @@ jobs:
       github.event_name == 'workflow_dispatch' ||
       (github.event.workflow_run.conclusion == 'success' &&
        github.event.workflow_run.event == 'push' &&
-       github.event.workflow_run.head_branch != 'main')
+       github.event.workflow_run.head_branch != 'main' &&
+       !contains(github.event.workflow_run.head_branch, '-preview'))
     runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
@@ -220,6 +221,13 @@ jobs:
                 create_latest=true
                 echo "🚀 Building with latest stable release version"
                 ;;
+              *-preview*)
+                build_type="preview"
+                is_prerelease=true
+                should_build=false
+                should_push=false
+                echo "⏭️  Preview tags do not publish Docker images"
+                ;;
               # Prerelease versions (must match first, more specific)
               v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                 build_type="prerelease"

.github/workflows/helm-package.yml

@@ -33,11 +33,12 @@ jobs:
   build-helm-package:
     runs-on: ubuntu-latest
     if: |
-      github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
       (
         github.event.workflow_run.conclusion == 'success' &&
         github.event.workflow_run.event == 'push' &&
-        contains(github.event.workflow_run.head_branch, '.')
+        contains(github.event.workflow_run.head_branch, '.') &&
+        !contains(github.event.workflow_run.head_branch, '-preview')
       )
 
     outputs:

Cargo.lock

@@ -642,9 +642,9 @@ dependencies = [
 
 [[package]]
 name = "async-compression"
-version = "0.4.42"
+version = "0.4.43"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e79b3f8a79cccc2898f31920fc69f304859b3bd567490f75ebf51ae1c792a9ac"
+checksum = "3976abdc8fe7d1133d43d304afd42abdf5bc3e1319d263d223bde07b5efc4be8"
 dependencies = [
  "compression-codecs",
  "compression-core",
@@ -870,7 +870,7 @@ dependencies = [
  "bytes",
  "fastrand",
  "hex",
- "http 1.4.2",
+ "http 1.5.0",
  "sha1 0.10.7",
  "time",
  "tokio",
@@ -934,7 +934,7 @@ dependencies = [
  "bytes-utils",
  "fastrand",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 0.4.6",
  "http-body 1.1.0",
  "percent-encoding",
@@ -970,7 +970,7 @@ dependencies = [
  "hex",
  "hmac 0.13.0",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "lru 0.16.4",
  "percent-encoding",
@@ -1001,7 +1001,7 @@ dependencies = [
  "bytes",
  "fastrand",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "regex-lite",
  "tracing",
 ]
@@ -1027,7 +1027,7 @@ dependencies = [
  "bytes",
  "fastrand",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "regex-lite",
  "tracing",
 ]
@@ -1054,7 +1054,7 @@ dependencies = [
  "aws-types",
  "fastrand",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "regex-lite",
  "tracing",
 ]
@@ -1076,7 +1076,7 @@ dependencies = [
  "hex",
  "hmac 0.13.0",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "p256 0.13.2",
  "percent-encoding",
  "sha2 0.11.0",
@@ -1108,7 +1108,7 @@ dependencies = [
  "bytes",
  "crc-fast",
  "hex",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "md-5 0.11.0",
@@ -1142,7 +1142,7 @@ dependencies = [
  "bytes-utils",
  "futures-core",
  "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "percent-encoding",
@@ -1161,7 +1161,7 @@ dependencies = [
  "aws-smithy-runtime-api",
  "aws-smithy-types",
  "h2",
- "http 1.4.2",
+ "http 1.5.0",
  "hyper",
  "hyper-rustls",
  "hyper-util",
@@ -1224,7 +1224,7 @@ dependencies = [
  "bytes",
  "fastrand",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 0.4.6",
  "http-body 1.1.0",
  "http-body-util",
@@ -1245,7 +1245,7 @@ dependencies = [
  "aws-smithy-types",
  "bytes",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "pin-project-lite",
  "tokio",
  "tracing",
@@ -1271,7 +1271,7 @@ checksum = "7d56e0a4e53127a632224e43633b0fe045fa9e1e3cfc68b9830f1115e103f910"
 dependencies = [
  "aws-smithy-runtime-api",
  "aws-smithy-types",
- "http 1.4.2",
+ "http 1.5.0",
 ]
 
 [[package]]
@@ -1285,7 +1285,7 @@ dependencies = [
  "bytes-utils",
  "futures-core",
  "http 0.2.12",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 0.4.6",
  "http-body 1.1.0",
  "http-body-util",
@@ -1337,7 +1337,7 @@ dependencies = [
  "bytes",
  "form_urlencoded",
  "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "hyper",
@@ -1368,7 +1368,7 @@ checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1"
 dependencies = [
  "bytes",
  "futures-core",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "mime",
@@ -3229,7 +3229,7 @@ dependencies = [
  "futures-util",
  "headers",
  "htmlescape",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "libc",
@@ -3654,7 +3654,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
 
 [[package]]
 name = "e2e_test"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "anyhow",
  "astral-tokio-tar",
@@ -3672,7 +3672,7 @@ dependencies = [
  "flate2",
  "futures",
  "hex",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body-util",
  "hyper",
  "hyper-util",
@@ -4384,7 +4384,7 @@ dependencies = [
  "google-cloud-gax",
  "hex",
  "hmac 0.13.0",
- "http 1.4.2",
+ "http 1.5.0",
  "jsonwebtoken 10.4.0",
  "reqwest",
  "rustc_version",
@@ -4409,7 +4409,7 @@ dependencies = [
  "futures",
  "google-cloud-rpc",
  "google-cloud-wkt",
- "http 1.4.2",
+ "http 1.5.0",
  "pin-project",
  "rand 0.10.2",
  "serde",
@@ -4431,7 +4431,7 @@ dependencies = [
  "google-cloud-rpc",
  "google-cloud-wkt",
  "h2",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "hyper",
@@ -4544,7 +4544,7 @@ dependencies = [
  "google-cloud-type",
  "google-cloud-wkt",
  "hex",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "md5",
  "percent-encoding",
@@ -4624,7 +4624,7 @@ dependencies = [
  "fnv",
  "futures-core",
  "futures-sink",
- "http 1.4.2",
+ "http 1.5.0",
  "indexmap 2.14.0",
  "slab",
  "tokio",
@@ -4727,7 +4727,7 @@ dependencies = [
  "base64 0.22.1",
  "bytes",
  "headers-core",
- "http 1.4.2",
+ "http 1.5.0",
  "httpdate",
  "mime",
  "sha1 0.10.7",
@@ -4739,7 +4739,7 @@ version = "0.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "54b4a22553d4242c49fddb9ba998a99962b5cc6f22cb5a3482bec22522403ce4"
 dependencies = [
- "http 1.4.2",
+ "http 1.5.0",
 ]
 
 [[package]]
@@ -4961,9 +4961,9 @@ dependencies = [
 
 [[package]]
 name = "http"
-version = "1.4.2"
+version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425"
+checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0"
 dependencies = [
  "bytes",
  "itoa",
@@ -4987,7 +4987,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c"
 dependencies = [
  "bytes",
- "http 1.4.2",
+ "http 1.5.0",
 ]
 
 [[package]]
@@ -4998,7 +4998,7 @@ checksum = "e9f41fd6a08e4d4ec69df65976da761afd5ad5e58a9d4acb46bd1c953a9e3ff2"
 dependencies = [
  "bytes",
  "futures-core",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "pin-project-lite",
 ]
@@ -5044,7 +5044,7 @@ dependencies = [
  "futures-channel",
  "futures-core",
  "h2",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "httparse",
  "httpdate",
@@ -5061,7 +5061,7 @@ version = "0.27.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f"
 dependencies = [
- "http 1.4.2",
+ "http 1.5.0",
  "hyper",
  "hyper-util",
  "log",
@@ -5095,7 +5095,7 @@ dependencies = [
  "bytes",
  "futures-channel",
  "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "hyper",
  "ipnet",
@@ -6704,10 +6704,10 @@ version = "5.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d"
 dependencies = [
- "base64 0.22.1",
+ "base64 0.21.7",
  "chrono",
  "getrandom 0.2.17",
- "http 1.4.2",
+ "http 1.5.0",
  "rand 0.8.7",
  "serde",
  "serde_json",
@@ -6815,7 +6815,7 @@ dependencies = [
  "futures-channel",
  "futures-core",
  "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
  "humantime",
  "itertools 0.14.0",
  "parking_lot",
@@ -6871,7 +6871,7 @@ dependencies = [
  "dyn-clone",
  "ed25519-dalek 2.2.0",
  "hmac 0.12.1",
- "http 1.4.2",
+ "http 1.5.0",
  "itertools 0.10.5",
  "log",
  "oauth2",
@@ -6932,7 +6932,7 @@ checksum = "5683015d09e2df236ef005b17f6f196f0d5f6313c4fa43a7b6a53b52776e4331"
 dependencies = [
  "async-trait",
  "bytes",
- "http 1.4.2",
+ "http 1.5.0",
  "opentelemetry",
  "reqwest",
 ]
@@ -6944,7 +6944,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "9966929966d17620d7c316c643ba62631826e10021409357772d5eea84f62c35"
 dependencies = [
  "flate2",
- "http 1.4.2",
+ "http 1.5.0",
  "opentelemetry",
  "opentelemetry-http",
  "opentelemetry-proto",
@@ -7820,7 +7820,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
 dependencies = [
  "heck",
- "itertools 0.14.0",
+ "itertools 0.10.5",
  "log",
  "multimap",
  "once_cell",
@@ -7840,7 +7840,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
 dependencies = [
  "heck",
- "itertools 0.14.0",
+ "itertools 0.10.5",
  "log",
  "multimap",
  "petgraph 0.8.3",
@@ -7861,7 +7861,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
 dependencies = [
  "anyhow",
- "itertools 0.14.0",
+ "itertools 0.10.5",
  "proc-macro2",
  "quote",
  "syn 2.0.119",
@@ -7874,7 +7874,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
 dependencies = [
  "anyhow",
- "itertools 0.14.0",
+ "itertools 0.10.5",
  "proc-macro2",
  "quote",
  "syn 2.0.119",
@@ -8329,9 +8329,9 @@ dependencies = [
 
 [[package]]
 name = "redis"
-version = "1.4.1"
+version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0b9503711b03773e43b31668c7b5bd279ee7cd9b7d18cff7c23a42cc1d08e5a"
+checksum = "3257df217f7eab0044627a268c9cc6cdb60c0c421c88f83ac41c4e31520b6b84"
 dependencies = [
  "arc-swap",
  "arcstr",
@@ -8481,7 +8481,7 @@ dependencies = [
  "futures-core",
  "futures-util",
  "h2",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "hyper",
@@ -8627,7 +8627,7 @@ dependencies = [
  "async-tungstenite",
  "futures-io",
  "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
  "rustls-native-certs",
  "rustls-pki-types",
  "rustls-webpki",
@@ -8657,7 +8657,7 @@ dependencies = [
  "flume",
  "futures-io",
  "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
  "log",
  "mqttbytes-core-next",
  "rumqttc-core-next",
@@ -8845,7 +8845,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "aes-gcm",
  "anyhow",
@@ -8874,7 +8874,7 @@ dependencies = [
  "hex-simd",
  "hmac 0.13.0",
  "hotpath",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "hyper",
@@ -8981,7 +8981,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-audit"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "chrono",
@@ -9003,12 +9003,12 @@ dependencies = [
 
 [[package]]
 name = "rustfs-checksums"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "base64-simd",
  "bytes",
  "crc-fast",
- "http 1.4.2",
+ "http 1.5.0",
  "md-5 0.11.0",
  "pretty_assertions",
  "sha1 0.11.0",
@@ -9018,7 +9018,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-common"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "chrono",
  "metrics",
@@ -9033,7 +9033,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-concurrency"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "insta",
  "rustfs-io-core",
@@ -9045,7 +9045,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-config"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "const-str",
  "serde",
@@ -9054,7 +9054,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-credentials"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "base64-simd",
  "hmac 0.13.0",
@@ -9067,7 +9067,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-crypto"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "aes-gcm",
  "argon2",
@@ -9087,7 +9087,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-data-usage"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "rmp-serde",
@@ -9097,9 +9097,8 @@ dependencies = [
 
 [[package]]
 name = "rustfs-ecstore"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
- "aes-gcm",
  "arc-swap",
  "async-channel",
  "async-recursion",
@@ -9115,7 +9114,6 @@ dependencies = [
  "byteorder",
  "bytes",
  "bytesize",
- "chacha20poly1305",
  "chrono",
  "criterion",
  "enumset",
@@ -9128,8 +9126,9 @@ dependencies = [
  "google-cloud-storage",
  "hex-simd",
  "hmac 0.13.0",
+ "hostname",
  "hotpath",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "hyper",
@@ -9168,7 +9167,6 @@ dependencies = [
  "rustfs-erasure-codec",
  "rustfs-filemeta",
  "rustfs-io-metrics",
- "rustfs-kms",
  "rustfs-lifecycle",
  "rustfs-lock",
  "rustfs-madmin",
@@ -9235,7 +9233,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-extension-schema"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "serde",
  "serde_json",
@@ -9244,7 +9242,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-filemeta"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "arc-swap",
  "byteorder",
@@ -9270,12 +9268,12 @@ dependencies = [
 
 [[package]]
 name = "rustfs-heal"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "base64 0.23.0",
  "futures",
- "http 1.4.2",
+ "http 1.5.0",
  "metrics",
  "rustfs-common",
  "rustfs-concurrency",
@@ -9300,13 +9298,13 @@ dependencies = [
 
 [[package]]
 name = "rustfs-iam"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "arc-swap",
  "async-trait",
  "base64-simd",
  "futures",
- "http 1.4.2",
+ "http 1.5.0",
  "jsonwebtoken 11.0.0",
  "moka",
  "openidconnect",
@@ -9337,7 +9335,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-io-core"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "bytes",
  "memmap2",
@@ -9349,13 +9347,15 @@ dependencies = [
 
 [[package]]
 name = "rustfs-io-metrics"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "criterion",
  "metrics",
  "metrics-util",
  "num_cpus",
+ "rustfs-common",
  "rustfs-s3-ops",
+ "rustfs-utils",
  "sysinfo",
  "thiserror 2.0.19",
  "tokio",
@@ -9412,11 +9412,11 @@ dependencies = [
 
 [[package]]
 name = "rustfs-keystone"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "bytes",
  "futures",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "hyper",
@@ -9438,7 +9438,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-kms"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "aes-gcm",
  "arc-swap",
@@ -9472,7 +9472,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-lifecycle"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "metrics",
@@ -9494,7 +9494,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-lock"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "crossbeam-queue",
@@ -9516,7 +9516,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-log-analyzer"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "chrono",
  "flate2",
@@ -9534,7 +9534,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-madmin"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "chrono",
  "humantime",
@@ -9548,7 +9548,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-notify"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "arc-swap",
  "async-trait",
@@ -9582,7 +9582,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-object-capacity"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "criterion",
  "futures",
@@ -9600,7 +9600,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-object-data-cache"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "bytes",
  "criterion",
@@ -9616,7 +9616,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-obs"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "chrono",
  "crossbeam-channel",
@@ -9668,7 +9668,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-policy"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "base64-simd",
@@ -9697,7 +9697,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-protocols"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "astral-tokio-tar",
  "async-compression",
@@ -9710,7 +9710,7 @@ dependencies = [
  "futures-util",
  "hex",
  "hmac 0.13.0",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body-util",
  "hyper",
  "hyper-util",
@@ -9759,7 +9759,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-protos"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "flatbuffers",
  "prost 0.14.4",
@@ -9782,7 +9782,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-replication"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "byteorder",
  "bytes",
@@ -9799,7 +9799,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-rio"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "aes-gcm",
  "arc-swap",
@@ -9811,7 +9811,7 @@ dependencies = [
  "futures",
  "hex-simd",
  "hotpath",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body-util",
  "md-5 0.11.0",
  "pin-project-lite",
@@ -9837,7 +9837,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-rio-v2"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "aes-gcm",
  "bytes",
@@ -9859,14 +9859,14 @@ dependencies = [
 
 [[package]]
 name = "rustfs-s3-ops"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "rustfs-s3-types",
 ]
 
 [[package]]
 name = "rustfs-s3-types"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "serde",
  "serde_json",
@@ -9874,7 +9874,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-s3select-api"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "bytes",
@@ -9882,7 +9882,7 @@ dependencies = [
  "datafusion",
  "futures",
  "futures-core",
- "http 1.4.2",
+ "http 1.5.0",
  "metrics",
  "parking_lot",
  "rustfs-common",
@@ -9903,7 +9903,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-s3select-query"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-recursion",
  "async-trait",
@@ -9919,7 +9919,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-scanner"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "bytes",
@@ -9927,7 +9927,7 @@ dependencies = [
  "futures",
  "hex-simd",
  "hmac 0.13.0",
- "http 1.4.2",
+ "http 1.5.0",
  "metrics",
  "rand 0.10.2",
  "rmp-serde",
@@ -9957,18 +9957,18 @@ dependencies = [
 
 [[package]]
 name = "rustfs-security-governance"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "thiserror 2.0.19",
 ]
 
 [[package]]
 name = "rustfs-signer"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "base64-simd",
  "bytes",
- "http 1.4.2",
+ "http 1.5.0",
  "hyper",
  "rustfs-utils",
  "s3s",
@@ -9981,7 +9981,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-storage-api"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "insta",
@@ -9995,7 +9995,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-targets"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "arc-swap",
  "async-nats",
@@ -10048,7 +10048,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-test-utils"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "rustfs-data-usage",
  "rustfs-ecstore",
@@ -10063,7 +10063,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-tls-runtime"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "arc-swap",
  "metrics",
@@ -10083,11 +10083,11 @@ dependencies = [
 
 [[package]]
 name = "rustfs-trusted-proxies"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "async-trait",
  "axum",
- "http 1.4.2",
+ "http 1.5.0",
  "ipnetwork",
  "metrics",
  "moka",
@@ -10119,7 +10119,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-utils"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "base64-simd",
  "blake2",
@@ -10133,7 +10133,7 @@ dependencies = [
  "hex-simd",
  "highway",
  "hmac 0.13.0",
- "http 1.4.2",
+ "http 1.5.0",
  "hyper",
  "local-ip-address",
  "lz4",
@@ -10160,7 +10160,7 @@ dependencies = [
 
 [[package]]
 name = "rustfs-zip"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 dependencies = [
  "astral-tokio-tar",
  "async-compression",
@@ -10190,7 +10190,7 @@ dependencies = [
  "anyhow",
  "async-trait",
  "bytes",
- "http 1.4.2",
+ "http 1.5.0",
  "reqwest",
  "rustify_derive",
  "serde",
@@ -10230,9 +10230,9 @@ dependencies = [
 
 [[package]]
 name = "rustls"
-version = "0.23.42"
+version = "0.23.43"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3c54fcab019b409d04215d3a17cb438fd7fbf192ee61461f20f4fe18704bc138"
+checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06"
 dependencies = [
  "aws-lc-rs",
  "log",
@@ -10370,7 +10370,7 @@ dependencies = [
  "futures",
  "hex-simd",
  "hmac 0.13.0",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "httparse",
@@ -11498,7 +11498,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
 dependencies = [
  "fastrand",
- "getrandom 0.4.3",
+ "getrandom 0.3.4",
  "once_cell",
  "rustix",
  "windows-sys 0.61.2",
@@ -11734,13 +11734,13 @@ dependencies = [
 
 [[package]]
 name = "tokio-macros"
-version = "2.7.1"
+version = "2.7.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6328af13490e73a9b4694030fafd93f8c8c6a9dede33e821c3fc63eddf8042ba"
+checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.119",
+ "syn 3.0.3",
 ]
 
 [[package]]
@@ -11844,7 +11844,7 @@ dependencies = [
  "bytes",
  "futures-core",
  "futures-sink",
- "http 1.4.2",
+ "http 1.5.0",
  "httparse",
  "rand 0.8.7",
  "rustls-pki-types",
@@ -11896,7 +11896,7 @@ dependencies = [
  "bytes",
  "flate2",
  "h2",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "hyper",
@@ -11983,7 +11983,7 @@ dependencies = [
  "bitflags 2.13.1",
  "bytes",
  "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "pin-project-lite",
  "tower",
@@ -12003,7 +12003,7 @@ dependencies = [
  "bytes",
  "futures-core",
  "futures-util",
- "http 1.4.2",
+ "http 1.5.0",
  "http-body 1.1.0",
  "http-body-util",
  "percent-encoding",
@@ -12176,7 +12176,7 @@ checksum = "6c01152af293afb9c7c2a57e4b559c5620b421f6d133261c60dd2d0cdb38e6b8"
 dependencies = [
  "bytes",
  "data-encoding",
- "http 1.4.2",
+ "http 1.5.0",
  "httparse",
  "log",
  "rand 0.9.5",
@@ -12363,7 +12363,7 @@ checksum = "30ffcc0e81025065dda612ec1e26a3d81bb16ef3062354873d17a35965d68522"
 dependencies = [
  "async-trait",
  "derive_builder",
- "http 1.4.2",
+ "http 1.5.0",
  "reqwest",
  "rustify",
  "rustify_derive",

Cargo.toml

@@ -69,7 +69,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.97.1"
-version = "1.0.0-beta.11"
+version = "1.0.0-beta.12"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,58 +86,58 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-beta.11" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.11" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.11" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.11" }
-rustfs-common = { path = "crates/common", version = "1.0.0-beta.11" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.11" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-beta.11" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.11" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.11" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.11" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.11" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.11" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.11" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.11" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.11" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.11" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.11" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.11" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.11" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.11" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.11" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.11" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.11" }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.11" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.11" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.11" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.11" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.11" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.11" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.11" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.11" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.11" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.11" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.11" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.11" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.11" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.11" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.11" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.11" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.11" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.11" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.11" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.11" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.11" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.11" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.11" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.12" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.12" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.12" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.12" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.12" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.12" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.12" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.12" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.12" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.12" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.12" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.12" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.12" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.12" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.12" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.12" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.12" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.12" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.12" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.12" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.12" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.12" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.12" }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.12" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.12" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.12" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.12" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.12" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.12" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.12" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.12" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.12" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.12" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.12" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.12" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.12" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.12" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.12" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.12" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.12" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.12" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.12" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.12" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.12" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.12" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.12" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
 async_zip = { default-features = false, version = "0.0.18" }
 mysql_async = { default-features = false, version = "0.37" }
-async-compression = { version = "0.4.42" }
+async-compression = { version = "0.4.43" }
 async-recursion = "1.1.1"
 async-trait = "0.1.91"
 async-nats = { version = "0.50.0", default-features = false }
@@ -152,7 +152,7 @@ lapin = { default-features = false, version = "4.10.0" }
 hyper = { version = "1.11.0" }
 hyper-rustls = { default-features = false, version = "0.27.9" }
 hyper-util = { version = "0.1.20" }
-http = "1.4.2"
+http = "1.5.0"
 http-body = "1.1.0"
 http-body-util = "0.1.4"
 minlz = "1.2.3"
@@ -200,7 +200,7 @@ jsonwebtoken = { version = "11.0.0" }
 openidconnect = { default-features = false, version = "4.0" }
 pbkdf2 = "0.13.0"
 rsa = { version = "=0.10.0-rc.18" }
-rustls = { default-features = false, version = "0.23.42" }
+rustls = { default-features = false, version = "0.23.43" }
 rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.1"
 sha1 = "0.11.0"
@@ -256,6 +256,7 @@ hashbrown = { version = "0.17.1" }
 hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
+hostname = "0.4.2"
 ipnetwork = { version = "0.21.1" }
 lazy_static = "1.5.0"
 libc = "0.2.189"
@@ -283,7 +284,7 @@ reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.33.3" }
-redis = { version = "1.4.1" }
+redis = { version = "1.5.0" }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }

README.md

@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.11
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command

README_ZH.md

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.11
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：

crates/config/README.md

@@ -66,6 +66,10 @@ Current guidance:
 
 - `RUSTFS_BROWSER_REDIRECT_URL` sets the externally reachable browser origin used for OIDC callback, console success redirect, and logout fallback URLs. Configure it to the public scheme and authority without a path, for example `https://console.example.com`. In load-balancer deployments, keep OIDC authorize and callback requests on the same backend node because the in-flight OIDC `state` is local to the RustFS node.
 
+## Distributed endpoint locality
+
+- `RUSTFS_LOCAL_ENDPOINT_HOST` identifies this server's host in a distributed `RUSTFS_VOLUMES` topology without resolving every peer during startup. Set it to exactly one host, without a scheme, port, or path. It is accepted only for orchestrated URL topologies and must match at least one endpoint on the RustFS server port; invalid or unmatched values fail startup. Leave it unset to retain DNS-based locality discovery.
+
 ## Scanner environment aliases
 
 - `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)

crates/config/src/constants/app.rs

@@ -131,6 +131,10 @@ pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
 /// Environment variable for server volumes.
 pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
 
+/// Environment variable identifying this server's host in distributed endpoint
+/// lists without relying on DNS locality discovery.
+pub const ENV_LOCAL_ENDPOINT_HOST: &str = "RUSTFS_LOCAL_ENDPOINT_HOST";
+
 /// Environment variable to explicitly bypass local physical disk independence checks.
 pub const ENV_UNSAFE_BYPASS_DISK_CHECK: &str = "RUSTFS_UNSAFE_BYPASS_DISK_CHECK";
 

crates/e2e_test/src/inline_fast_path_cluster_test.rs

@@ -2136,11 +2136,20 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
     assert_eq!(terminal["bucket"].as_str(), Some(bucket.as_str()));
     assert_eq!(terminal["prefix"].as_str(), Some(prefix));
     assert_eq!(terminal["dry_run"].as_bool(), Some(false));
-    assert_eq!(
-        terminal["status"].as_str(),
-        Some("partial"),
+    let terminal_status = terminal["status"].as_str();
+    assert!(
+        matches!(terminal_status, Some("partial" | "unknown")),
         "small transition queue should surface terminal backpressure: {terminal}"
     );
+    if terminal_status == Some("unknown") {
+        let failure_reason = terminal["failure_reason"]
+            .as_str()
+            .ok_or_else(|| format!("unknown terminal status omitted failure_reason: {terminal}"))?;
+        assert!(
+            failure_reason.contains("worker result was not persisted before the transition queue drained"),
+            "unknown terminal status should identify lost worker-result persistence: {terminal}"
+        );
+    }
     let skipped_queue_full = terminal["report"]["skipped_queue_full"]
         .as_u64()
         .ok_or_else(|| format!("terminal status omitted report.skipped_queue_full: {terminal}"))?;

crates/ecstore/Cargo.toml

@@ -57,7 +57,6 @@ rustfs-policy.workspace = true
 rustfs-protos.workspace = true
 rustfs-replication.workspace = true
 rustfs-lifecycle.workspace = true
-rustfs-kms.workspace = true
 rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
 rustfs-object-capacity.workspace = true
@@ -105,6 +104,7 @@ tempfile.workspace = true
 hyper = { workspace = true, features = ["http2", "http1", "server"] }
 hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
 hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
+hostname.workspace = true
 rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
 rustls-pki-types.workspace = true
 tokio = { workspace = true, features = ["io-util", "sync", "signal", "fs", "rt-multi-thread"] }
@@ -124,8 +124,6 @@ libc.workspace = true
 rustix = { workspace = true, features = ["process", "fs"] }
 rustfs-madmin.workspace = true
 reqwest = { workspace = true }
-aes-gcm = { workspace = true, features = ["rand_core"] }
-chacha20poly1305.workspace = true
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 urlencoding = { workspace = true }
 smallvec = { workspace = true, features = ["serde"] }

crates/ecstore/src/api/mod.rs

@@ -184,8 +184,8 @@ pub mod bucket {
             get_global_replication_stats, init_background_replication, read_durable_mrf_backlog, replication_state_to_filemeta,
             replication_status_to_filemeta, replication_statuses_map, replication_target_arns, resync_start_conflict_id,
             should_remove_replication_target, should_schedule_delete_replication, should_use_existing_delete_replication_info,
-            should_use_existing_delete_replication_source, unsupported_replication_config_field,
-            validate_replication_config_target_arns, version_purge_status_to_filemeta,
+            should_use_existing_delete_replication_source, validate_replication_config_target_arns,
+            version_purge_status_to_filemeta,
         };
     }
 
@@ -391,10 +391,12 @@ pub mod notification {
 
 pub mod object {
     pub use crate::object_api::{
-        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, GetObjectBodySource,
-        GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, StreamConsumer,
-        get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
-        register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
+        BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
+        GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, ObjectEncryptionResolver, ObjectInfo,
+        ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode,
+        ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook,
+        register_get_object_body_cache_hook, register_object_mutation_hook, unregister_get_object_body_cache_hook,
+        unregister_object_mutation_hook,
     };
     pub use crate::store::PreparedGetObjectReader;
 }

crates/ecstore/src/bucket/replication/mod.rs

@@ -46,7 +46,7 @@ mod runtime_boundary;
 pub use datatypes::ResyncStatusType;
 pub use replication_config_boundary::{
     ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, replication_target_arns,
-    should_remove_replication_target, unsupported_replication_config_field, validate_replication_config_target_arns,
+    should_remove_replication_target, validate_replication_config_target_arns,
 };
 #[cfg(test)]
 pub(crate) use replication_filemeta_boundary::ReplicateTargetDecision;

crates/ecstore/src/bucket/replication/replication_config_boundary.rs

@@ -14,5 +14,5 @@
 
 pub use rustfs_replication::{
     ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, replication_target_arns,
-    should_remove_replication_target, unsupported_replication_config_field, validate_replication_config_target_arns,
+    should_remove_replication_target, validate_replication_config_target_arns,
 };

crates/ecstore/src/bucket/replication/replication_object_bridge.rs

@@ -14,11 +14,8 @@
 
 use std::{collections::HashMap, sync::Arc};
 
-use super::replication_error_boundary::Result;
 use super::replication_filemeta_boundary::{ReplicateDecision, ReplicationStatusType, ReplicationType};
-use super::replication_object_config::{
-    check_replicate_delete, check_replicate_delete_strict, get_must_replicate_options, must_replicate,
-};
+use super::replication_object_config::{check_replicate_delete, get_must_replicate_options, must_replicate};
 use super::replication_object_decision_boundary::MustReplicateOptions;
 use super::replication_pool::{schedule_replication, schedule_replication_delete};
 use super::replication_queue_boundary::DeletedObjectReplicationInfo;
@@ -53,16 +50,6 @@ impl ReplicationObjectBridge {
         check_replicate_delete(bucket, object, source, opts, get_error).await
     }
 
-    pub async fn check_delete_strict(
-        bucket: &str,
-        object: &ObjectToDelete,
-        source: &ObjectInfo,
-        opts: &ObjectOptions,
-        get_error: Option<String>,
-    ) -> Result<ReplicateDecision> {
-        check_replicate_delete_strict(bucket, object, source, opts, get_error).await
-    }
-
     pub async fn schedule_object<S: ReplicationStorage>(
         object: ObjectInfo,
         storage: Arc<S>,

crates/ecstore/src/bucket/replication/replication_object_config.rs

@@ -169,8 +169,9 @@ pub(crate) async fn check_replicate_delete(
     del_opts: &ObjectOptions,
     gerr: Option<String>,
 ) -> ReplicateDecision {
-    match check_replicate_delete_strict(bucket, dobj, oi, del_opts, gerr).await {
-        Ok(decision) => decision,
+    let rcfg = match get_replication_config(bucket).await {
+        Ok(Some(config)) => config,
+        Ok(None) => return ReplicateDecision::default(),
         Err(err) => {
             error!(
                 event = EVENT_RESYNC_CONFIG_LOOKUP_SKIPPED,
@@ -181,30 +182,16 @@ pub(crate) async fn check_replicate_delete(
                 error = %err,
                 "Failed to look up replication config for delete replication"
             );
-            ReplicateDecision::default()
+            return ReplicateDecision::default();
         }
-    }
-}
-
-pub(crate) async fn check_replicate_delete_strict(
-    bucket: &str,
-    dobj: &ObjectToDelete,
-    oi: &ObjectInfo,
-    del_opts: &ObjectOptions,
-    gerr: Option<String>,
-) -> Result<ReplicateDecision> {
-    let rcfg = match get_replication_config(bucket).await {
-        Ok(Some(config)) => config,
-        Ok(None) => return Ok(ReplicateDecision::default()),
-        Err(err) => return Err(err),
     };
 
     if del_opts.replication_request {
-        return Ok(ReplicateDecision::default());
+        return ReplicateDecision::default();
     }
 
-    if !del_opts.versioned && !del_opts.version_suspended {
-        return Ok(ReplicateDecision::default());
+    if !del_opts.versioned {
+        return ReplicateDecision::default();
     }
 
     let replication_delete = object_to_delete_for_replication(dobj);
@@ -222,7 +209,7 @@ pub(crate) async fn check_replicate_delete_strict(
     let mut dsc = ReplicateDecision::new();
 
     if tgt_arns.is_empty() {
-        return Ok(dsc);
+        return dsc;
     }
 
     for tgt_arn in tgt_arns {
@@ -252,7 +239,7 @@ pub(crate) async fn check_replicate_delete_strict(
         dsc.set(tgt_dsc);
     }
 
-    Ok(dsc)
+    dsc
 }
 
 pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplicateOptions) -> ReplicateDecision {

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -1151,6 +1151,60 @@ pub async fn replicate_delete<S: ReplicationStorage>(dobj: DeletedObjectReplicat
         dobj.delete_object.version_id
     };
 
+    let _rcfg = match get_replication_config(&bucket).await {
+        Ok(Some(config)) => config,
+        Ok(None) => {
+            debug!(
+                event = EVENT_REPLICATION_DELETE_SKIPPED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
+                bucket = %bucket,
+                reason = "replication_config_missing",
+                "Skipping replication delete because replication config is missing"
+            );
+            send_local_event(EventArgs {
+                event_name: EventName::ObjectReplicationNotTracked.to_string(),
+                bucket_name: bucket.clone(),
+                object: ObjectInfo {
+                    bucket: bucket.clone(),
+                    name: dobj.delete_object.object_name.clone(),
+                    version_id,
+                    delete_marker: dobj.delete_object.delete_marker,
+                    ..Default::default()
+                },
+                user_agent: "Internal: [Replication]".to_string(),
+                ..Default::default()
+            });
+
+            return;
+        }
+        Err(err) => {
+            debug!(
+                event = EVENT_REPLICATION_DELETE_SKIPPED,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
+                bucket = %bucket,
+                error = %err,
+                reason = "replication_config_lookup_failed",
+                "Skipping replication delete because replication config lookup failed"
+            );
+            send_local_event(EventArgs {
+                event_name: EventName::ObjectReplicationNotTracked.to_string(),
+                bucket_name: bucket.clone(),
+                object: ObjectInfo {
+                    bucket: bucket.clone(),
+                    name: dobj.delete_object.object_name.clone(),
+                    version_id,
+                    delete_marker: dobj.delete_object.delete_marker,
+                    ..Default::default()
+                },
+                user_agent: "Internal: [Replication]".to_string(),
+                ..Default::default()
+            });
+            return;
+        }
+    };
+
     if dobj.delete_object.delete_marker
         && let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id
     {

crates/ecstore/src/client/utils.rs

@@ -46,16 +46,6 @@ lazy_static! {
         m.insert("x-amz-replication-status".to_string(), true);
         m
     };
-    static ref SSE_HEADERS: HashMap<String, bool> = {
-        let mut m = HashMap::new();
-        m.insert("x-amz-server-side-encryption".to_string(), true);
-        m.insert("x-amz-server-side-encryption-aws-kms-key-id".to_string(), true);
-        m.insert("x-amz-server-side-encryption-context".to_string(), true);
-        m.insert("x-amz-server-side-encryption-customer-algorithm".to_string(), true);
-        m.insert("x-amz-server-side-encryption-customer-key".to_string(), true);
-        m.insert("x-amz-server-side-encryption-customer-key-md5".to_string(), true);
-        m
-    };
 }
 
 pub fn is_standard_query_value(qs_key: &str) -> bool {
@@ -70,16 +60,12 @@ pub fn is_standard_header(header_key: &str) -> bool {
     *SUPPORTED_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
 }
 
-pub fn is_sse_header(header_key: &str) -> bool {
-    *SSE_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
-}
-
 pub fn is_amz_header(header_key: &str) -> bool {
     let key = header_key.to_lowercase();
     key.starts_with("x-amz-meta-")
         || key.starts_with("x-amz-grant-")
         || key == "x-amz-acl"
-        || is_sse_header(header_key)
+        || rustfs_utils::http::is_sse_header(header_key)
         || key.starts_with("x-amz-checksum-")
 }
 

crates/ecstore/src/cluster/rpc/peer_rest_client.rs

@@ -58,7 +58,7 @@ use std::{
     collections::HashMap,
     io::Cursor,
     sync::{
-        Arc,
+        Arc, Weak,
         atomic::{AtomicBool, Ordering},
     },
     time::SystemTime,
@@ -350,6 +350,44 @@ impl PeerRestClient {
         }
     }
 
+    fn parse_topology_host(peer_host_port: &str, grid_host: &str) -> Result<XHost> {
+        let url = url::Url::parse(grid_host).map_err(|_| Error::other("peer grid host is not a valid URL"))?;
+        if !matches!(url.scheme(), "http" | "https")
+            || !url.username().is_empty()
+            || url.password().is_some()
+            || url.query().is_some()
+            || url.fragment().is_some()
+            || url.path() != "/"
+        {
+            return Err(Error::other("peer grid host has an invalid URL shape"));
+        }
+        let url_host = url.host().ok_or_else(|| Error::other("peer grid host is missing a host"))?;
+        let topology_host = match url.port() {
+            Some(port) => format!("{url_host}:{port}"),
+            None => url_host.to_string(),
+        };
+        let explicit_port = url.port();
+        let name = match url_host {
+            url::Host::Domain(domain) => domain.to_string(),
+            url::Host::Ipv4(address) => address.to_string(),
+            url::Host::Ipv6(address) if explicit_port.is_none() => format!("[{address}]"),
+            url::Host::Ipv6(address) => address.to_string(),
+        };
+        let port = url
+            .port_or_known_default()
+            .filter(|port| *port > 0)
+            .ok_or_else(|| Error::other("peer grid host is missing a valid port"))?;
+        let host = XHost {
+            name,
+            port,
+            is_port_set: explicit_port.is_some(),
+        };
+        if topology_host != peer_host_port {
+            return Err(Error::other("peer topology host does not match its grid URL"));
+        }
+        Ok(host)
+    }
+
     fn build_clients_from_slots(
         slots: Vec<(String, Option<String>, bool)>,
     ) -> (Vec<Option<Self>>, Vec<Option<Self>>, Vec<String>) {
@@ -363,14 +401,14 @@ impl PeerRestClient {
             }
 
             let client = match grid_host {
-                Some(grid_host) => match XHost::try_from(peer_host_port.clone()) {
+                Some(grid_host) => match Self::parse_topology_host(&peer_host_port, &grid_host) {
                     Ok(host) => {
                         let mut client = PeerRestClient::new(host, grid_host);
                         client.topology_member = peer_host_port.clone();
                         Some(client)
                     }
                     Err(err) => {
-                        warn!(peer = %peer_host_port, "Xhost parse failed while constructing peer client: {err:?}");
+                        warn!(peer = %peer_host_port, "peer topology host parse failed while constructing peer client: {err:?}");
                         None
                     }
                 },
@@ -519,9 +557,8 @@ impl PeerRestClient {
         }
 
         let grid_host = self.grid_host.clone();
-        let offline = Arc::clone(&self.offline);
-        let recovery_running = Arc::clone(&self.recovery_running);
-        let span = Self::recovery_monitor_span(&grid_host);
+        let offline = Arc::downgrade(&self.offline);
+        let recovery_running = Arc::downgrade(&self.recovery_running);
         // The offline flag and its recovery are the silent half of
         // rustfs/backlog#888: log the monitor's start and its success so an
         // "offline then back" episode leaves a trace on the observing node.
@@ -530,13 +567,34 @@ impl PeerRestClient {
             grid_host = %self.grid_host,
             "peer RPC connection marked offline after a network-like failure; starting background recovery monitor"
         );
+        drop(Self::spawn_recovery_monitor(grid_host, offline, recovery_running));
+    }
+
+    fn spawn_recovery_monitor(
+        grid_host: String,
+        offline: Weak<AtomicBool>,
+        recovery_running: Weak<AtomicBool>,
+    ) -> tokio::task::JoinHandle<()> {
+        let span = Self::recovery_monitor_span(&grid_host);
         super::spawn_background_monitor(span, async move {
             let mut delay = get_drive_active_check_interval();
             let connect_timeout = get_drive_active_check_timeout();
 
             for attempt in 1..=PEER_REST_RECOVERY_MAX_ATTEMPTS {
+                if offline.strong_count() == 0 || recovery_running.strong_count() == 0 {
+                    return;
+                }
                 tokio::time::sleep(delay).await;
+                if offline.strong_count() == 0 || recovery_running.strong_count() == 0 {
+                    return;
+                }
                 if Self::perform_connectivity_check(&grid_host, connect_timeout).await.is_ok() {
+                    let Some(offline) = offline.upgrade() else {
+                        return;
+                    };
+                    let Some(recovery_running) = recovery_running.upgrade() else {
+                        return;
+                    };
                     offline.store(false, Ordering::Release);
                     recovery_running.store(false, Ordering::Release);
                     info!(
@@ -556,8 +614,10 @@ impl PeerRestClient {
                 attempts = PEER_REST_RECOVERY_MAX_ATTEMPTS,
                 "peer recovery monitor reached max attempts; will retry on next request"
             );
-            recovery_running.store(false, Ordering::Release);
-        });
+            if let Some(recovery_running) = recovery_running.upgrade() {
+                recovery_running.store(false, Ordering::Release);
+            }
+        })
     }
 
     #[cfg(test)]
@@ -1807,9 +1867,13 @@ fn tier_config_reload_status_outcome(status: tonic::Status) -> TierConfigReloadO
 mod tests {
     use super::*;
     use crate::config::com::STORAGE_CLASS_SUB_SYS;
+    use crate::layout::{disks_layout::DisksLayout, endpoints::SetupType};
+    use rustfs_config::{ENV_KUBERNETES_SERVICE_HOST, ENV_LOCAL_ENDPOINT_HOST, ENV_STARTUP_TOPOLOGY_WAIT_MODE};
     use serde_json::Value;
+    use serial_test::serial;
     use std::io::{self, Write};
     use std::sync::{Arc, Mutex};
+    use temp_env::async_with_vars;
     use tracing_subscriber::{Registry, fmt::MakeWriter, layer::SubscriberExt};
 
     #[test]
@@ -1927,30 +1991,115 @@ mod tests {
     fn build_clients_from_slots_preserves_missing_remote_topology_slots() {
         let slots = vec![
             ("127.0.0.1:9000".to_string(), None, true),
-            ("127.0.0.1:9001".to_string(), Some("http://127.0.0.1:9001".to_string()), false),
+            (
+                "rustfs-1.invalid:9001".to_string(),
+                Some("http://rustfs-1.invalid:9001".to_string()),
+                false,
+            ),
+            ("rustfs-2.invalid".to_string(), Some("http://rustfs-2.invalid".to_string()), false),
             ("127.0.0.1:notaport".to_string(), Some("http://127.0.0.1:notaport".to_string()), false),
             ("127.0.0.1:9003".to_string(), None, false),
         ];
 
         let (remote, all, remote_topology_hosts) = PeerRestClient::build_clients_from_slots(slots);
 
-        assert_eq!(remote.len(), 3, "local node is excluded but remote slots are not compacted away");
-        assert_eq!(all.len(), 4, "all slots preserve the sorted cluster topology shape");
+        assert_eq!(remote.len(), 4, "local node is excluded but remote slots are not compacted away");
+        assert_eq!(all.len(), 5, "all slots preserve the sorted cluster topology shape");
         assert_eq!(
             remote_topology_hosts,
             vec![
-                "127.0.0.1:9001".to_string(),
+                "rustfs-1.invalid:9001".to_string(),
+                "rustfs-2.invalid".to_string(),
                 "127.0.0.1:notaport".to_string(),
                 "127.0.0.1:9003".to_string()
             ]
         );
-        assert!(remote[0].is_some(), "valid remote peer should get a client");
-        assert!(remote[1].is_none(), "unparseable remote peer should remain observable as a missing slot");
-        assert!(remote[2].is_none(), "missing grid host should remain observable as a missing slot");
+        let unresolved = remote[0]
+            .as_ref()
+            .expect("temporarily unresolved remote peer should retain a client");
+        assert_eq!(unresolved.host.to_string(), "rustfs-1.invalid:9001");
+        let default_port = remote[1]
+            .as_ref()
+            .expect("temporarily unresolved scheme-default remote peer should retain a client");
+        assert_eq!(default_port.host.to_string(), "rustfs-2.invalid");
+        assert_eq!(default_port.host.port, 80);
+        assert!(!default_port.host.is_port_set);
+        assert!(remote[2].is_none(), "unparseable remote peer should remain observable as a missing slot");
+        assert!(remote[3].is_none(), "missing grid host should remain observable as a missing slot");
         assert!(all[0].is_none(), "local node is represented by the local server_info row");
         assert!(all[1].is_some());
-        assert!(all[2].is_none());
+        assert!(all[2].is_some());
         assert!(all[3].is_none());
+        assert!(all[4].is_none());
+    }
+
+    #[test]
+    fn topology_host_parser_preserves_names_and_bracketed_ipv6() {
+        let domain = PeerRestClient::parse_topology_host("rustfs-1.invalid", "https://rustfs-1.invalid")
+            .expect("unresolved HTTPS topology host should parse without DNS");
+        assert_eq!(domain.to_string(), "rustfs-1.invalid");
+        assert_eq!(domain.port, 443);
+        assert!(!domain.is_port_set);
+
+        let ipv6 = PeerRestClient::parse_topology_host("[2001:db8::1]:9000", "http://[2001:db8::1]:9000")
+            .expect("bracketed IPv6 topology host should parse without changing its identity");
+        assert_eq!(ipv6.to_string(), "[2001:db8::1]:9000");
+
+        let default_port_ipv6 = PeerRestClient::parse_topology_host("[2001:db8::2]", "http://[2001:db8::2]")
+            .expect("scheme-default IPv6 topology host should parse without DNS");
+        assert_eq!(default_port_ipv6.to_string(), "[2001:db8::2]");
+        assert_eq!(default_port_ipv6.port, 80);
+        assert!(!default_port_ipv6.is_port_set);
+
+        assert!(PeerRestClient::parse_topology_host("peer.invalid:0", "http://peer.invalid:0").is_err());
+        assert!(PeerRestClient::parse_topology_host("peer-a.invalid:9000", "http://peer-b.invalid:9000").is_err());
+        assert!(PeerRestClient::parse_topology_host("peer.invalid:9000", "http://peer.invalid:9000/unexpected").is_err());
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn unresolved_default_port_endpoint_topology_retains_all_peer_clients() {
+        let volumes = (0..4)
+            .map(|index| format!("http://rustfs-{index}.invalid:80/data{index}"))
+            .collect::<Vec<_>>();
+        let layout = DisksLayout::from_volumes(&volumes).expect("distributed default-port topology should parse");
+
+        async_with_vars(
+            [
+                (ENV_STARTUP_TOPOLOGY_WAIT_MODE, Some("orchestrated")),
+                (ENV_LOCAL_ENDPOINT_HOST, Some("rustfs-0.invalid")),
+                (ENV_KUBERNETES_SERVICE_HOST, None),
+            ],
+            async {
+                let (server_pools, setup_type) = EndpointServerPools::create_server_endpoints("0.0.0.0:80", &layout)
+                    .await
+                    .expect("explicit local identity should avoid peer DNS during endpoint construction");
+                assert_eq!(setup_type, SetupType::DistErasure);
+
+                let (remote, all, remote_topology_hosts) =
+                    PeerRestClient::build_clients_from_slots(server_pools.peer_grid_host_slots_sorted());
+                assert_eq!(remote.len(), 3);
+                assert!(
+                    remote.iter().all(Option::is_some),
+                    "unresolved remote peers must retain reconnectable clients"
+                );
+                assert_eq!(all.len(), 4);
+                assert_eq!(all.iter().filter(|client| client.is_none()).count(), 1);
+                assert_eq!(remote_topology_hosts.len(), 3);
+                assert!(
+                    remote_topology_hosts.iter().all(|host| !host.contains(':')),
+                    "scheme-default ports must preserve the legacy topology identity"
+                );
+                assert!(
+                    remote
+                        .iter()
+                        .flatten()
+                        .all(|client| client.host.port == 80 && !client.host.is_port_set),
+                    "scheme-default peers must retain the effective dial port"
+                );
+            },
+        )
+        .await;
     }
 
     #[test]
@@ -2740,6 +2889,31 @@ mod tests {
         assert!(!client.offline.load(Ordering::Acquire));
     }
 
+    #[tokio::test(start_paused = true)]
+    async fn dropped_peer_client_releases_and_stops_its_recovery_monitor() {
+        let client = test_peer_client();
+        client.offline.store(true, Ordering::Release);
+        client.recovery_running.store(true, Ordering::Release);
+        let offline = Arc::downgrade(&client.offline);
+        let recovery_running = Arc::downgrade(&client.recovery_running);
+        let handle = PeerRestClient::spawn_recovery_monitor(client.grid_host.clone(), offline.clone(), recovery_running.clone());
+        let started = tokio::time::Instant::now();
+
+        drop(client);
+
+        assert!(offline.upgrade().is_none(), "detached recovery must not retain offline state");
+        assert!(
+            recovery_running.upgrade().is_none(),
+            "detached recovery must not retain its running state"
+        );
+        handle.await.expect("recovery monitor should not panic");
+        assert_eq!(
+            tokio::time::Instant::now(),
+            started,
+            "recovery monitor should stop before advancing to its first delayed probe"
+        );
+    }
+
     #[tokio::test]
     async fn peer_rest_client_finalize_result_keeps_online_for_app_errors_mentioning_unavailable() {
         // Regression: application error text containing "unavailable" (a

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -4814,9 +4814,11 @@ mod tests {
     async fn test_remote_disk_endpoints_with_different_schemes() {
         let test_cases = vec![
             ("http://server:9000", "server:9000"),
-            ("https://secure-server:443", "secure-server"), // Default HTTPS port is omitted
+            ("http://plain-server:80", "plain-server"),
+            ("http://plain-server", "plain-server"),
+            ("https://secure-server:443", "secure-server"),
             ("http://192.168.1.100:8080", "192.168.1.100:8080"),
-            ("https://secure-server", "secure-server"), // No port specified
+            ("https://secure-server", "secure-server"),
         ];
 
         for (url_str, expected_hostname) in test_cases {

crates/ecstore/src/core/sets.rs

@@ -37,7 +37,9 @@ use crate::{
     runtime::instance::{InstanceContext, bootstrap_ctx},
     runtime::sources as runtime_sources,
     set_disk::{PreparedGetObjectMetadata, SetDisks},
-    store::init_format::{check_format_erasure_values, get_format_erasure_in_quorum, load_format_erasure_all, save_format_file},
+    store::init_format::{
+        check_format_erasure_values, load_format_erasure_all, save_format_file, select_format_erasure_in_quorum,
+    },
 };
 use futures::{
     future::join_all,
@@ -947,7 +949,7 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets {
 
     #[tracing::instrument(skip(self))]
     async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option<Error>)> {
-        let (disks, _) = init_storage_disks_with_errors(
+        let (disks, init_errs) = init_storage_disks_with_errors(
             &self.endpoints.endpoints,
             &DiskOption {
                 cleanup: false,
@@ -955,15 +957,36 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets {
             },
         )
         .await;
-        let (formats, errs) = load_format_erasure_all(&disks, true).await;
+        let (formats, mut errs) = load_format_erasure_all(&disks, true).await;
+        for (err, init_err) in errs.iter_mut().zip(init_errs) {
+            if init_err.is_some() {
+                *err = init_err;
+            }
+        }
+        if errs.iter().any(|err| {
+            matches!(
+                err,
+                Some(DiskError::InconsistentDisk | DiskError::CorruptedFormat | DiskError::CorruptedBackend)
+            )
+        }) {
+            return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat)));
+        }
         if let Err(err) = check_format_erasure_values(&formats, self.set_drive_count) {
             info!("failed to check formats erasure values: {}", err);
             return Ok((HealResultItem::default(), Some(err)));
         }
-        let ref_format = match get_format_erasure_in_quorum(&formats) {
-            Ok(format) => format,
+        let (ref_format, quorum_members) = match select_format_erasure_in_quorum(&formats, 0) {
+            Ok((format, members)) if format.shared_identity() == self.format.shared_identity() => (format, members),
+            Ok(_) => return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat))),
             Err(err) => return Ok((HealResultItem::default(), Some(err))),
         };
+        if formats
+            .iter()
+            .zip(quorum_members)
+            .any(|(format, member)| format.is_some() && !member)
+        {
+            return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat)));
+        }
         let mut res = HealResultItem {
             heal_item_type: HealItemType::Metadata.to_string(),
             detail: "disk-format".to_string(),
@@ -985,11 +1008,6 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets {
             return Ok((res, Some(StorageError::NoHealRequired)));
         }
 
-        // if !self.format.eq(&ref_format) {
-        //     info!("format ({:?}) not eq ref_format ({:?})", self.format, ref_format);
-        //     return Ok((res, Some(Error::new(DiskError::CorruptedFormat))));
-        // }
-
         let (new_format_sets, _) = new_heal_format_sets(&ref_format, self.set_count, self.set_drive_count, &formats, &errs);
         if !dry_run {
             let mut tmp_new_formats = vec![None; self.set_count * self.set_drive_count];
@@ -1298,7 +1316,7 @@ mod tests {
         assert_eq!(result, (Some(3), Some(1), Some(0)));
     }
 
-    async fn multipart_listing_test_sets() -> (Vec<tempfile::TempDir>, Arc<Sets>) {
+    async fn two_set_test_sets() -> (Vec<tempfile::TempDir>, Arc<Sets>) {
         let format = FormatV3::new(2, 2);
         let mut temp_dirs = Vec::new();
         let mut all_endpoints = Vec::new();
@@ -1339,8 +1357,8 @@ mod tests {
                     Arc::new(RwLock::new(disks)),
                     2,
                     1,
-                    0,
                     set_index,
+                    0,
                     endpoints,
                     format.clone(),
                     vec![Arc::new(LocalClient::new()), Arc::new(LocalClient::new())],
@@ -1373,11 +1391,114 @@ mod tests {
         (temp_dirs, sets)
     }
 
+    #[tokio::test]
+    async fn set_format_heal_accepts_quorum_from_a_nonzero_set() {
+        let (_temp_dirs, sets) = two_set_test_sets().await;
+
+        let (result, err) = sets.disk_set[1]
+            .heal_format(false)
+            .await
+            .expect("the second erasure set should load its own format quorum");
+
+        assert!(matches!(err, Some(StorageError::NoHealRequired)), "unexpected heal result: {err:?}");
+        assert_eq!(result.disk_count, 2);
+        assert_eq!(result.set_count, 1);
+    }
+
+    #[tokio::test]
+    async fn format_heal_rejects_foreign_majorities_at_set_and_pool_scopes() {
+        let (_temp_dirs, _canonical_format, sets) = setup_heal_format_sets(2, true).await;
+        let set_disks = set_level_heal_view(&sets).await;
+
+        let (_, set_err) = set_disks
+            .heal_format(false)
+            .await
+            .expect("set format heal should report a typed mismatch");
+        assert!(
+            matches!(set_err, Some(StorageError::CorruptedFormat)),
+            "foreign set majority must not replace the cached format: {set_err:?}"
+        );
+
+        let (_, pool_err) = sets
+            .heal_format(false)
+            .await
+            .expect("pool format heal should report a typed mismatch");
+        assert!(
+            matches!(pool_err, Some(StorageError::CorruptedFormat)),
+            "foreign pool majority must not replace the cached format: {pool_err:?}"
+        );
+    }
+
+    #[tokio::test]
+    async fn pool_format_heal_rejects_a_wrong_slot_minority() {
+        let (_temp_dirs, canonical_format, sets) = setup_heal_format_sets(3, false).await;
+        let mut poisoned_format = canonical_format.clone();
+        poisoned_format.erasure.this = canonical_format.erasure.sets[0][0];
+        replace_heal_test_format(&sets, 2, &poisoned_format).await;
+        let probe_err = new_disk(
+            &sets.endpoints.endpoints.as_ref()[2],
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+        )
+        .await
+        .expect_err("a wrong-slot local format must fail disk initialization");
+        assert_eq!(probe_err, DiskError::InconsistentDisk);
+
+        let (_, pool_err) = sets
+            .heal_format(false)
+            .await
+            .expect("pool format heal should report a typed slot mismatch");
+        assert!(
+            matches!(pool_err, Some(StorageError::CorruptedFormat)),
+            "a wrong-slot minority must not be reported as no-heal-required: {pool_err:?}"
+        );
+        assert_eq!(
+            read_heal_test_format(&sets, 2).await,
+            poisoned_format,
+            "format heal must not overwrite a wrong-slot disk"
+        );
+    }
+
+    #[tokio::test]
+    async fn format_heal_rejects_a_foreign_minority_at_set_and_pool_scopes() {
+        let (_temp_dirs, canonical_format, sets) = setup_heal_format_sets(3, false).await;
+        let mut poisoned_format = canonical_format.clone();
+        poisoned_format.id = Uuid::new_v4();
+        poisoned_format.erasure.this = poisoned_format.erasure.sets[0][2];
+        replace_heal_test_format(&sets, 2, &poisoned_format).await;
+        let set_disks = set_level_heal_view(&sets).await;
+
+        let (_, set_err) = set_disks
+            .heal_format(false)
+            .await
+            .expect("set format heal should report a typed identity mismatch");
+        assert!(
+            matches!(set_err, Some(StorageError::CorruptedFormat)),
+            "a foreign minority must not be reported as no-heal-required: {set_err:?}"
+        );
+
+        let (_, pool_err) = sets
+            .heal_format(false)
+            .await
+            .expect("pool format heal should report a typed identity mismatch");
+        assert!(
+            matches!(pool_err, Some(StorageError::CorruptedFormat)),
+            "a foreign minority must not be reported as no-heal-required: {pool_err:?}"
+        );
+        assert_eq!(
+            read_heal_test_format(&sets, 2).await,
+            poisoned_format,
+            "format heal must not overwrite a foreign disk"
+        );
+    }
+
     #[tokio::test(flavor = "multi_thread")]
     #[serial]
     async fn list_multipart_uploads_merges_all_sets_without_pagination_loss() {
         let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::Erasure).await;
-        let (_temp_dirs, sets) = multipart_listing_test_sets().await;
+        let (_temp_dirs, sets) = two_set_test_sets().await;
         let bucket = format!("multipart-list-{}", Uuid::new_v4().simple());
         sets.make_bucket(&bucket, &MakeBucketOptions::default())
             .await
@@ -1616,11 +1737,15 @@ mod tests {
     // formatting the first `num_formatted` of them against a shared reference
     // format and leaving the rest unformatted. Returns the live TempDir handles
     // (must be kept alive), the reference format, and the assembled `Sets`.
-    // `disk_set` is intentionally empty: these tests only drive `heal_format`
-    // with `dry_run == true`, which never touches `disk_set`.
-    async fn setup_heal_format_sets(num_formatted: usize) -> (Vec<tempfile::TempDir>, FormatV3, Sets) {
+    // `disk_set` is intentionally empty: these tests only exercise paths that
+    // return before pool-level healing delegates into a set.
+    async fn setup_heal_format_sets(num_formatted: usize, foreign_identity: bool) -> (Vec<tempfile::TempDir>, FormatV3, Sets) {
         const SET_DRIVE_COUNT: usize = 3;
         let ref_format = FormatV3::new(1, SET_DRIVE_COUNT);
+        let mut stored_format = ref_format.clone();
+        if foreign_identity {
+            stored_format.id = Uuid::new_v4();
+        }
 
         let mut dirs = Vec::with_capacity(SET_DRIVE_COUNT);
         let mut endpoints = Vec::with_capacity(SET_DRIVE_COUNT);
@@ -1645,8 +1770,8 @@ mod tests {
             )
             .await
             .expect("disk should be created");
-            let mut disk_format = ref_format.clone();
-            disk_format.erasure.this = ref_format.erasure.sets[0][i];
+            let mut disk_format = stored_format.clone();
+            disk_format.erasure.this = stored_format.erasure.sets[0][i];
             save_format_file(&Some(disk), &Some(disk_format))
                 .await
                 .expect("format should be saved");
@@ -1677,6 +1802,60 @@ mod tests {
         (dirs, ref_format, sets)
     }
 
+    async fn set_level_heal_view(sets: &Sets) -> Arc<SetDisks> {
+        let endpoints = sets.endpoints.endpoints.as_ref().clone();
+        let mut disks = Vec::with_capacity(endpoints.len());
+        for endpoint in &endpoints {
+            disks.push(Some(
+                new_disk(
+                    endpoint,
+                    &DiskOption {
+                        cleanup: false,
+                        health_check: false,
+                    },
+                )
+                .await
+                .expect("fresh set-level disk handle should open"),
+            ));
+        }
+
+        SetDisks::new(
+            "test-owner".to_string(),
+            Arc::new(RwLock::new(disks)),
+            endpoints.len(),
+            1,
+            0,
+            0,
+            endpoints,
+            sets.format.clone(),
+            Vec::new(),
+        )
+        .await
+    }
+
+    async fn replace_heal_test_format(sets: &Sets, disk_index: usize, format: &FormatV3) {
+        let disk = new_disk(
+            &sets.endpoints.endpoints.as_ref()[disk_index],
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+        )
+        .await
+        .expect("heal test disk should open");
+        save_format_file(&Some(disk.clone()), &Some(format.clone()))
+            .await
+            .expect("poisoned test format should be written");
+    }
+
+    async fn read_heal_test_format(sets: &Sets, disk_index: usize) -> FormatV3 {
+        let path = std::path::Path::new(&sets.endpoints.endpoints.as_ref()[disk_index].get_file_path())
+            .join(crate::disk::RUSTFS_META_BUCKET)
+            .join(crate::disk::FORMAT_CONFIG_FILE);
+        let data = tokio::fs::read(path).await.expect("test format should be readable");
+        FormatV3::try_from(data.as_slice()).expect("test format should parse")
+    }
+
     // Regression for #956 (NoHealRequired path): with every disk already
     // formatted, `heal_format` reports exactly one drive record per disk
     // (N = set_count * set_drive_count), each carrying a real endpoint. Before
@@ -1685,7 +1864,7 @@ mod tests {
     #[tokio::test]
     #[serial]
     async fn heal_format_no_heal_required_reports_one_record_per_disk() {
-        let (_dirs, _ref_format, sets) = setup_heal_format_sets(3).await;
+        let (_dirs, _ref_format, sets) = setup_heal_format_sets(3, false).await;
 
         let (res, err) = sets.heal_format(true).await.expect("heal_format should succeed");
         // All disks formatted -> NoHealRequired early return, still returns `res`.
@@ -1715,7 +1894,7 @@ mod tests {
     #[serial]
     async fn heal_format_heal_path_reports_one_record_per_disk_aligned() {
         // Disks 0 and 1 formatted (quorum), disk 2 unformatted.
-        let (_dirs, _ref_format, sets) = setup_heal_format_sets(2).await;
+        let (_dirs, _ref_format, sets) = setup_heal_format_sets(2, false).await;
 
         let (res, err) = sets.heal_format(true).await.expect("heal_format should succeed");
         // Unformatted disk present -> heal path, not NoHealRequired.

crates/ecstore/src/disk/disk_store.rs

@@ -1049,6 +1049,10 @@ impl LocalDiskWrapper {
         Ok(())
     }
 
+    pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) {
+        *self.disk_id.write().await = id;
+    }
+
     /// Get the current disk ID
     pub async fn get_current_disk_id(&self) -> Option<Uuid> {
         *self.disk_id.read().await

crates/ecstore/src/disk/mod.rs

@@ -132,6 +132,18 @@ pub enum Disk {
     Remote(Box<RemoteDisk>),
 }
 
+impl Disk {
+    pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
+        match self {
+            Disk::Local(local_disk) => {
+                local_disk.set_disk_id_state(id).await;
+                Ok(())
+            }
+            Disk::Remote(remote_disk) => remote_disk.set_disk_id(id).await,
+        }
+    }
+}
+
 #[async_trait::async_trait]
 impl DiskAPI for Disk {
     fn to_string(&self) -> String {
@@ -1552,6 +1564,72 @@ mod tests {
         let _ = fs::remove_dir_all(&test_dir).await;
     }
 
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn local_disk_id_state_does_not_publish_to_the_process_registry() {
+        let local_dir = tempfile::tempdir().expect("local disk tempdir should be created");
+        let mut endpoint =
+            Endpoint::try_from(local_dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
+        endpoint.set_pool_index(0);
+        endpoint.set_set_index(0);
+        endpoint.set_disk_index(0);
+        let local_disk = LocalDisk::new(&endpoint, false).await.expect("local disk should initialize");
+        let disk = Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(local_disk), false)));
+        let disk_id = Uuid::new_v4();
+
+        disk.set_disk_id_state(Some(disk_id))
+            .await
+            .expect("local wrapper state should accept a disk ID");
+
+        let Disk::Local(local_disk) = &disk else {
+            panic!("test disk should remain local");
+        };
+        assert_eq!(local_disk.get_current_disk_id().await, Some(disk_id));
+        assert!(
+            !crate::runtime::global::current_ctx()
+                .local_disk_id_map()
+                .read()
+                .await
+                .contains_key(&disk_id),
+            "state-only startup publication must not update the process disk-ID registry"
+        );
+
+        disk.set_disk_id_state(None)
+            .await
+            .expect("local wrapper state should clear a disk ID");
+        assert_eq!(local_disk.get_current_disk_id().await, None);
+    }
+
+    #[tokio::test]
+    async fn remote_disk_id_state_delegates_some_and_none() {
+        let mut endpoint = Endpoint::try_from("http://remote-server:9000/data").expect("remote endpoint should parse");
+        endpoint.set_pool_index(0);
+        endpoint.set_set_index(0);
+        endpoint.set_disk_index(0);
+        let remote_disk = RemoteDisk::new(
+            &endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+            Arc::new(crate::cluster::rpc::TcpHttpInternodeDataTransport),
+        )
+        .await
+        .expect("remote disk should initialize");
+        let disk = Disk::Remote(Box::new(remote_disk));
+        let disk_id = Uuid::new_v4();
+
+        disk.set_disk_id_state(Some(disk_id))
+            .await
+            .expect("remote state should accept a disk ID");
+        assert_eq!(disk.get_disk_id().await.expect("remote disk ID should be readable"), Some(disk_id));
+
+        disk.set_disk_id_state(None)
+            .await
+            .expect("remote state should clear a disk ID");
+        assert_eq!(disk.get_disk_id().await.expect("remote disk ID should be readable"), None);
+    }
+
     #[tokio::test]
     async fn reset_health_for_store_init_retry_delegates_to_disk_variants() {
         let local_dir = tempfile::tempdir().unwrap();

crates/ecstore/src/layout/disks_layout.rs

@@ -203,7 +203,7 @@ fn get_all_sets<T: AsRef<str>>(set_drive_count: usize, is_ellipses: bool, args:
     for args in set_args.iter() {
         for arg in args {
             if unique_args.contains(arg) {
-                return Err(Error::other(format!("Input args {arg} has duplicate ellipses")));
+                return Err(Error::other("input arguments contain a duplicate endpoint after ellipsis expansion"));
             }
             unique_args.insert(arg);
         }
@@ -924,4 +924,15 @@ mod test {
             }
         }
     }
+
+    #[test]
+    fn layout_errors_do_not_echo_url_credentials() {
+        for volumes in [
+            vec!["http://:duplicate-secret@server/path", "http://:duplicate-secret@server/path"],
+            vec!["http://:ellipsis...secret@server/path"],
+        ] {
+            let err = DisksLayout::from_volumes(&volumes).unwrap_err();
+            assert!(!err.to_string().contains("secret"), "layout error leaked endpoint credentials: {err}");
+        }
+    }
 }

crates/ecstore/src/layout/endpoint.rs

@@ -88,6 +88,7 @@ impl TryFrom<&str> for Endpoint {
                 // - All field should be empty except Host and Path.
                 if !((url.scheme() == "http" || url.scheme() == "https")
                     && url.username().is_empty()
+                    && url.password().is_none()
                     && url.fragment().is_none()
                     && url.query().is_none())
                 {
@@ -366,6 +367,12 @@ mod test {
                 expected_type: None,
                 expected_err: Some(Error::other("invalid URL endpoint format")),
             },
+            TestCase {
+                arg: "http://:topsecret@server/path",
+                expected_endpoint: None,
+                expected_type: None,
+                expected_err: Some(Error::other("invalid URL endpoint format")),
+            },
             TestCase {
                 arg: "http://:/path",
                 expected_endpoint: None,
@@ -505,8 +512,18 @@ mod test {
         let endpoint = Endpoint::try_from("http://example.com:9000/path").unwrap();
         assert_eq!(endpoint.host_port(), "example.com:9000");
 
-        let endpoint_no_port = Endpoint::try_from("https://example.com/path").unwrap();
-        assert_eq!(endpoint_no_port.host_port(), "example.com");
+        for endpoint in [
+            Endpoint::try_from("http://example.com/path").unwrap(),
+            Endpoint::try_from("http://example.com:80/path").unwrap(),
+        ] {
+            assert_eq!(endpoint.host_port(), "example.com");
+        }
+        for endpoint in [
+            Endpoint::try_from("https://example.com/path").unwrap(),
+            Endpoint::try_from("https://example.com:443/path").unwrap(),
+        ] {
+            assert_eq!(endpoint.host_port(), "example.com");
+        }
 
         let file_endpoint = Endpoint::try_from("/tmp/data").unwrap();
         assert_eq!(file_endpoint.host_port(), "");

crates/ecstore/src/layout/format.rs

@@ -18,7 +18,7 @@ use serde::{Deserialize, Serialize};
 use serde_json::Error as JsonError;
 use uuid::Uuid;
 
-#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
+#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
 pub enum FormatMetaVersion {
     #[serde(rename = "1")]
     V1,
@@ -27,7 +27,7 @@ pub enum FormatMetaVersion {
     Unknown,
 }
 
-#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
+#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
 pub enum FormatBackend {
     #[serde(rename = "xl")]
     Erasure,
@@ -64,7 +64,7 @@ pub struct FormatErasureV3 {
     pub distribution_algo: DistributionAlgoVersion,
 }
 
-#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
+#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
 pub enum FormatErasureVersion {
     #[serde(rename = "1")]
     V1,
@@ -77,7 +77,7 @@ pub enum FormatErasureVersion {
     Unknown,
 }
 
-#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
+#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq, Hash)]
 pub enum DistributionAlgoVersion {
     #[serde(rename = "CRCMOD")]
     V1,
@@ -121,6 +121,15 @@ pub struct FormatV3 {
     pub disk_info: Option<DiskInfo>,
 }
 
+pub(crate) type SharedFormatIdentity<'a> = (
+    &'a FormatMetaVersion,
+    &'a FormatBackend,
+    &'a Uuid,
+    &'a FormatErasureVersion,
+    &'a [Vec<Uuid>],
+    &'a DistributionAlgoVersion,
+);
+
 impl TryFrom<&[u8]> for FormatV3 {
     type Error = JsonError;
 
@@ -198,52 +207,24 @@ impl FormatV3 {
     }
 
     pub fn check_other(&self, other: &FormatV3) -> Result<()> {
-        let mut tmp = other.clone();
-        let this = tmp.erasure.this;
-        tmp.erasure.this = Uuid::nil();
-
-        if self.erasure.sets.len() != other.erasure.sets.len() {
-            return Err(Error::other(format!(
-                "Expected number of sets {}, got {}",
-                self.erasure.sets.len(),
-                other.erasure.sets.len()
-            )));
+        if self.shared_identity() != other.shared_identity() {
+            return Err(Error::other("storage formats do not match"));
         }
 
-        for i in 0..self.erasure.sets.len() {
-            if self.erasure.sets[i].len() != other.erasure.sets[i].len() {
-                return Err(Error::other(format!(
-                    "Each set should be of same size, expected {}, got {}",
-                    self.erasure.sets[i].len(),
-                    other.erasure.sets[i].len()
-                )));
-            }
+        self.find_disk_index_by_disk_id(other.erasure.this).map(|_| ())
+    }
 
-            for j in 0..self.erasure.sets[i].len() {
-                if self.erasure.sets[i][j] != other.erasure.sets[i][j] {
-                    return Err(Error::other(format!(
-                        "UUID on positions {}:{} do not match with, expected {:?} got {:?}: (%w)",
-                        i,
-                        j,
-                        self.erasure.sets[i][j].to_string(),
-                        other.erasure.sets[i][j].to_string(),
-                    )));
-                }
-            }
-        }
-
-        for i in 0..tmp.erasure.sets.len() {
-            for j in 0..tmp.erasure.sets[i].len() {
-                if this == tmp.erasure.sets[i][j] {
-                    return Ok(());
-                }
-            }
-        }
-
-        Err(Error::other(format!(
-            "DriveID {:?} not found in any drive sets {:?}",
-            this, other.erasure.sets
-        )))
+    /// Fields that must agree across every disk in one erasure format,
+    /// excluding the disk-specific `this` UUID and runtime-only `disk_info`.
+    pub(crate) fn shared_identity(&self) -> SharedFormatIdentity<'_> {
+        (
+            &self.version,
+            &self.format,
+            &self.id,
+            &self.erasure.version,
+            &self.erasure.sets,
+            &self.erasure.distribution_algo,
+        )
     }
 }
 
@@ -437,6 +418,30 @@ mod test {
         assert!(result.is_ok());
     }
 
+    #[test]
+    fn test_check_other_rejects_shared_identity_mismatches() {
+        type FormatMutation = (&'static str, fn(&mut FormatV3));
+
+        let format = FormatV3::new(1, 2);
+        let mutations: [FormatMutation; 5] = [
+            ("meta version", |other| other.version = FormatMetaVersion::Unknown),
+            ("backend", |other| other.format = FormatBackend::ErasureSingle),
+            ("deployment id", |other| other.id = Uuid::new_v4()),
+            ("erasure version", |other| other.erasure.version = FormatErasureVersion::V2),
+            ("distribution algorithm", |other| {
+                other.erasure.distribution_algo = DistributionAlgoVersion::V2
+            }),
+        ];
+
+        for (field, mutate) in mutations {
+            let mut other = format.clone();
+            other.erasure.this = format.erasure.sets[0][0];
+            mutate(&mut other);
+
+            assert!(format.check_other(&other).is_err(), "{field} mismatch must be rejected");
+        }
+    }
+
     #[test]
     fn test_check_other_different_set_count() {
         let format1 = FormatV3::new(2, 4);

crates/ecstore/src/object_api/encryption.rs

@@ -0,0 +1,80 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use async_trait::async_trait;
+use http::{HeaderMap, HeaderValue};
+use std::collections::HashMap;
+use std::error::Error;
+use std::fmt::{Display, Formatter};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ReadEncryptionMode {
+    Direct { base_nonce: [u8; 12] },
+    Object,
+}
+
+pub struct ReadEncryptionMaterial {
+    pub key_bytes: [u8; 32],
+    pub mode: ReadEncryptionMode,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum EncryptionResolutionErrorKind {
+    InvalidRequest,
+    InvalidMetadata,
+    ServiceUnavailable,
+    DecryptionFailed,
+}
+
+#[derive(Debug)]
+pub struct EncryptionResolutionError {
+    kind: EncryptionResolutionErrorKind,
+    message: String,
+}
+
+impl EncryptionResolutionError {
+    pub fn new(kind: EncryptionResolutionErrorKind, message: impl Into<String>) -> Self {
+        Self {
+            kind,
+            message: message.into(),
+        }
+    }
+
+    pub fn kind(&self) -> EncryptionResolutionErrorKind {
+        self.kind
+    }
+}
+
+impl Display for EncryptionResolutionError {
+    fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result {
+        formatter.write_str(&self.message)
+    }
+}
+
+impl Error for EncryptionResolutionError {}
+
+pub struct ReadEncryptionRequest<'a> {
+    pub bucket: &'a str,
+    pub object: &'a str,
+    pub metadata: &'a HashMap<String, String>,
+    pub headers: &'a HeaderMap<HeaderValue>,
+}
+
+#[async_trait]
+pub trait ObjectEncryptionResolver: Send + Sync {
+    async fn resolve_read_material(
+        &self,
+        request: ReadEncryptionRequest<'_>,
+    ) -> Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError>;
+}

crates/ecstore/src/object_api/mod.rs

@@ -84,6 +84,7 @@ pub(crate) fn legacy_encrypted_range_seek_enabled() -> bool {
 }
 
 mod body_cache_hook;
+mod encryption;
 mod hook_slot;
 mod object_mutation_hook;
 mod readers;
@@ -98,6 +99,10 @@ pub use body_cache_hook::{
 pub(crate) use body_cache_hook::{
     get_object_body_cache_hook, get_object_body_cache_hook_suppressed, without_get_object_body_cache_hook,
 };
+pub use encryption::{
+    EncryptionResolutionError, EncryptionResolutionErrorKind, ObjectEncryptionResolver, ReadEncryptionMaterial,
+    ReadEncryptionMode, ReadEncryptionRequest,
+};
 pub(crate) use object_mutation_hook::notify_object_mutation;
 pub use object_mutation_hook::{ObjectMutationHook, register_object_mutation_hook, unregister_object_mutation_hook};
 pub use readers::*;

crates/ecstore/src/object_api/types.rs

@@ -273,29 +273,9 @@ impl ObjectInfo {
     }
 
     pub fn is_encrypted(&self) -> bool {
-        // Corresponding to the logic in rustfs/src/sse.rs/encryption_material_to_metadata function
-        use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
-
-        self.user_defined.keys().any(|key| {
-            let lower = key.to_ascii_lowercase();
-            lower.starts_with("x-minio-encryption-")
-                || lower.starts_with("x-minio-internal-server-side-encryption-")
-                || matches!(
-                    lower.as_str(),
-                    "x-minio-internal-encrypted-multipart"
-                        | "x-rustfs-encryption-key"
-                        | "x-rustfs-encryption-algorithm"
-                        | "x-rustfs-encryption-iv"
-                        | "x-rustfs-encryption-key-id"
-                        | "x-rustfs-encryption-context"
-                        | "x-rustfs-encryption-tag"
-                        | "x-amz-server-side-encryption-aws-kms-key-id"
-                        | SSEC_ALGORITHM_HEADER
-                        | SSEC_KEY_HEADER
-                        | SSEC_KEY_MD5_HEADER
-                        | "x-amz-server-side-encryption"
-                )
-        })
+        self.user_defined
+            .keys()
+            .any(|key| rustfs_utils::http::is_object_encryption_marker(key))
     }
 
     /// Maximum inline size for non-versioned objects (128 KiB).
@@ -339,26 +319,7 @@ impl ObjectInfo {
     }
 
     pub fn encryption_original_size(&self) -> std::io::Result<Option<i64>> {
-        let actual_size = rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE);
-        if let Some(size_str) = self
-            .user_defined
-            .get("x-rustfs-encryption-original-size")
-            .map(String::as_str)
-            .or_else(|| {
-                self.user_defined
-                    .get("x-amz-server-side-encryption-customer-original-size")
-                    .map(String::as_str)
-            })
-            .or(actual_size.as_deref())
-            && !size_str.is_empty()
-        {
-            let size = size_str
-                .parse::<i64>()
-                .map_err(|e| std::io::Error::other(format!("Failed to parse encryption original size: {e}")))?;
-            return Ok(Some(size));
-        }
-
-        Ok(None)
+        rustfs_utils::http::get_object_encryption_original_size(&self.user_defined)
     }
 
     pub fn decrypted_size(&self) -> std::io::Result<i64> {
@@ -388,9 +349,6 @@ impl ObjectInfo {
             return Ok(actual_size);
         }
 
-        // Check if object is encrypted
-        // Managed SSE stores original size in x-rustfs-encryption-original-size metadata
-        // SSE-C stores original size in x-amz-server-side-encryption-customer-original-size
         if let Some(size) = self.encryption_original_size()? {
             return Ok(size);
         }
@@ -881,6 +839,19 @@ mod tests {
         assert!(!object.is_inline_fast_path_eligible(), "transitioned objects must fall back");
     }
 
+    #[test]
+    fn minio_internal_encryption_metadata_is_not_treated_as_plaintext() {
+        let object = ObjectInfo {
+            user_defined: Arc::new(HashMap::from([(
+                "X-Minio-Internal-Server-Side-Encryption-Sealed-Key".to_string(),
+                "sealed".to_string(),
+            )])),
+            ..Default::default()
+        };
+
+        assert!(object.is_encrypted());
+    }
+
     #[test]
     fn versions_after_marker_handles_null_version_marker() {
         let first_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap();

crates/ecstore/src/runtime/instance.rs

@@ -46,6 +46,7 @@ use crate::bucket::metadata_sys::BucketMetadataSys;
 use crate::bucket::replication::{DynReplicationPool, ReplicationStats};
 use crate::disk::DiskStore;
 use crate::layout::endpoints::{EndpointServerPools, SetupType};
+use crate::object_api::ObjectEncryptionResolver;
 use crate::services::event_notification::EventNotifier;
 use crate::services::tier::tier::TierConfigMgr;
 use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
@@ -159,6 +160,8 @@ pub struct InstanceContext {
     /// workers (scanner/heal/tier/lifecycle) without touching another instance.
     /// Replaces the process-global cancel-token static.
     background_cancel_token: OnceLock<CancellationToken>,
+    /// Resolves object-encryption material at the application boundary.
+    object_encryption_resolver: OnceLock<Arc<dyn ObjectEncryptionResolver>>,
     tier_delete_journal_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
     transition_transaction_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
     #[cfg(test)]
@@ -197,6 +200,7 @@ impl InstanceContext {
             local_disk_set_drives: Arc::new(RwLock::new(Vec::new())),
             bucket_metadata_sys: std::sync::Mutex::new(None),
             background_cancel_token: OnceLock::new(),
+            object_encryption_resolver: OnceLock::new(),
             tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()),
             transition_transaction_recovery_stores: std::sync::Mutex::new(HashSet::new()),
             #[cfg(test)]
@@ -209,6 +213,19 @@ impl InstanceContext {
         self.lock_manager.clone()
     }
 
+    /// Install the application-owned object-encryption resolver once.
+    pub fn set_object_encryption_resolver(
+        &self,
+        resolver: Arc<dyn ObjectEncryptionResolver>,
+    ) -> Result<(), Arc<dyn ObjectEncryptionResolver>> {
+        self.object_encryption_resolver.set(resolver)
+    }
+
+    /// Return the configured object-encryption resolver, if startup installed one.
+    pub fn object_encryption_resolver(&self) -> Option<&dyn ObjectEncryptionResolver> {
+        self.object_encryption_resolver.get().map(Arc::as_ref)
+    }
+
     /// Set this instance's S3 region.
     ///
     /// Write-once: panics on a second write, preserving the startup fail-fast

crates/ecstore/src/runtime/sources.rs

@@ -27,7 +27,7 @@ use crate::{
     bucket::replication::{DynReplicationPool, ReplicationStats},
     config::{get_global_storage_class, get_global_storage_class_snapshot, set_global_storage_class, storageclass},
     disk::{DiskAPI, DiskOption, DiskStore, new_disk},
-    error::Result,
+    error::{Error, Result},
     layout::endpoints::{EndpointServerPools, SetupType},
     runtime::global::{
         GLOBAL_BOOT_TIME, GLOBAL_LIFECYCLE_SYS, GLOBAL_LOCAL_NODE_NAME_FALLBACK, GLOBAL_ROOT_DISK_THRESHOLD,
@@ -46,7 +46,6 @@ use crate::{
 use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
 use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
 use rustfs_io_metrics::internode_metrics::global_internode_metrics;
-use rustfs_kms::{ObjectEncryptionService, get_global_encryption_service};
 use rustfs_lock::client::LockClient;
 use s3s::dto::BucketLifecycleConfiguration;
 use s3s::region::Region;
@@ -105,10 +104,6 @@ pub(crate) fn record_erasure_write_quorum_failure(stage: &'static str, dominant_
     global_internode_metrics().record_erasure_write_quorum_failure(stage, dominant_error);
 }
 
-pub(crate) async fn object_encryption_service() -> Option<Arc<ObjectEncryptionService>> {
-    get_global_encryption_service().await
-}
-
 pub fn object_store_handle() -> Option<Arc<ECStore>> {
     resolve_object_store_handle()
 }
@@ -417,14 +412,14 @@ pub(crate) async fn clear_local_disk_id_map_for_test() {
     local_disk_id_map_handle().write().await.clear();
 }
 
-pub(crate) async fn record_local_disk_id(instance_ctx: &Arc<InstanceContext>, disk_id: Uuid, endpoint: String) {
-    instance_ctx.local_disk_id_map().write().await.insert(disk_id, endpoint);
-}
-
 pub(crate) async fn replace_local_disk_id(previous: Option<Uuid>, current: Option<Uuid>, endpoint: String) {
     let id_map = local_disk_id_map_handle();
     let mut disk_id_map = id_map.write().await;
-    if let Some(previous_id) = previous {
+    if let Some(previous_id) = previous
+        && disk_id_map
+            .get(&previous_id)
+            .is_some_and(|registered_endpoint| registered_endpoint == &endpoint)
+    {
         disk_id_map.remove(&previous_id);
     }
     if let Some(current_id) = current {
@@ -432,6 +427,53 @@ pub(crate) async fn replace_local_disk_id(previous: Option<Uuid>, current: Optio
     }
 }
 
+pub(crate) async fn reconcile_local_disk_ids(
+    instance_ctx: &InstanceContext,
+    pool_endpoints: &[String],
+    selected: &[(Uuid, String)],
+) {
+    let pool_endpoints = pool_endpoints.iter().map(String::as_str).collect::<HashSet<_>>();
+    let disk_id_map = instance_ctx.local_disk_id_map();
+    let mut disk_ids = disk_id_map.write().await;
+    disk_ids.retain(|_, registered_endpoint| !pool_endpoints.contains(registered_endpoint.as_str()));
+    disk_ids.extend(selected.iter().cloned());
+}
+
+pub(crate) async fn quarantine_local_disks(instance_ctx: &InstanceContext, endpoints: &[Endpoint]) -> Result<()> {
+    let slots = endpoints
+        .iter()
+        .map(|endpoint| {
+            Ok((
+                usize::try_from(endpoint.pool_idx).map_err(|_| Error::CorruptedFormat)?,
+                usize::try_from(endpoint.set_idx).map_err(|_| Error::CorruptedFormat)?,
+                usize::try_from(endpoint.disk_idx).map_err(|_| Error::CorruptedFormat)?,
+            ))
+        })
+        .collect::<Result<Vec<_>>>()?;
+
+    let local_disk_map = instance_ctx.local_disk_map();
+    let mut local_disks = local_disk_map.write().await;
+    for endpoint in endpoints {
+        local_disks.insert(endpoint.to_string(), None);
+    }
+    drop(local_disks);
+
+    let set_drives = instance_ctx.local_disk_set_drives();
+    let mut local_set_drives = set_drives.write().await;
+    if local_set_drives.is_empty() {
+        return Ok(());
+    }
+    for (pool_idx, set_idx, disk_idx) in slots {
+        let disk = local_set_drives
+            .get_mut(pool_idx)
+            .and_then(|sets| sets.get_mut(set_idx))
+            .and_then(|disks| disks.get_mut(disk_idx))
+            .ok_or(Error::CorruptedFormat)?;
+        *disk = None;
+    }
+    Ok(())
+}
+
 pub(crate) async fn record_local_disks(instance_ctx: &Arc<InstanceContext>, disks: Vec<DiskStore>) {
     let map = instance_ctx.local_disk_map();
     let mut global_local_disk_map = map.write().await;
@@ -558,10 +600,14 @@ pub(crate) async fn init_tier_config_mgr(store: Arc<ECStore>) -> Result<()> {
 
 #[cfg(test)]
 mod tests {
-    use super::{LockRegistry, local_node_name, set_local_node_name};
+    use super::{
+        LockRegistry, clear_local_disk_id_map_for_test, local_disk_path_by_id, local_node_name, reconcile_local_disk_ids,
+        replace_local_disk_id, set_local_node_name,
+    };
     use crate::disk::endpoint::Endpoint;
     use rustfs_lock::{LocalClient, LockClient};
     use std::{collections::HashMap, sync::Arc};
+    use uuid::Uuid;
 
     fn url_endpoint(raw: &str) -> Endpoint {
         Endpoint {
@@ -607,4 +653,78 @@ mod tests {
 
         assert_eq!(observed, next);
     }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn clearing_a_stale_disk_id_does_not_remove_another_endpoint() {
+        clear_local_disk_id_map_for_test().await;
+        let disk_id = Uuid::new_v4();
+        replace_local_disk_id(None, Some(disk_id), "endpoint-a".to_string()).await;
+
+        replace_local_disk_id(Some(disk_id), None, "endpoint-b".to_string()).await;
+
+        assert_eq!(local_disk_path_by_id(&disk_id).await, Some("endpoint-a".to_string()));
+        clear_local_disk_id_map_for_test().await;
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn reconciling_pool_disk_ids_preserves_other_endpoints() {
+        let instance_ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
+        let process_ctx = crate::runtime::global::current_ctx();
+        let bootstrap_ctx = crate::runtime::instance::bootstrap_ctx();
+        let retained_id = Uuid::new_v4();
+        let removed_id = Uuid::new_v4();
+        let selected_id = Uuid::new_v4();
+        let process_sentinel = Uuid::new_v4();
+        let bootstrap_sentinel = Uuid::new_v4();
+        instance_ctx.local_disk_id_map().write().await.extend([
+            (retained_id, "endpoint-a".to_string()),
+            (removed_id, "endpoint-b".to_string()),
+        ]);
+        process_ctx
+            .local_disk_id_map()
+            .write()
+            .await
+            .insert(process_sentinel, "endpoint-b".to_string());
+        bootstrap_ctx
+            .local_disk_id_map()
+            .write()
+            .await
+            .insert(bootstrap_sentinel, "endpoint-b".to_string());
+
+        reconcile_local_disk_ids(
+            &instance_ctx,
+            &["endpoint-b".to_string(), "endpoint-c".to_string()],
+            &[(selected_id, "endpoint-c".to_string())],
+        )
+        .await;
+
+        let disk_ids = instance_ctx.local_disk_id_map();
+        let disk_ids = disk_ids.read().await;
+        assert_eq!(disk_ids.get(&retained_id).map(String::as_str), Some("endpoint-a"));
+        assert_eq!(disk_ids.get(&removed_id), None);
+        assert_eq!(disk_ids.get(&selected_id).map(String::as_str), Some("endpoint-c"));
+        drop(disk_ids);
+        assert_eq!(
+            process_ctx
+                .local_disk_id_map()
+                .read()
+                .await
+                .get(&process_sentinel)
+                .map(String::as_str),
+            Some("endpoint-b")
+        );
+        assert_eq!(
+            bootstrap_ctx
+                .local_disk_id_map()
+                .read()
+                .await
+                .get(&bootstrap_sentinel)
+                .map(String::as_str),
+            Some("endpoint-b")
+        );
+        process_ctx.local_disk_id_map().write().await.remove(&process_sentinel);
+        bootstrap_ctx.local_disk_id_map().write().await.remove(&bootstrap_sentinel);
+    }
 }

crates/ecstore/src/services/notification_sys.rs

@@ -2044,15 +2044,10 @@ fn synthesized_disks(host: &str, endpoints: &EndpointServerPools, state: ItemSta
 /// Whether `peer_host` refers to the same node as an endpoint whose
 /// `host_port()` is `ep_host_port`.
 ///
-/// `PeerRestClient::host` is an `XHost`, which resolves names to an address on
-/// construction (`hosts_sorted` -> `XHost::try_from` -> `to_socket_addrs`), so
-/// `peer_host` is the resolved `IP:port`. An endpoint's `host_port()`, however,
-/// is `url.host():port` — still the raw `hostname:port` on hostname-based
-/// deployments. A plain string compare therefore misses on hostname clusters,
-/// leaving the synthesized/degraded drive list empty and `unknownDisks` at 0
-/// (rustfs/rustfs#4607 follow-up). Compare directly first (fast path / IP
-/// deployments), then canonicalize the endpoint side through the same `XHost`
-/// resolution and compare again.
+/// Current topology clients preserve the endpoint `hostname:port`, so the
+/// direct comparison is the normal path. The resolution fallback keeps
+/// compatibility with older or manually constructed clients whose `XHost`
+/// contains a resolved `IP:port` (rustfs/rustfs#4607 follow-up).
 fn endpoint_host_matches(peer_host: &str, ep_host_port: &str) -> bool {
     if peer_host == ep_host_port {
         return true;

crates/ecstore/src/services/tier/tier.rs

@@ -72,6 +72,7 @@ use crate::{
     cluster::rpc::peer_rest_client::{PeerRestClient, PeerTierMutationState},
     config::com::{CONFIG_PREFIX, read_config, read_config_with_metadata},
     disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET},
+    layout::endpoints::EndpointServerPools,
     object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader},
     runtime::sources as runtime_sources,
     set_disk::get_lock_acquire_timeout,
@@ -904,14 +905,17 @@ async fn remote_tier_mutation_peers() -> io::Result<Vec<Arc<dyn TierMutationPeer
     let Some(endpoints) = runtime_sources::endpoint_pools() else {
         return Err(tier_mutation_replay_error("cluster endpoint topology is not initialized"));
     };
-    let remote_host_count = endpoints.hosts_sorted().iter().flatten().count();
-    let (peers, _) = PeerRestClient::new_clients(endpoints).await;
+    remote_tier_mutation_peers_from_topology(endpoints).await
+}
+
+async fn remote_tier_mutation_peers_from_topology(endpoints: EndpointServerPools) -> io::Result<Vec<Arc<dyn TierMutationPeer>>> {
+    let (peers, _, remote_topology_hosts) = PeerRestClient::new_clients_with_topology(endpoints).await;
     let peers = peers
         .into_iter()
         .flatten()
         .map(|peer| Arc::new(peer) as Arc<dyn TierMutationPeer>)
         .collect::<Vec<_>>();
-    ensure_complete_tier_mutation_commit_peer_set(peers.len(), remote_host_count)?;
+    ensure_complete_tier_mutation_commit_peer_set(peers.len(), remote_topology_hosts.len())?;
     Ok(peers)
 }
 
@@ -4307,6 +4311,34 @@ fn tier_config_not_initialized_error(operation: &str) -> std::io::Error {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::layout::{
+        endpoint::Endpoint,
+        endpoints::{Endpoints, PoolEndpoints, SetupType},
+    };
+
+    struct SetupTypeGuard {
+        previous: SetupType,
+    }
+
+    impl SetupTypeGuard {
+        async fn switch_to(next: SetupType) -> Self {
+            let previous = runtime_sources::current_setup_type().await;
+            runtime_sources::set_setup_type(next).await;
+            Self { previous }
+        }
+    }
+
+    impl Drop for SetupTypeGuard {
+        fn drop(&mut self) {
+            let previous = self.previous.clone();
+            let handle = tokio::runtime::Handle::current();
+            tokio::task::block_in_place(|| {
+                handle.block_on(async move {
+                    runtime_sources::set_setup_type(previous).await;
+                });
+            });
+        }
+    }
 
     fn build_s3_tier(name: &str) -> TierConfig {
         TierConfig {
@@ -6354,6 +6386,42 @@ mod tests {
         assert!(err.to_string().contains("without peer commit clients"), "{err}");
     }
 
+    #[tokio::test(flavor = "multi_thread")]
+    #[serial_test::serial]
+    async fn tier_mutation_peer_composition_preserves_unresolved_topology_slots() {
+        let mut endpoints = Vec::new();
+        for disk_index in 0..4 {
+            let mut endpoint = Endpoint::try_from(format!("http://rustfs-{disk_index}.invalid:9000/data{disk_index}").as_str())
+                .expect("unresolved topology endpoint should parse without DNS");
+            endpoint.is_local = disk_index == 0;
+            endpoint.set_pool_index(0);
+            endpoint.set_set_index(0);
+            endpoint.set_disk_index(disk_index);
+            endpoints.push(endpoint);
+        }
+        let topology = EndpointServerPools::from(vec![PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 4,
+            endpoints: Endpoints::from(endpoints),
+            cmd_line: "unresolved-tier-mutation-topology".to_string(),
+            platform: "test".to_string(),
+        }]);
+        let _setup_type_guard = SetupTypeGuard::switch_to(SetupType::DistErasure).await;
+
+        let peers = remote_tier_mutation_peers_from_topology(topology)
+            .await
+            .expect("every unresolved remote topology slot should retain a tier mutation client");
+        assert_eq!(
+            peers.iter().map(|peer| peer.peer_label()).collect::<Vec<_>>(),
+            vec![
+                "http://rustfs-1.invalid:9000".to_string(),
+                "http://rustfs-2.invalid:9000".to_string(),
+                "http://rustfs-3.invalid:9000".to_string(),
+            ]
+        );
+    }
+
     #[tokio::test]
     async fn coordinator_fanout_prepare_failure_aborts_prepared_peers_without_cas() {
         let manager = TierConfigMgr::new();

crates/ecstore/src/set_disk/metadata.rs

@@ -505,9 +505,7 @@ impl SetDisks {
     }
 
     fn file_info_has_encryption_metadata(meta: &FileInfo) -> bool {
-        meta.metadata
-            .keys()
-            .any(|name| http::is_encryption_metadata_key(name) || http::is_sse_header(name))
+        meta.metadata.keys().any(|name| http::is_object_encryption_marker(name))
     }
 
     fn starts_with_ignore_ascii_case(value: &str, prefix: &str) -> bool {

crates/ecstore/src/set_disk/mod.rs

@@ -110,7 +110,10 @@ use crate::{
     object_api::{GetObjectReader, ObjectInfo, PutObjReader},
     // event::name::EventName,
     services::event_notification::{EventArgs, send_event},
-    store::init_format::{get_format_erasure_in_quorum, load_format_erasure, load_format_erasure_all, save_format_file},
+    store::init_format::{
+        formats_match_reference_slots, get_format_erasure_in_quorum, load_format_erasure, load_format_erasure_all,
+        save_format_file,
+    },
 };
 use bytes::Bytes;
 use bytesize::ByteSize;
@@ -144,15 +147,17 @@ use rustfs_object_capacity::capacity_scope::{
     CapacityScope, CapacityScopeDisk, current_dirty_generation, record_capacity_scope, record_global_dirty_scope,
 };
 use rustfs_s3_types::EventName;
+#[cfg(test)]
+use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
 use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
 use rustfs_utils::http::headers::AMZ_STORAGE_CLASS;
 use rustfs_utils::http::headers::{
     CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
 };
 use rustfs_utils::http::{
-    SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE,
-    SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_OPERATION_ID, contains_key_str,
-    get_header_map, get_str, insert_str, is_encryption_metadata_key, remove_header_map,
+    SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
+    SUFFIX_RESTORE_OPERATION_ID, contains_key_str, get_header_map, get_str, insert_str, is_object_encryption_marker,
+    remove_header_map,
 };
 use rustfs_utils::{
     HashAlgorithm,
@@ -667,10 +672,7 @@ pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, S
 }
 
 fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
-    metadata.keys().any(|key| is_encryption_metadata_key(key))
-        || metadata.contains_key(SSEC_ALGORITHM_HEADER)
-        || metadata.contains_key(SSEC_KEY_HEADER)
-        || metadata.contains_key(SSEC_KEY_MD5_HEADER)
+    metadata.keys().any(|key| is_object_encryption_marker(key))
 }
 
 /// Per-set memoized capacity dirty scope.
@@ -5849,23 +5851,27 @@ mod tests {
             crate::disk::DataDirDeleteStatus::Deleted
         );
         release_slow_candidate.notify_one();
-        for _ in 0..10 {
-            tokio::task::yield_now().await;
-        }
-        assert_eq!(
-            disk2
-                .delete_data_dir(
-                    bucket,
-                    data_dir,
-                    DeleteOptions {
-                        recursive: true,
-                        ..Default::default()
-                    },
-                )
-                .await
-                .expect("a token acquired after the deadline must be released"),
-            crate::disk::DataDirDeleteStatus::Deleted
-        );
+        timeout(Duration::from_secs(1), async {
+            loop {
+                match disk2
+                    .delete_data_dir(
+                        bucket,
+                        data_dir,
+                        DeleteOptions {
+                            recursive: true,
+                            ..Default::default()
+                        },
+                    )
+                    .await
+                    .expect("a token acquired after the deadline must be released")
+                {
+                    crate::disk::DataDirDeleteStatus::Deleted => return,
+                    crate::disk::DataDirDeleteStatus::Deferred => tokio::time::sleep(Duration::from_millis(1)).await,
+                }
+            }
+        })
+        .await
+        .expect("late snapshot lease cleanup must finish within the bounded wait");
     }
 
     #[tokio::test(start_paused = true)]

crates/ecstore/src/set_disk/ops/heal.rs

@@ -1373,11 +1373,24 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
     async fn heal_format(&self, dry_run: bool) -> Result<(HealResultItem, Option<Error>)> {
         let disks = self.disks.read().await.clone();
         let (formats, errs) = load_format_erasure_all(&disks, true).await;
-        let ref_format = match get_format_erasure_in_quorum(&formats) {
-            Ok(format) => format,
+        if errs.iter().any(|err| {
+            matches!(
+                err,
+                Some(DiskError::InconsistentDisk | DiskError::CorruptedFormat | DiskError::CorruptedBackend)
+            )
+        }) {
+            return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat)));
+        }
+        let slot_offset = self
+            .set_index
+            .checked_mul(self.set_drive_count)
+            .ok_or_else(|| Error::other("erasure set slot offset overflow"))?;
+        let ref_format = match get_format_erasure_in_quorum(&formats, slot_offset) {
+            Ok(format) if format.shared_identity() == self.format.shared_identity() => format,
+            Ok(_) => return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat))),
             Err(err) => {
                 let can_use_cached_layout = count_errs(&errs, &DiskError::UnformattedDisk) > 0
-                    && formats.iter().flatten().all(|format| self.format.check_other(format).is_ok())
+                    && formats_match_reference_slots(&formats, &self.format, slot_offset)
                     && errs
                         .iter()
                         .all(|err| err.is_none() || matches!(err, Some(DiskError::UnformattedDisk)));
@@ -1388,6 +1401,9 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
                 }
             }
         };
+        if !formats_match_reference_slots(&formats, &ref_format, slot_offset) {
+            return Ok((HealResultItem::default(), Some(StorageError::CorruptedFormat)));
+        }
 
         let endpoints = crate::layout::endpoints::Endpoints::from(self.set_endpoints.clone());
         let before_drives = crate::layout::set_heal::formats_to_drives_info(&endpoints, &formats, &errs);
@@ -1543,11 +1559,16 @@ mod heal_result_report_tests {
     use crate::disk::error::DiskError;
     use crate::disk::format::FormatV3;
     use crate::disk::{DiskAPI as _, DiskOption, DiskStore, RUSTFS_META_TMP_BUCKET, ReadOptions, new_disk};
+    use crate::error::Error;
     use crate::object_api::{ObjectOptions, PutObjReader};
     use crate::set_disk::ops::object::hermetic_set_disks_support::hermetic_set_disks_isolated;
     use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
+    use crate::storage_api_contracts::heal::HealOperations as _;
     use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _};
-    use crate::{config::storageclass, store::init_format::save_format_file};
+    use crate::{
+        config::storageclass,
+        store::init_format::{load_format_erasure, save_format_file},
+    };
     use rustfs_common::heal_channel::{DriveState, HealOpts, HealScanMode};
     use rustfs_filemeta::{BLOCK_SIZE_V2, FileInfo, ObjectPartInfo, TRANSITION_COMPLETE};
     use std::sync::Arc;
@@ -1863,6 +1884,44 @@ mod heal_result_report_tests {
         }
     }
 
+    #[tokio::test]
+    async fn format_heal_cached_layout_rejects_a_disk_from_another_slot() {
+        let mut _temp_dirs = Vec::new();
+        let mut endpoints = Vec::new();
+        let mut disks = Vec::new();
+        for disk_index in 0..3 {
+            let (temp_dir, mut endpoint, disk) = real_disk().await;
+            endpoint.set_pool_index(0);
+            endpoint.set_set_index(0);
+            endpoint.set_disk_index(disk_index);
+            _temp_dirs.push(temp_dir);
+            endpoints.push(endpoint);
+            disks.push(Some(disk));
+        }
+        let set = set_disks_with(disks.clone(), endpoints, 1).await;
+        let mut wrong_slot = set.format.clone();
+        wrong_slot.erasure.this = set.format.erasure.sets[0][1];
+        save_format_file(&disks[0], &Some(wrong_slot))
+            .await
+            .expect("wrong-slot format fixture should be saved");
+        let mut correct_slot = set.format.clone();
+        correct_slot.erasure.this = set.format.erasure.sets[0][2];
+        save_format_file(&disks[2], &Some(correct_slot))
+            .await
+            .expect("correct format fixture should be saved");
+
+        let (_, heal_err) = set
+            .heal_format(false)
+            .await
+            .expect("format heal should report the quorum failure in its result");
+
+        assert!(matches!(heal_err, Some(Error::CorruptedFormat)));
+        let unformatted = load_format_erasure(disks[1].as_ref().expect("second disk should be online"), true)
+            .await
+            .expect_err("a rejected fallback must not format the missing slot");
+        assert_eq!(unformatted, DiskError::UnformattedDisk);
+    }
+
     // Regression for #955: an offline disk must contribute exactly one drive
     // record. Before the fix the offline branch fell through and pushed a second
     // (Corrupt) record for the same disk, so `before/after.drives` grew to

crates/ecstore/src/set_disk/ops/locking.rs

@@ -343,20 +343,23 @@ impl SetDisks {
             }
         };
 
-        // The drive's format may place it in a different erasure set than this
-        // one. Claiming a misplaced drive into `self.disks` would let two sets
-        // manage the same drive and degrade together, so reject it here
-        // (backlog#799 B19).
-        if set_idx != self.set_index {
+        // Claiming a misplaced drive into `self.disks` would let two slots or
+        // sets manage the same drive and degrade together (backlog#799 B19).
+        if set_idx != self.set_index || self.set_endpoints.get(disk_idx) != Some(ep) {
             warn!(
-                "renew_disk: drive {:?} belongs to set {} but is being renewed on set {}; skipping",
-                ep, set_idx, self.set_index
+                endpoint = %ep,
+                format_set_index = set_idx,
+                format_disk_index = disk_idx,
+                endpoint_pool_index = ep.pool_idx,
+                endpoint_set_index = ep.set_idx,
+                endpoint_disk_index = ep.disk_idx,
+                expected_pool_index = self.pool_index,
+                expected_set_index = self.set_index,
+                "renew_disk rejected a drive whose endpoint and format do not identify the same topology slot"
             );
             return;
         }
 
-        // Check that the endpoint matches
-
         let _ = new_disk.set_disk_id(Some(fm.erasure.this)).await;
         new_disk.enable_health_check();
 
@@ -715,6 +718,108 @@ mod tests {
         drop(temp_dirs);
     }
 
+    #[tokio::test]
+    async fn renew_disk_rejects_a_format_from_another_slot_or_cluster() {
+        let disk_count = 3;
+        let format = FormatV3::new(1, disk_count);
+        let mut temp_dirs = Vec::with_capacity(disk_count);
+        let mut endpoints = Vec::with_capacity(disk_count);
+        let mut fixture_disks = Vec::with_capacity(disk_count);
+
+        for disk_idx in 0..disk_count {
+            let (temp_dir, endpoint, disk) = make_formatted_local_disk(disk_idx, &format).await;
+            temp_dirs.push(temp_dir);
+            endpoints.push(endpoint);
+            fixture_disks.push(disk);
+        }
+
+        let set_disks = SetDisks::new(
+            "test-owner".to_string(),
+            Arc::new(RwLock::new(vec![Some(fixture_disks[0].clone()), None, None])),
+            disk_count,
+            disk_count / 2,
+            0,
+            0,
+            endpoints.clone(),
+            format.clone(),
+            Vec::new(),
+        )
+        .await;
+
+        let mut other_cluster_format = format.clone();
+        other_cluster_format.id = Uuid::new_v4();
+        other_cluster_format.erasure.this = format.erasure.sets[0][2];
+        save_format_file(&Some(fixture_disks[2].clone()), &Some(other_cluster_format))
+            .await
+            .expect("other-cluster format should be written for the rejection test");
+
+        set_disks.renew_disk(&endpoints[2]).await;
+
+        let disks = set_disks.get_disks_internal().await;
+        assert_eq!(
+            disks[0]
+                .as_ref()
+                .expect("the canonical first slot must remain attached")
+                .endpoint(),
+            endpoints[0]
+        );
+        assert!(
+            disks[2].is_none(),
+            "a disk from another deployment must remain detached even when its slot UUID matches"
+        );
+
+        let mut correct_format = format.clone();
+        correct_format.erasure.this = format.erasure.sets[0][2];
+        let replacement_disk = new_disk(
+            &endpoints[2],
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+        )
+        .await
+        .expect("third endpoint should reopen after other-cluster rejection");
+        save_format_file(&Some(replacement_disk), &Some(correct_format))
+            .await
+            .expect("correct slot format should be restored");
+
+        set_disks.renew_disk(&endpoints[2]).await;
+
+        let disks = set_disks.get_disks_internal().await;
+        assert_eq!(
+            disks[0]
+                .as_ref()
+                .expect("the canonical first slot must remain attached")
+                .endpoint(),
+            endpoints[0]
+        );
+        assert_eq!(disks[2].as_ref().expect("the restored third slot should attach").endpoint(), endpoints[2]);
+
+        let third_disk = disks[2].clone();
+        let mut wrong_slot_format = format.clone();
+        wrong_slot_format.erasure.this = format.erasure.sets[0][0];
+        save_format_file(&third_disk, &Some(wrong_slot_format))
+            .await
+            .expect("wrong-slot format should be written for the rejection test");
+        set_disks.disks.write().await[2] = None;
+
+        let mut misplaced_endpoint = endpoints[2].clone();
+        misplaced_endpoint.set_disk_index(0);
+        set_disks.renew_disk(&misplaced_endpoint).await;
+
+        let disks = set_disks.get_disks_internal().await;
+        assert_eq!(
+            disks[0]
+                .as_ref()
+                .expect("the canonical first slot must remain attached")
+                .endpoint(),
+            endpoints[0]
+        );
+        assert!(disks[2].is_none(), "a disk claiming another endpoint's slot must remain detached");
+
+        drop(temp_dirs);
+    }
+
     // SetDisks split P0 (#816): the borrow handle must mirror the core state and
     // the List operation family must run identically through it.
     #[tokio::test]

crates/ecstore/src/set_disk/ops/object.rs

@@ -42,6 +42,7 @@ use crate::object_api::{GetObjectBodySource, get_object_body_cache_hook_suppress
 use crate::services::tier::tier::{TierConfigMgr, TierOperationLease};
 use crate::store::ECStore;
 use futures::FutureExt as _;
+use http::HeaderValue;
 use std::future::Future;
 
 fn erasure_from_file_info(fi: &FileInfo, uses_legacy: bool) -> Result<coding::Erasure> {
@@ -49,6 +50,17 @@ fn erasure_from_file_info(fi: &FileInfo, uses_legacy: bool) -> Result<coding::Er
         .map_err(Error::from)
 }
 
+async fn get_object_reader_with_context(
+    ctx: &InstanceContext,
+    reader: Box<dyn AsyncRead + Unpin + Send + Sync>,
+    range: Option<HTTPRangeSpec>,
+    object_info: &ObjectInfo,
+    opts: &ObjectOptions,
+    headers: &HeaderMap<HeaderValue>,
+) -> Result<(GetObjectReader, usize, i64)> {
+    GetObjectReader::new_with_resolver(reader, range, object_info, opts, headers, ctx.object_encryption_resolver()).await
+}
+
 /// Length of the full plaintext body when — and only when — this read's output
 /// is exactly the object's complete plaintext, so the app-layer body cache may
 /// serve it in place of the erasure read.
@@ -713,7 +725,8 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
                             size_bucket,
                         );
                         record_get_object_reader_path_observation(GET_OBJECT_PATH_CODEC_STREAMING, object_class, size_bucket);
-                        let (mut reader, _offset, _length) = GetObjectReader::new(stream, range, &object_info, opts, &h).await?;
+                        let (mut reader, _offset, _length) =
+                            get_object_reader_with_context(&self.ctx, stream, range, &object_info, opts, &h).await?;
                         // Carry the hook probe result so the app layer skips its
                         // now-redundant lookup on the streaming miss path (ODC-16).
                         reader.body_source = body_source;
@@ -745,7 +758,8 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
         let (rd, wd) = tokio::io::duplex(duplex_buffer_size);
         debug!(bucket, object, duplex_buffer_size, "Created duplex pipe for object data transfer");
 
-        let (mut reader, offset, length) = GetObjectReader::new(Box::new(rd), range, &object_info, opts, &h).await?;
+        let (mut reader, offset, length) =
+            get_object_reader_with_context(&self.ctx, Box::new(rd), range, &object_info, opts, &h).await?;
         // Carry the hook probe result so the app layer skips its now-redundant
         // lookup on the streaming miss path (ODC-16).
         reader.body_source = body_source;
@@ -4536,6 +4550,61 @@ mod erasure_construction_tests {
     }
 }
 
+#[cfg(test)]
+mod object_encryption_resolver_wiring_tests {
+    use super::*;
+    use crate::object_api::{EncryptionResolutionError, ObjectEncryptionResolver, ReadEncryptionMaterial, ReadEncryptionRequest};
+    use std::io::Cursor;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+
+    struct CountingResolver {
+        calls: AtomicUsize,
+    }
+
+    #[async_trait::async_trait]
+    impl ObjectEncryptionResolver for CountingResolver {
+        async fn resolve_read_material(
+            &self,
+            _request: ReadEncryptionRequest<'_>,
+        ) -> std::result::Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError> {
+            self.calls.fetch_add(1, Ordering::Relaxed);
+            Ok(None)
+        }
+    }
+
+    #[tokio::test]
+    async fn get_object_reader_forwards_instance_resolver() {
+        let resolver = Arc::new(CountingResolver {
+            calls: AtomicUsize::new(0),
+        });
+        let ctx = InstanceContext::new();
+        assert!(
+            ctx.set_object_encryption_resolver(resolver.clone()).is_ok(),
+            "fresh context should accept resolver"
+        );
+        let object_info = ObjectInfo {
+            bucket: "bucket".to_string(),
+            name: "object".to_string(),
+            size: 1,
+            user_defined: Arc::new(HashMap::from([("x-amz-server-side-encryption".to_string(), "AES256".to_string())])),
+            ..Default::default()
+        };
+
+        let result = get_object_reader_with_context(
+            &ctx,
+            Box::new(Cursor::new(Vec::<u8>::new())),
+            None,
+            &object_info,
+            &ObjectOptions::default(),
+            &HeaderMap::new(),
+        )
+        .await;
+
+        assert!(result.is_err(), "resolver returning no material must fail closed");
+        assert_eq!(resolver.calls.load(Ordering::Relaxed), 1);
+    }
+}
+
 #[cfg(test)]
 pub(in crate::set_disk::ops) mod hermetic_set_disks_support {
     //! Shared hermetic `SetDisks` construction for the ops tests below: the

crates/ecstore/src/store/bucket.rs

@@ -310,12 +310,13 @@ impl ECStore {
         meta.set_created(opts.created_at);
 
         if opts.lock_enabled {
-            meta.object_lock_config_xml = crate::bucket::utils::serialize::<ObjectLockConfiguration>(&enableObjcetLockConfig)?;
-            meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&enableVersioningConfig)?;
+            meta.object_lock_config_xml =
+                crate::bucket::utils::serialize::<ObjectLockConfiguration>(&ENABLED_OBJECT_LOCK_CONFIG)?;
+            meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
         }
 
         if opts.versioning_enabled {
-            meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&enableVersioningConfig)?;
+            meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
         }
 
         await_bucket_namespace_operation(

crates/ecstore/src/store/heal.rs

@@ -30,6 +30,7 @@ impl ECStore {
         };
 
         let mut count_no_heal = 0;
+        let mut first_error = None;
         for pool in self.pools.iter() {
             let (mut result, err) = pool.heal_format(dry_run).await?;
             if let Some(err) = err {
@@ -37,8 +38,8 @@ impl ECStore {
                     StorageError::NoHealRequired => {
                         count_no_heal += 1;
                     }
-                    _ => {
-                        continue;
+                    err => {
+                        first_error.get_or_insert(err);
                     }
                 }
             }
@@ -47,6 +48,9 @@ impl ECStore {
             r.before.drives.append(&mut result.before.drives);
             r.after.drives.append(&mut result.after.drives);
         }
+        if let Some(err) = first_error {
+            return Ok((r, Some(err)));
+        }
         if count_no_heal == self.pools.len() {
             info!(
                 event = EVENT_HEAL_FORMAT_COMPLETED,
@@ -165,3 +169,134 @@ impl ECStore {
         Err(StorageError::NotImplemented)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::disk::{DiskOption, format::FormatV3, new_disk};
+    use crate::layout::endpoints::{Endpoints, PoolEndpoints};
+    use crate::store::init_format::{load_format_erasure, save_format_file};
+
+    #[tokio::test]
+    async fn handle_heal_format_continues_after_a_pool_error() {
+        let canonical_format = FormatV3::new(1, 3);
+        let mut foreign_format = canonical_format.clone();
+        foreign_format.id = Uuid::new_v4();
+        let mut temp_dirs = Vec::new();
+        let mut endpoints = Vec::new();
+        let mut disks = Vec::new();
+
+        for disk_index in 0..3 {
+            let temp_dir = tempfile::tempdir().expect("temporary disk root should be created");
+            let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("temporary path should be UTF-8"))
+                .expect("temporary endpoint should parse");
+            endpoint.set_pool_index(0);
+            endpoint.set_set_index(0);
+            endpoint.set_disk_index(disk_index);
+            let disk = new_disk(
+                &endpoint,
+                &DiskOption {
+                    cleanup: false,
+                    health_check: false,
+                },
+            )
+            .await
+            .expect("temporary disk should open");
+            let mut disk_format = foreign_format.clone();
+            disk_format.erasure.this = foreign_format.erasure.sets[0][disk_index];
+            save_format_file(&Some(disk.clone()), &Some(disk_format))
+                .await
+                .expect("foreign format should be written");
+            temp_dirs.push(temp_dir);
+            endpoints.push(endpoint);
+            disks.push(Some(disk));
+        }
+
+        let pool_endpoints = PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 3,
+            endpoints: Endpoints::from(endpoints),
+            cmd_line: "foreign-format-majority-test".to_string(),
+            platform: "test".to_string(),
+        };
+        let pool = Sets::new(disks, &pool_endpoints, &canonical_format, 0, 1)
+            .await
+            .expect("test pool should build around the cached canonical format");
+
+        let mut recoverable_format = FormatV3::new(1, 3);
+        recoverable_format.id = canonical_format.id;
+        let mut recoverable_temp_dirs = Vec::new();
+        let mut recoverable_endpoints = Vec::new();
+        let mut recoverable_disks = Vec::new();
+        let mut unformatted_disk = None;
+        for disk_index in 0..3 {
+            let temp_dir = tempfile::tempdir().expect("temporary disk root should be created");
+            let mut endpoint = Endpoint::try_from(temp_dir.path().to_str().expect("temporary path should be UTF-8"))
+                .expect("temporary endpoint should parse");
+            endpoint.set_pool_index(1);
+            endpoint.set_set_index(0);
+            endpoint.set_disk_index(disk_index);
+            let disk = new_disk(
+                &endpoint,
+                &DiskOption {
+                    cleanup: false,
+                    health_check: false,
+                },
+            )
+            .await
+            .expect("temporary disk should open");
+            if disk_index < 2 {
+                let mut disk_format = recoverable_format.clone();
+                disk_format.erasure.this = recoverable_format.erasure.sets[0][disk_index];
+                save_format_file(&Some(disk.clone()), &Some(disk_format))
+                    .await
+                    .expect("recoverable format should be written");
+            } else {
+                unformatted_disk = Some(disk.clone());
+            }
+            recoverable_temp_dirs.push(temp_dir);
+            recoverable_endpoints.push(endpoint);
+            recoverable_disks.push(Some(disk));
+        }
+        let recoverable_pool_endpoints = PoolEndpoints {
+            legacy: false,
+            set_count: 1,
+            drives_per_set: 3,
+            endpoints: Endpoints::from(recoverable_endpoints),
+            cmd_line: "recoverable-format-test".to_string(),
+            platform: "test".to_string(),
+        };
+        let recoverable_pool = Sets::new(recoverable_disks, &recoverable_pool_endpoints, &recoverable_format, 1, 1)
+            .await
+            .expect("recoverable test pool should build");
+
+        let endpoint_pools = EndpointServerPools::from(vec![pool_endpoints.clone(), recoverable_pool_endpoints.clone()]);
+        let store = ECStore {
+            id: canonical_format.id,
+            disk_map: HashMap::new(),
+            pools: vec![pool, recoverable_pool],
+            peer_sys: S3PeerSys::new(&endpoint_pools),
+            pool_meta: RwLock::new(PoolMeta::default()),
+            rebalance_meta: RwLock::new(None),
+            decommission_cancelers: RwLock::new(Vec::new()),
+            start_gate: Mutex::new(()),
+            pool_meta_save_gate: Mutex::new(()),
+            ctx: crate::runtime::instance::bootstrap_ctx(),
+        };
+
+        let (result, err) = store
+            .handle_heal_format(false)
+            .await
+            .expect("format heal should return the typed pool error");
+        assert!(
+            matches!(err, Some(StorageError::CorruptedFormat)),
+            "foreign format majority must not be downgraded to a successful heal: {err:?}"
+        );
+        assert_eq!(result.disk_count, 3, "the recoverable pool should still be inspected");
+        let healed = load_format_erasure(&unformatted_disk.expect("the unformatted disk handle should be retained"), true)
+            .await
+            .expect("the later pool should be healed despite the first pool error");
+        assert_eq!(healed.erasure.this, recoverable_format.erasure.sets[0][2]);
+    }
+}

crates/ecstore/src/store/init.rs

@@ -101,6 +101,10 @@ fn should_retry_local_decommission_resume(err: &Error, attempt: usize) -> bool {
     matches!(err, Error::ConfigNotFound) && attempt < LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES
 }
 
+fn should_retry_format_load(err: &Error) -> bool {
+    !matches!(err, Error::CorruptedFormat)
+}
+
 fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_meta_loaded: bool) -> bool {
     rebalance_meta_loaded && !decommission_running
 }
@@ -294,7 +298,7 @@ impl ECStore {
             // periodic monitoring until format loading succeeds. Startup RPC
             // failures can still spawn recovery probes for peers that come up
             // after this node.
-            let (disks, errs) = init_format::init_disks(
+            let (mut disks, errs) = init_format::init_disks(
                 &pool_eps.endpoints,
                 &DiskOption {
                     cleanup: true,
@@ -309,9 +313,10 @@ impl ECStore {
                 let mut times = 0;
                 let mut interval = 1;
                 loop {
-                    match init_format::connect_load_init_formats(
+                    match init_format::connect_load_init_formats_with_instance_ctx(
+                        &instance_ctx,
                         pool_first_is_local,
-                        &disks,
+                        &mut disks,
                         pool_eps.set_count,
                         pool_eps.drives_per_set,
                         deployment_id,
@@ -319,6 +324,7 @@ impl ECStore {
                     .await
                     {
                         Ok(fm) => break Ok(fm),
+                        Err(e) if !should_retry_format_load(&e) => break Err(e),
                         // Wrap the final error if we are giving up
                         Err(e) if times >= 10 => {
                             break Err(Error::other(format!("store init failed to load formats after {times} retries: {e}")));
@@ -551,7 +557,7 @@ mod tests {
         LOCAL_DECOMMISSION_RESUME_MAX_CONFIG_RETRIES, load_pool_meta_for_startup, pool_first_endpoint_is_local,
         pool_meta_has_active_decommission, preflight_startup_rpc_secret_with, resolve_startup_pool_defaults_with,
         resolve_store_init_stage_result, save_validated_pool_meta_for_startup, should_auto_start_rebalance_after_init,
-        should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
+        should_retry_format_load, should_retry_local_decommission_resume, wait_for_local_decommission_resume_delay,
     };
     #[cfg(feature = "test-util")]
     use crate::{
@@ -773,6 +779,13 @@ mod tests {
         assert!(!should_retry_local_decommission_resume(&StorageError::SlowDown, 0));
     }
 
+    #[test]
+    fn test_should_retry_format_load_rejects_permanent_corruption() {
+        assert!(!should_retry_format_load(&StorageError::CorruptedFormat));
+        assert!(should_retry_format_load(&StorageError::ErasureReadQuorum));
+        assert!(should_retry_format_load(&StorageError::FirstDiskWait));
+    }
+
     #[test]
     fn test_should_auto_start_rebalance_after_init_allows_loaded_rebalance_without_decommission() {
         assert!(should_auto_start_rebalance_after_init(false, true));
@@ -1247,6 +1260,16 @@ mod tests {
 
         let registered: Vec<String> = instance_ctx.local_disk_map().read().await.keys().cloned().collect();
         assert_eq!(registered.len(), 4, "the passed context must register all four local disks");
+        let registered_disk_ids = instance_ctx.local_disk_id_map();
+        let registered_disk_ids = registered_disk_ids.read().await;
+        assert_eq!(registered_disk_ids.len(), 4, "the passed context must publish all four disk IDs");
+        for endpoint in registered_disk_ids.values() {
+            assert!(
+                registered.contains(endpoint),
+                "every disk ID in the passed context must resolve to one of its registered endpoints"
+            );
+        }
+        drop(registered_disk_ids);
         let bootstrap = crate::runtime::instance::bootstrap_ctx();
         assert_ne!(
             bootstrap.deployment_id(),
@@ -1261,6 +1284,15 @@ mod tests {
                 "the bootstrap context must not absorb the fresh store's disks"
             );
         }
+        drop(bootstrap_map);
+        let bootstrap_disk_ids = bootstrap.local_disk_id_map();
+        let bootstrap_disk_ids = bootstrap_disk_ids.read().await;
+        for endpoint in bootstrap_disk_ids.values() {
+            assert!(
+                !registered.contains(endpoint),
+                "the bootstrap context must not absorb the fresh store's disk IDs"
+            );
+        }
     }
 
     #[tokio::test]

crates/ecstore/src/store/mod.rs

@@ -428,11 +428,11 @@ impl ECStore {
 }
 
 lazy_static! {
-    static ref enableObjcetLockConfig: ObjectLockConfiguration = ObjectLockConfiguration {
+    static ref ENABLED_OBJECT_LOCK_CONFIG: ObjectLockConfiguration = ObjectLockConfiguration {
         object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
         ..Default::default()
     };
-    static ref enableVersioningConfig: VersioningConfiguration = VersioningConfiguration {
+    static ref ENABLED_VERSIONING_CONFIG: VersioningConfiguration = VersioningConfiguration {
         status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
         ..Default::default()
     };
@@ -989,7 +989,7 @@ mod tests {
 
         init_local_disks(endpoint_pools.clone()).await.expect("init local disks");
 
-        let (disks, errs) = init_disks(
+        let (mut disks, errs) = init_disks(
             &endpoint_pools.as_ref().first().expect("pool endpoints").endpoints,
             &DiskOption {
                 cleanup: true,
@@ -999,7 +999,7 @@ mod tests {
         .await;
 
         assert!(errs.iter().all(|err| err.is_none()), "disk init should succeed: {errs:?}");
-        connect_load_init_formats(true, &disks, 1, 4, None)
+        connect_load_init_formats(true, &mut disks, 1, 4, None)
             .await
             .expect("initialize format metadata");
 

crates/ecstore/src/store/peer.rs

@@ -28,8 +28,26 @@ async fn remember_local_disk_id(disk: &DiskStore) -> Option<Uuid> {
 
 async fn remember_local_disk_id_with_instance_ctx(instance_ctx: &Arc<InstanceContext>, disk: &DiskStore) -> Option<Uuid> {
     let disk_id = disk.get_disk_id().await.ok().flatten()?;
-    runtime_sources::record_local_disk_id(instance_ctx, disk_id, disk.endpoint().to_string()).await;
-    Some(disk_id)
+    record_local_disk_id_if_active(instance_ctx, disk, disk_id)
+        .await
+        .then_some(disk_id)
+}
+
+async fn record_local_disk_id_if_active(instance_ctx: &Arc<InstanceContext>, disk: &DiskStore, disk_id: Uuid) -> bool {
+    let endpoint = disk.endpoint().to_string();
+    let local_disk_map = instance_ctx.local_disk_map();
+    let local_disks = local_disk_map.read().await;
+    let Some(active_disk) = local_disks.get(&endpoint).and_then(Option::as_ref) else {
+        return false;
+    };
+    if !Arc::ptr_eq(active_disk, disk) {
+        return false;
+    }
+
+    // Lock order is local_disk_map -> local_disk_id_map so quarantine is the
+    // linearization point for rejecting an in-flight stale disk snapshot.
+    instance_ctx.local_disk_id_map().write().await.insert(disk_id, endpoint);
+    true
 }
 
 pub async fn find_local_disk(disk_path: &str) -> Option<DiskStore> {
@@ -228,6 +246,7 @@ pub async fn get_disk_infos(disks: &[Option<DiskStore>]) -> Vec<Option<DiskInfo>
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::disk::new_disk;
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
 
     fn single_local_disk_pools(dir: &std::path::Path) -> EndpointServerPools {
@@ -314,4 +333,56 @@ mod tests {
             );
         }
     }
+
+    #[tokio::test]
+    async fn stale_local_disk_snapshot_cannot_repopulate_the_id_registry() {
+        let temp_dir = tempfile::tempdir().expect("create temp disk dir");
+        let endpoint_pools = single_local_disk_pools(temp_dir.path());
+        let instance_ctx = Arc::new(InstanceContext::new());
+        init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools)
+            .await
+            .expect("local disk should be registered");
+        let disk = instance_ctx
+            .local_disk_map()
+            .read()
+            .await
+            .values()
+            .find_map(|disk| disk.clone())
+            .expect("registered local disk");
+        let endpoint = disk.endpoint().to_string();
+        let disk_id = Uuid::new_v4();
+
+        let local_disk_map = instance_ctx.local_disk_map();
+        let mut quarantine = local_disk_map.write().await;
+        let replacement = new_disk(
+            &disk.endpoint(),
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+        )
+        .await
+        .expect("replacement disk should initialize");
+        assert!(!Arc::ptr_eq(&disk, &replacement));
+        let task_ctx = instance_ctx.clone();
+        let task_disk = disk.clone();
+        let remember = tokio::spawn(async move { record_local_disk_id_if_active(&task_ctx, &task_disk, disk_id).await });
+        tokio::task::yield_now().await;
+        quarantine.insert(endpoint.clone(), Some(replacement.clone()));
+        drop(quarantine);
+
+        assert!(!remember.await.expect("stale lookup task should complete"));
+        assert!(!instance_ctx.local_disk_id_map().read().await.contains_key(&disk_id));
+        let active = instance_ctx
+            .local_disk_map()
+            .read()
+            .await
+            .get(&endpoint)
+            .cloned()
+            .flatten()
+            .expect("replacement disk should remain registered");
+        assert!(Arc::ptr_eq(&active, &replacement));
+        assert!(record_local_disk_id_if_active(&instance_ctx, &replacement, disk_id).await);
+        assert_eq!(instance_ctx.local_disk_id_map().read().await.get(&disk_id), Some(&endpoint));
+    }
 }

crates/ecstore/tests/README.md

@@ -2,7 +2,7 @@
 
 ## MinIO-generated encrypted fixtures
 
-`minio_generated_read_test.rs` validates the `bitrot -> GetObjectReader` path against raw MinIO backend data captured by
+`rustfs/src/storage/minio_generated_read_test.rs` validates the `bitrot -> GetObjectReader` path against raw MinIO backend data captured by
 `.\rustfs\scripts\minio_fixture_lab\lab.py`.
 
 It currently covers multipart fixtures for:
@@ -20,5 +20,5 @@ Example:
 ```powershell
 $env:RUSTFS_MINIO_FIXTURE_ROOT = '.\rustfs\tmp\minio-fixture-lab-local-key'
 $env:RUSTFS_MINIO_STATIC_KMS_KEY_B64 = '<base64-32-byte-local-minio-kms-key>'
-cargo +1.97.1 test -p rustfs-ecstore --features rio-v2 --test minio_generated_read_test -- --ignored
+cargo +1.97.1 test -p rustfs --features rio-v2 storage::minio_generated_read_test --lib -- --ignored
 ```

crates/iam/src/manager.rs

@@ -556,7 +556,7 @@ where
         Ok(now)
     }
 
-    pub async fn list_polices(&self, bucket_name: &str) -> Result<HashMap<String, Policy>> {
+    pub async fn list_policies(&self, bucket_name: &str) -> Result<HashMap<String, Policy>> {
         let mut m = HashMap::new();
 
         self.api.load_policy_docs(&mut m).await?;
@@ -588,6 +588,15 @@ where
         Ok(filtered)
     }
 
+    /// Backward-compatible misspelling retained until the next breaking release.
+    #[deprecated(
+        since = "1.0.0",
+        note = "use list_policies instead; this alias will be removed in the next breaking release"
+    )]
+    pub async fn list_polices(&self, bucket_name: &str) -> Result<HashMap<String, Policy>> {
+        self.list_policies(bucket_name).await
+    }
+
     pub async fn merge_policies(&self, name: &str) -> (String, Policy) {
         let mut policies = Vec::new();
         let mut to_merge = Vec::new();
@@ -2184,7 +2193,7 @@ where
     }
 }
 
-pub fn get_default_policyes() -> HashMap<String, PolicyDoc> {
+pub fn get_default_policies() -> HashMap<String, PolicyDoc> {
     let default_policies = &DEFAULT_POLICIES;
     default_policies
         .iter()
@@ -2201,6 +2210,15 @@ pub fn get_default_policyes() -> HashMap<String, PolicyDoc> {
         .collect()
 }
 
+/// Backward-compatible misspelling retained until the next breaking release.
+#[deprecated(
+    since = "1.0.0",
+    note = "use get_default_policies instead; this alias will be removed in the next breaking release"
+)]
+pub fn get_default_policyes() -> HashMap<String, PolicyDoc> {
+    get_default_policies()
+}
+
 fn set_default_canned_policies(policies: &mut HashMap<String, PolicyDoc>) {
     let default_policies = &DEFAULT_POLICIES;
     for (k, v) in default_policies.iter() {
@@ -2900,7 +2918,7 @@ mod tests {
 
     #[test]
     fn test_get_default_policies() {
-        let policies = get_default_policyes();
+        let policies = get_default_policies();
 
         // Should contain some default policies
         assert!(!policies.is_empty());
@@ -2913,6 +2931,12 @@ mod tests {
         }
     }
 
+    #[test]
+    #[allow(deprecated)]
+    fn deprecated_get_default_policyes_matches_current_api() {
+        assert_eq!(get_default_policyes().len(), get_default_policies().len());
+    }
+
     #[test]
     fn test_get_token_signing_key() {
         // This function returns the global action credential's secret key

crates/iam/src/store/object.rs

@@ -22,7 +22,7 @@ use crate::{
     cache::{Cache, CacheEntity},
     error::{is_err_no_such_policy, is_err_no_such_user},
     keyring,
-    manager::{extract_jwt_claims, extract_jwt_claims_allow_missing_exp, get_default_policyes},
+    manager::{extract_jwt_claims, extract_jwt_claims_allow_missing_exp, get_default_policies},
     root_credentials,
 };
 use futures::future::join_all;
@@ -1127,7 +1127,7 @@ impl Store for ObjectStore {
         let cache_snapshot = cache.snapshot();
         let listed_config_items = self.list_all_iamconfig_items().await?;
 
-        let mut policy_docs_cache = CacheEntity::new(get_default_policyes());
+        let mut policy_docs_cache = CacheEntity::new(get_default_policies());
 
         if let Some(policies_list) = listed_config_items.get(POLICIES_LIST_KEY) {
             // Load in fixed-size chunks so each policy is fetched exactly once.

crates/iam/src/sys.rs

@@ -18,7 +18,7 @@ use crate::error::is_err_no_such_temp_account;
 use crate::error::{Error, Result};
 use crate::federation::OIDC_VIRTUAL_PARENT_CLAIM;
 use crate::manager::extract_jwt_claims;
-use crate::manager::get_default_policyes;
+use crate::manager::get_default_policies;
 use crate::manager::{IamCache, IamSyncMetricsSnapshot};
 use crate::store::GroupInfo;
 use crate::store::MappedPolicy;
@@ -249,7 +249,7 @@ impl<T: Store> IamSys<T> {
     }
 
     pub async fn delete_policy(&self, name: &str, notify: bool) -> Result<()> {
-        for k in get_default_policyes().keys() {
+        for k in get_default_policies().keys() {
             if k == name {
                 return Err(Error::other("system policy can not be deleted"));
             }
@@ -291,8 +291,17 @@ impl<T: Store> IamSys<T> {
         self.store.api.load_mapped_policies(user_type, is_group, m).await
     }
 
+    pub async fn list_policies(&self, bucket_name: &str) -> Result<HashMap<String, Policy>> {
+        self.store.list_policies(bucket_name).await
+    }
+
+    /// Backward-compatible misspelling retained until the next breaking release.
+    #[deprecated(
+        since = "1.0.0",
+        note = "use list_policies instead; this alias will be removed in the next breaking release"
+    )]
     pub async fn list_polices(&self, bucket_name: &str) -> Result<HashMap<String, Policy>> {
-        self.store.list_polices(bucket_name).await
+        self.list_policies(bucket_name).await
     }
 
     pub async fn list_policy_docs(&self, bucket_name: &str) -> Result<HashMap<String, PolicyDoc>> {
@@ -1683,11 +1692,17 @@ mod tests {
     use super::*;
     use crate::cache::{Cache, CacheEntity};
     use crate::error::Error;
-    use crate::manager::get_default_policyes;
+    use crate::manager::get_default_policies;
     use crate::store::{GroupInfo, MappedPolicy, Store, UserType};
     use rustfs_credentials::{Credentials, init_global_action_credentials};
     use rustfs_policy::auth::{UserIdentity, get_new_credentials_with_metadata};
     use rustfs_policy::policy::Args;
+
+    #[test]
+    #[allow(deprecated)]
+    fn deprecated_list_polices_api_is_available() {
+        let _ = IamSys::<StsTestMockStore>::list_polices;
+    }
     use rustfs_policy::policy::action::{Action, AdminAction, S3Action};
     use rustfs_policy::policy::policy_uses_existing_object_tag_conditions;
     use serde_json::Value;
@@ -1925,7 +1940,7 @@ mod tests {
         }
 
         async fn load_all(&self, cache: &Cache) -> Result<()> {
-            let mut policy_docs = get_default_policyes();
+            let mut policy_docs = get_default_policies();
             let custom_claim_policy =
                 Policy::parse_config(CUSTOM_STS_CLAIM_POLICY_JSON.as_bytes()).expect("custom STS claim policy should parse");
             policy_docs.insert(CUSTOM_STS_CLAIM_POLICY.to_string(), PolicyDoc::new(custom_claim_policy));

crates/io-metrics/Cargo.toml

@@ -30,7 +30,9 @@ harness = false
 
 [dependencies]
 metrics = { workspace = true }
+rustfs-common = { workspace = true }
 rustfs-s3-ops = { workspace = true }
+rustfs-utils = { workspace = true, features = ["ip"] }
 num_cpus = { workspace = true }
 thiserror = { workspace = true }
 tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }

crates/io-metrics/src/internode_metrics.rs

@@ -15,7 +15,7 @@
 use metrics::{counter, gauge};
 use std::collections::HashMap;
 use std::sync::{
-    Arc, LazyLock, RwLock,
+    Arc, LazyLock, OnceLock, RwLock,
     atomic::{AtomicU64, Ordering},
 };
 use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
@@ -40,6 +40,7 @@ pub const INTERNODE_MSGPACK_CODEC_JSON: &str = "json";
 
 const OPERATION_LABEL: &str = "operation";
 const BACKEND_LABEL: &str = "backend";
+const SERVER_LABEL: &str = "server";
 const CLASSIFICATION_LABEL: &str = "classification";
 const STAGE_LABEL: &str = "stage";
 const DOMINANT_ERROR_LABEL: &str = "dominant_error";
@@ -77,74 +78,93 @@ pub struct InternodeOperationMetricDescriptor {
     pub labels: &'static [&'static str],
 }
 
-const OPERATION_BACKEND_LABELS: &[&str] = &[OPERATION_LABEL, BACKEND_LABEL];
-const OPERATION_BACKEND_CLASSIFICATION_LABELS: &[&str] = &[OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL];
-const OPERATION_BACKEND_HTTP_VERSION_LABELS: &[&str] = &[OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL];
-const QUORUM_FAILURE_LABELS: &[&str] = &[STAGE_LABEL, DOMINANT_ERROR_LABEL];
+const SERVER_OPERATION_BACKEND_LABELS: &[&str] = &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL];
+const SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS: &[&str] =
+    &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL];
+const SERVER_OPERATION_BACKEND_HTTP_VERSION_LABELS: &[&str] = &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL];
+const SERVER_QUORUM_FAILURE_LABELS: &[&str] = &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL];
 
 pub const INTERNODE_OPERATION_METRICS: &[InternodeOperationMetricDescriptor] = &[
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_SENT_BYTES_TOTAL,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_RECV_BYTES_TOTAL,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_ERRORS_TOTAL,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_DURATION_MS,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL,
-        labels: OPERATION_BACKEND_CLASSIFICATION_LABELS,
+        labels: SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_RETRIES_TOTAL,
-        labels: OPERATION_BACKEND_CLASSIFICATION_LABELS,
+        labels: SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_RETRY_SUCCESSES_TOTAL,
-        labels: OPERATION_BACKEND_CLASSIFICATION_LABELS,
+        labels: SERVER_OPERATION_BACKEND_CLASSIFICATION_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_HTTP_VERSIONS_TOTAL,
-        labels: OPERATION_BACKEND_HTTP_VERSION_LABELS,
+        labels: SERVER_OPERATION_BACKEND_HTTP_VERSION_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_STALL_TIMEOUTS_TOTAL,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_WRITE_SHUTDOWN_ERRORS_TOTAL,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: ERASURE_WRITE_QUORUM_FAILURES_TOTAL,
-        labels: QUORUM_FAILURE_LABELS,
+        labels: SERVER_QUORUM_FAILURE_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_PAYLOAD_BYTES,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
     InternodeOperationMetricDescriptor {
         name: INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL,
-        labels: OPERATION_BACKEND_LABELS,
+        labels: SERVER_OPERATION_BACKEND_LABELS,
     },
 ];
 
+fn current_server_label() -> &'static str {
+    static STABLE_SERVER_LABEL: OnceLock<String> = OnceLock::new();
+    static FALLBACK_SERVER_LABEL: LazyLock<String> = LazyLock::new(rustfs_utils::get_local_ip_with_default);
+
+    if let Some(server) = STABLE_SERVER_LABEL.get() {
+        return server.as_str();
+    }
+
+    if let Some(server) = rustfs_common::try_get_global_local_node_name() {
+        let _ = STABLE_SERVER_LABEL.set(server);
+        if let Some(server) = STABLE_SERVER_LABEL.get() {
+            return server.as_str();
+        }
+    }
+
+    FALLBACK_SERVER_LABEL.as_str()
+}
+
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
 pub struct InternodeMetricsSnapshot {
     pub sent_bytes_total: u64,
@@ -193,7 +213,7 @@ impl InternodeMetrics {
             return;
         }
         self.sent_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_sent_bytes_total").increment(bytes);
+        counter!("rustfs_system_network_internode_sent_bytes_total", SERVER_LABEL => current_server_label()).increment(bytes);
     }
 
     pub fn record_sent_bytes_for_operation(&self, operation: &'static str, bytes: usize) {
@@ -207,7 +227,13 @@ impl InternodeMetrics {
         if bytes == 0 {
             return;
         }
-        counter!(INTERNODE_OPERATION_SENT_BYTES_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(bytes);
+        counter!(
+            INTERNODE_OPERATION_SENT_BYTES_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .increment(bytes);
     }
 
     pub fn record_recv_bytes(&self, bytes: usize) {
@@ -216,7 +242,7 @@ impl InternodeMetrics {
             return;
         }
         self.recv_bytes_total.fetch_add(bytes, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_recv_bytes_total").increment(bytes);
+        counter!("rustfs_system_network_internode_recv_bytes_total", SERVER_LABEL => current_server_label()).increment(bytes);
     }
 
     pub fn record_recv_bytes_for_operation(&self, operation: &'static str, bytes: usize) {
@@ -230,12 +256,18 @@ impl InternodeMetrics {
         if bytes == 0 {
             return;
         }
-        counter!(INTERNODE_OPERATION_RECV_BYTES_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(bytes);
+        counter!(
+            INTERNODE_OPERATION_RECV_BYTES_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .increment(bytes);
     }
 
     pub fn record_outgoing_request(&self) {
         self.outgoing_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_requests_outgoing_total").increment(1);
+        counter!("rustfs_system_network_internode_requests_outgoing_total", SERVER_LABEL => current_server_label()).increment(1);
     }
 
     pub fn record_outgoing_request_for_operation(&self, operation: &'static str) {
@@ -244,13 +276,18 @@ impl InternodeMetrics {
 
     pub fn record_outgoing_request_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
         self.record_outgoing_request();
-        counter!(INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
-            .increment(1);
+        counter!(
+            INTERNODE_OPERATION_REQUESTS_OUTGOING_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .increment(1);
     }
 
     pub fn record_incoming_request(&self) {
         self.incoming_requests_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_requests_incoming_total").increment(1);
+        counter!("rustfs_system_network_internode_requests_incoming_total", SERVER_LABEL => current_server_label()).increment(1);
     }
 
     pub fn record_incoming_request_for_operation(&self, operation: &'static str) {
@@ -259,13 +296,18 @@ impl InternodeMetrics {
 
     pub fn record_incoming_request_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
         self.record_incoming_request();
-        counter!(INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
-            .increment(1);
+        counter!(
+            INTERNODE_OPERATION_REQUESTS_INCOMING_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .increment(1);
     }
 
     pub fn record_error(&self) {
         self.errors_total.fetch_add(1, Ordering::Relaxed);
-        counter!("rustfs_system_network_internode_errors_total").increment(1);
+        counter!("rustfs_system_network_internode_errors_total", SERVER_LABEL => current_server_label()).increment(1);
     }
 
     pub fn record_error_for_operation(&self, operation: &'static str) {
@@ -274,13 +316,24 @@ impl InternodeMetrics {
 
     pub fn record_error_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
         self.record_error();
-        counter!(INTERNODE_OPERATION_ERRORS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(1);
+        counter!(
+            INTERNODE_OPERATION_ERRORS_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .increment(1);
     }
 
     pub fn record_duration_for_operation_and_backend(&self, operation: &'static str, backend: &'static str, duration: Duration) {
         let duration_ms = duration.as_secs_f64() * 1000.0;
-        metrics::histogram!(INTERNODE_OPERATION_DURATION_MS, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
-            .record(duration_ms);
+        metrics::histogram!(
+            INTERNODE_OPERATION_DURATION_MS,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .record(duration_ms);
     }
 
     pub fn record_classified_error_for_operation_and_backend(
@@ -291,6 +344,7 @@ impl InternodeMetrics {
     ) {
         counter!(
             INTERNODE_OPERATION_CLASSIFIED_ERRORS_TOTAL,
+            SERVER_LABEL => current_server_label(),
             OPERATION_LABEL => operation,
             BACKEND_LABEL => backend,
             CLASSIFICATION_LABEL => classification
@@ -306,6 +360,7 @@ impl InternodeMetrics {
     ) {
         counter!(
             INTERNODE_OPERATION_RETRIES_TOTAL,
+            SERVER_LABEL => current_server_label(),
             OPERATION_LABEL => operation,
             BACKEND_LABEL => backend,
             CLASSIFICATION_LABEL => classification
@@ -321,6 +376,7 @@ impl InternodeMetrics {
     ) {
         counter!(
             INTERNODE_OPERATION_RETRY_SUCCESSES_TOTAL,
+            SERVER_LABEL => current_server_label(),
             OPERATION_LABEL => operation,
             BACKEND_LABEL => backend,
             CLASSIFICATION_LABEL => classification
@@ -337,6 +393,7 @@ impl InternodeMetrics {
         self.operation_http_versions_total.fetch_add(1, Ordering::Relaxed);
         counter!(
             INTERNODE_OPERATION_HTTP_VERSIONS_TOTAL,
+            SERVER_LABEL => current_server_label(),
             OPERATION_LABEL => operation,
             BACKEND_LABEL => backend,
             HTTP_VERSION_LABEL => http_version
@@ -346,13 +403,24 @@ impl InternodeMetrics {
 
     pub fn record_stall_timeout_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
         self.operation_stall_timeouts_total.fetch_add(1, Ordering::Relaxed);
-        counter!(INTERNODE_OPERATION_STALL_TIMEOUTS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(1);
+        counter!(
+            INTERNODE_OPERATION_STALL_TIMEOUTS_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .increment(1);
     }
 
     pub fn record_write_shutdown_error_for_operation_and_backend(&self, operation: &'static str, backend: &'static str) {
         self.operation_write_shutdown_errors_total.fetch_add(1, Ordering::Relaxed);
-        counter!(INTERNODE_OPERATION_WRITE_SHUTDOWN_ERRORS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
-            .increment(1);
+        counter!(
+            INTERNODE_OPERATION_WRITE_SHUTDOWN_ERRORS_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .increment(1);
     }
 
     /// Record the payload size (bytes) of a completed internode operation into a histogram
@@ -360,15 +428,26 @@ impl InternodeMetrics {
     /// (`ReadAll`/`ReadMultiple`/`WriteAll`) would benefit from being moved off the shared
     /// control-plane channel (see docs/grpc-optimization P1).
     pub fn record_operation_payload_bytes(&self, operation: &'static str, backend: &'static str, bytes: usize) {
-        metrics::histogram!(INTERNODE_OPERATION_PAYLOAD_BYTES, OPERATION_LABEL => operation, BACKEND_LABEL => backend)
-            .record(bytes as f64);
+        metrics::histogram!(
+            INTERNODE_OPERATION_PAYLOAD_BYTES,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .record(bytes as f64);
     }
 
     /// Increment the large-payload counter for an operation+backend whose payload exceeded the
     /// caller-configured warning threshold. Feeds alerting on large unary RPCs that contend with
     /// latency-sensitive control-plane traffic on the shared connection.
     pub fn record_large_operation_payload(&self, operation: &'static str, backend: &'static str) {
-        counter!(INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL, OPERATION_LABEL => operation, BACKEND_LABEL => backend).increment(1);
+        counter!(
+            INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            OPERATION_LABEL => operation,
+            BACKEND_LABEL => backend
+        )
+        .increment(1);
     }
 
     /// Count a decode that fell back to the JSON compatibility field because the msgpack `_bin`
@@ -377,13 +456,20 @@ impl InternodeMetrics {
     /// dropped (grpc-optimization P2). `direction` is [`INTERNODE_MSGPACK_DIRECTION_REQUEST`] or
     /// [`INTERNODE_MSGPACK_DIRECTION_RESPONSE`]; `message` is the low-cardinality value name.
     pub fn record_msgpack_json_fallback(&self, direction: &'static str, message: &'static str) {
-        counter!(INTERNODE_MSGPACK_JSON_FALLBACK_TOTAL, DIRECTION_LABEL => direction, MESSAGE_LABEL => message).increment(1);
+        counter!(
+            INTERNODE_MSGPACK_JSON_FALLBACK_TOTAL,
+            SERVER_LABEL => current_server_label(),
+            DIRECTION_LABEL => direction,
+            MESSAGE_LABEL => message
+        )
+        .increment(1);
     }
 
     pub fn record_msgpack_json_decode(&self, direction: &'static str, message: &'static str, codec: &'static str) {
         self.msgpack_json_decode_total.fetch_add(1, Ordering::Relaxed);
         counter!(
             INTERNODE_MSGPACK_JSON_DECODE_TOTAL,
+            SERVER_LABEL => current_server_label(),
             DIRECTION_LABEL => direction,
             MESSAGE_LABEL => message,
             CODEC_LABEL => codec
@@ -395,6 +481,7 @@ impl InternodeMetrics {
         self.msgpack_json_decode_error_total.fetch_add(1, Ordering::Relaxed);
         counter!(
             INTERNODE_MSGPACK_JSON_DECODE_ERROR_TOTAL,
+            SERVER_LABEL => current_server_label(),
             DIRECTION_LABEL => direction,
             MESSAGE_LABEL => message,
             CODEC_LABEL => codec
@@ -420,7 +507,7 @@ impl InternodeMetrics {
     /// enabled; after the strict flip the legacy fallback path is closed and the counter stays flat.
     pub fn record_signature_v1_fallback(&self) {
         self.signature_v1_fallback_total.fetch_add(1, Ordering::Relaxed);
-        counter!(INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL).increment(1);
+        counter!(INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL, SERVER_LABEL => current_server_label()).increment(1);
     }
 
     /// Count a mutating internode disk RPC that was accepted without a signature-bound canonical
@@ -431,14 +518,14 @@ impl InternodeMetrics {
     /// mutations are rejected and the counter stays flat.
     pub fn record_body_digest_fallback(&self) {
         self.body_digest_fallback_total.fetch_add(1, Ordering::Relaxed);
-        counter!(INTERNODE_BODY_DIGEST_FALLBACK_TOTAL).increment(1);
+        counter!(INTERNODE_BODY_DIGEST_FALLBACK_TOTAL, SERVER_LABEL => current_server_label()).increment(1);
     }
 
     /// Count an accepted v1/v2 request that does not carry the replay-scoped signature. This is
     /// the convergence signal for `RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT`.
     pub fn record_replay_scope_fallback(&self) {
         self.replay_scope_fallback_total.fetch_add(1, Ordering::Relaxed);
-        counter!(INTERNODE_REPLAY_SCOPE_FALLBACK_TOTAL).increment(1);
+        counter!(INTERNODE_REPLAY_SCOPE_FALLBACK_TOTAL, SERVER_LABEL => current_server_label()).increment(1);
     }
 
     /// Count a body-bound internode RPC rejected because the replay-protection nonce cache was
@@ -447,12 +534,13 @@ impl InternodeMetrics {
     /// mutation rate and writes are being refused — alert on this counter.
     pub fn record_replay_cache_overflow(&self) {
         self.replay_cache_overflow_total.fetch_add(1, Ordering::Relaxed);
-        counter!(INTERNODE_REPLAY_CACHE_OVERFLOW_TOTAL).increment(1);
+        counter!(INTERNODE_REPLAY_CACHE_OVERFLOW_TOTAL, SERVER_LABEL => current_server_label()).increment(1);
     }
 
     pub fn record_erasure_write_quorum_failure(&self, stage: &'static str, dominant_error: &'static str) {
         counter!(
             ERASURE_WRITE_QUORUM_FAILURES_TOTAL,
+            SERVER_LABEL => current_server_label(),
             STAGE_LABEL => stage,
             DOMINANT_ERROR_LABEL => dominant_error
         )
@@ -464,11 +552,12 @@ impl InternodeMetrics {
         self.dial_total_time_nanos.fetch_add(elapsed_nanos, Ordering::Relaxed);
         let samples = self.dial_samples_total.fetch_add(1, Ordering::Relaxed) + 1;
         let total = self.dial_total_time_nanos.load(Ordering::Relaxed);
-        gauge!("rustfs_system_network_internode_dial_avg_time_nanos").set(total as f64 / samples as f64);
+        gauge!("rustfs_system_network_internode_dial_avg_time_nanos", SERVER_LABEL => current_server_label())
+            .set(total as f64 / samples as f64);
 
         if !success {
             self.dial_errors_total.fetch_add(1, Ordering::Relaxed);
-            counter!("rustfs_system_network_internode_dial_errors_total").increment(1);
+            counter!("rustfs_system_network_internode_dial_errors_total", SERVER_LABEL => current_server_label()).increment(1);
         }
 
         let now_ms = SystemTime::now()
@@ -687,6 +776,9 @@ fn cluster_peer_health_keys() -> Vec<String> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use metrics::with_local_recorder;
+    use metrics_util::debugging::DebuggingRecorder;
+    use std::collections::HashSet;
 
     #[test]
     fn snapshot_reports_recorded_values() {
@@ -750,22 +842,22 @@ mod tests {
     fn operation_metric_descriptors_include_backend_and_operation_labels() {
         assert_eq!(INTERNODE_OPERATION_METRICS.len(), 15);
         for metric in &INTERNODE_OPERATION_METRICS[..6] {
-            assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL]);
+            assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
         }
         for metric in &INTERNODE_OPERATION_METRICS[6..9] {
-            assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL]);
+            assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, CLASSIFICATION_LABEL]);
         }
         assert_eq!(
             INTERNODE_OPERATION_METRICS[9].labels,
-            &[OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL]
+            &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL, HTTP_VERSION_LABEL]
         );
         for metric in &INTERNODE_OPERATION_METRICS[10..12] {
-            assert_eq!(metric.labels, &[OPERATION_LABEL, BACKEND_LABEL]);
+            assert_eq!(metric.labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
         }
-        assert_eq!(INTERNODE_OPERATION_METRICS[12].labels, &[STAGE_LABEL, DOMINANT_ERROR_LABEL]);
+        assert_eq!(INTERNODE_OPERATION_METRICS[12].labels, &[SERVER_LABEL, STAGE_LABEL, DOMINANT_ERROR_LABEL]);
         // Payload histogram + large-payload counter carry operation+backend labels.
-        assert_eq!(INTERNODE_OPERATION_METRICS[13].labels, &[OPERATION_LABEL, BACKEND_LABEL]);
-        assert_eq!(INTERNODE_OPERATION_METRICS[14].labels, &[OPERATION_LABEL, BACKEND_LABEL]);
+        assert_eq!(INTERNODE_OPERATION_METRICS[13].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
+        assert_eq!(INTERNODE_OPERATION_METRICS[14].labels, &[SERVER_LABEL, OPERATION_LABEL, BACKEND_LABEL]);
     }
 
     #[test]
@@ -843,6 +935,59 @@ mod tests {
         );
     }
 
+    #[test]
+    fn direct_internode_metrics_emit_stable_server_label() {
+        let recorder = DebuggingRecorder::new();
+        let snapshotter = recorder.snapshotter();
+        let metrics = InternodeMetrics::default();
+
+        with_local_recorder(&recorder, || {
+            metrics.record_sent_bytes_for_operation_and_backend(
+                INTERNODE_OPERATION_READ_FILE_STREAM,
+                INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
+                128,
+            );
+            metrics.record_recv_bytes_for_operation_and_backend(
+                INTERNODE_OPERATION_PUT_FILE_STREAM,
+                INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
+                256,
+            );
+            metrics.record_dial_result(Duration::from_millis(3), false);
+        });
+
+        let observed: Vec<(String, HashSet<String>, Option<String>)> = snapshotter
+            .snapshot()
+            .into_vec()
+            .into_iter()
+            .filter(|(composite, _, _, _)| {
+                matches!(
+                    composite.key().name(),
+                    "rustfs_system_network_internode_sent_bytes_total"
+                        | "rustfs_system_network_internode_recv_bytes_total"
+                        | INTERNODE_OPERATION_SENT_BYTES_TOTAL
+                        | INTERNODE_OPERATION_RECV_BYTES_TOTAL
+                        | "rustfs_system_network_internode_dial_avg_time_nanos"
+                        | "rustfs_system_network_internode_dial_errors_total"
+                )
+            })
+            .map(|(composite, _, _, _)| {
+                let labels = composite.key().labels();
+                let keys = labels.clone().map(|label| label.key().to_string()).collect();
+                let server = labels
+                    .filter(|label| label.key() == SERVER_LABEL)
+                    .map(|label| label.value().to_string())
+                    .next();
+                (composite.key().name().to_string(), keys, server)
+            })
+            .collect();
+
+        assert_eq!(observed.len(), 6);
+        for (name, keys, server) in observed {
+            assert!(keys.contains(SERVER_LABEL), "{name} must carry the server label");
+            assert!(server.is_some_and(|value| !value.is_empty()), "{name} server label must not be empty");
+        }
+    }
+
     #[test]
     fn msgpack_json_fallback_counter_records_without_panicking() {
         // Smoke test: the counter accepts both directions and a static message label.

crates/kms/AGENTS.md

@@ -25,3 +25,25 @@ For local KMS end-to-end tests, keep proxy bypass settings:
 NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= \
 cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1
 ```
+
+## Local Key Export for SSE-S3 Migration Tests
+
+Use the read-only `local_kms_key_decrypt` example to export an AES-256 Local
+KMS key as the base64 value expected by `RUSTFS_SSE_S3_MASTER_KEY`:
+
+```bash
+export RUSTFS_KMS_LOCAL_MASTER_KEY='<local-kms-at-rest-master-key>'
+export RUSTFS_SSE_S3_MASTER_KEY="$(
+  cargo run -q -p rustfs-kms --example local_kms_key_decrypt -- \
+    /absolute/path/to/<key-id>.key
+)"
+```
+
+For a `plaintext-dev-only` Local KMS key file,
+`RUSTFS_KMS_LOCAL_MASTER_KEY` is not required.
+
+The example writes only the base64-encoded 32-byte key to stdout. Diagnostics
+go to stderr. Never paste its output into logs, shell history, issue comments,
+or committed configuration. The export path must remain read-only and must
+reuse `LocalKmsClient` decoding so current Argon2id and legacy key-file
+compatibility stay aligned with the backend.

crates/kms/examples/local_kms_key_decrypt.rs

@@ -0,0 +1,112 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
+use rustfs_kms::{LocalConfig, backends::local::LocalKmsClient};
+use std::io::{self, Write};
+use std::path::{Path, PathBuf};
+use zeroize::Zeroizing;
+
+const LOCAL_KMS_MASTER_KEY_ENV: &str = "RUSTFS_KMS_LOCAL_MASTER_KEY";
+
+fn usage(program: &str) -> String {
+    format!(
+        "Usage: {program} <local-kms-key-file>\n\
+         Reads {LOCAL_KMS_MASTER_KEY_ENV} when the key file is encrypted.\n\
+         Writes only the base64-encoded 32-byte key to stdout."
+    )
+}
+
+fn resolve_key_file(path: &Path) -> Result<(PathBuf, String), String> {
+    let canonical = std::fs::canonicalize(path).map_err(|error| format!("cannot open Local KMS key file: {error}"))?;
+    if canonical.extension().and_then(|extension| extension.to_str()) != Some("key") {
+        return Err("Local KMS key file must have a .key extension".to_string());
+    }
+    let key_dir = canonical
+        .parent()
+        .ok_or_else(|| "Local KMS key file must have a parent directory".to_string())?
+        .to_path_buf();
+    let key_id = canonical
+        .file_stem()
+        .and_then(|stem| stem.to_str())
+        .filter(|stem| !stem.is_empty())
+        .ok_or_else(|| "Local KMS key file name must contain a valid UTF-8 key ID".to_string())?
+        .to_string();
+    Ok((key_dir, key_id))
+}
+
+async fn run() -> Result<(), String> {
+    let mut args = std::env::args();
+    let program = args.next().unwrap_or_else(|| "local_kms_key_decrypt".to_string());
+    let Some(key_file) = args.next() else {
+        return Err(usage(&program));
+    };
+    if args.next().is_some() {
+        return Err(usage(&program));
+    }
+
+    let (key_dir, key_id) = resolve_key_file(Path::new(&key_file))?;
+    let master_key = std::env::var(LOCAL_KMS_MASTER_KEY_ENV).ok().filter(|value| !value.is_empty());
+    let client = LocalKmsClient::new_for_key_export(LocalConfig {
+        key_dir,
+        master_key,
+        file_permissions: Some(0o600),
+    })
+    .await
+    .map_err(|error| error.to_string())?;
+    let key_material = client
+        .decrypt_key_material_for_export(&key_id)
+        .await
+        .map_err(|error| error.to_string())?;
+    let encoded = Zeroizing::new(BASE64_STANDARD.encode(key_material.as_ref()));
+
+    let mut stdout = io::stdout().lock();
+    writeln!(stdout, "{}", encoded.as_str()).map_err(|error| format!("failed to write decrypted key: {error}"))
+}
+
+#[tokio::main]
+async fn main() {
+    if let Err(error) = run().await {
+        let _ = writeln!(io::stderr().lock(), "local_kms_key_decrypt: {error}");
+        std::process::exit(1);
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn resolve_key_file_extracts_directory_and_key_id() {
+        let directory = tempfile::tempdir().expect("create temporary directory");
+        let key_file = directory.path().join("migration-key.key");
+        std::fs::write(&key_file, b"{}").expect("create key file");
+
+        let (key_dir, key_id) = resolve_key_file(&key_file).expect("resolve key file");
+
+        assert_eq!(key_dir, directory.path().canonicalize().expect("canonical directory"));
+        assert_eq!(key_id, "migration-key");
+    }
+
+    #[test]
+    fn resolve_key_file_rejects_non_key_extension() {
+        let directory = tempfile::tempdir().expect("create temporary directory");
+        let key_file = directory.path().join("migration-key.json");
+        std::fs::write(&key_file, b"{}").expect("create key file");
+
+        let error = resolve_key_file(&key_file).expect_err("non-key file must be rejected");
+
+        assert!(error.contains(".key"));
+    }
+}

crates/kms/src/backends/local.rs

@@ -36,6 +36,7 @@ use std::path::{Component, Path, PathBuf};
 use std::time::Duration;
 use tokio::fs;
 use tracing::{debug, warn};
+use zeroize::Zeroizing;
 
 /// Reject key identifiers that would not name a single file directly inside the key
 /// directory.
@@ -146,6 +147,45 @@ impl LocalKmsClient {
         Ok(client)
     }
 
+    /// Open a Local KMS key directory without creating or modifying any files.
+    ///
+    /// This constructor is restricted to explicit key-export tooling. Normal
+    /// backend operation must use [`Self::new`].
+    pub async fn new_for_key_export(config: LocalConfig) -> Result<Self> {
+        if !fs::try_exists(&config.key_dir).await? {
+            return Err(KmsError::configuration_error("Local KMS key directory does not exist"));
+        }
+
+        let (master_cipher, legacy_master_cipher) = if let Some(ref master_key) = config.master_key {
+            let legacy_key = Self::derive_legacy_master_key(master_key)?;
+            let legacy_master_cipher = Aes256Gcm::new(&legacy_key);
+            let salt_path = Self::master_key_salt_path(&config);
+            let master_cipher = if fs::try_exists(&salt_path).await? {
+                let salt = fs::read(&salt_path).await?;
+                let salt: [u8; LOCAL_KMS_MASTER_KEY_SALT_LEN] = salt.try_into().map_err(|_| {
+                    KmsError::configuration_error(format!(
+                        "Local KMS master key salt at {} must be exactly {} bytes",
+                        salt_path.display(),
+                        LOCAL_KMS_MASTER_KEY_SALT_LEN
+                    ))
+                })?;
+                Aes256Gcm::new(&Self::derive_master_key(master_key, &salt)?)
+            } else {
+                Aes256Gcm::new(&legacy_key)
+            };
+            (Some(master_cipher), Some(legacy_master_cipher))
+        } else {
+            (None, None)
+        };
+
+        Ok(Self {
+            config,
+            master_cipher,
+            legacy_master_cipher,
+            dek_crypto: AesDekCrypto::new(),
+        })
+    }
+
     /// Derive a 256-bit key from the master key string using a persistent Argon2id salt.
     fn derive_master_key(master_key: &str, salt: &[u8]) -> Result<Key<Aes256Gcm>> {
         let params = Params::new(
@@ -435,6 +475,20 @@ impl LocalKmsClient {
         Ok(key_material)
     }
 
+    /// Decrypt an AES-256 Local KMS key for explicit migration tooling.
+    ///
+    /// The returned buffer is zeroized on drop. Callers must treat the value as
+    /// plaintext key material and avoid logging or persisting it.
+    pub async fn decrypt_key_material_for_export(&self, key_id: &str) -> Result<Zeroizing<[u8; 32]>> {
+        let (stored_key, key_material) = self.decode_stored_key(key_id).await?;
+        if stored_key.algorithm != "AES_256" {
+            return Err(KmsError::unsupported_algorithm(stored_key.algorithm));
+        }
+        let actual = key_material.len();
+        let key_material = key_material.try_into().map_err(|_| KmsError::invalid_key_size(32, actual))?;
+        Ok(Zeroizing::new(key_material))
+    }
+
     async fn validate_existing_keys(&self) -> Result<()> {
         let mut entries = fs::read_dir(&self.config.key_dir).await?;
         while let Some(entry) = entries.next_entry().await? {
@@ -1208,6 +1262,52 @@ mod tests {
         assert!(matches!(wrong_master_error, KmsError::CryptographicError { .. }));
     }
 
+    #[tokio::test]
+    async fn key_export_uses_existing_local_decryption_path_without_writing_files() {
+        let (client, _temp_dir) = create_test_client().await;
+        let key_id = "export-key";
+        client
+            .create_key(key_id, "AES_256", None)
+            .await
+            .expect("create encrypted key");
+        let expected = client.get_key_material(key_id).await.expect("load expected key material");
+        let salt_path = LocalKmsClient::master_key_salt_path(&client.config);
+        let salt_before = fs::read(&salt_path).await.expect("read existing salt");
+
+        let export_client = LocalKmsClient::new_for_key_export(client.config.clone())
+            .await
+            .expect("open read-only export client");
+        let exported = export_client
+            .decrypt_key_material_for_export(key_id)
+            .await
+            .expect("decrypt key for export");
+
+        assert_eq!(exported.as_ref(), expected.as_slice());
+        assert_eq!(fs::read(&salt_path).await.expect("read unchanged salt"), salt_before);
+    }
+
+    #[tokio::test]
+    async fn key_export_accepts_plaintext_dev_only_key_without_master_key() {
+        let (client, _temp_dir) = create_dev_mode_client().await;
+        let key_id = "plaintext-export-key";
+        client
+            .create_key(key_id, "AES_256", None)
+            .await
+            .expect("create plaintext-dev-only key");
+        let expected = client.get_key_material(key_id).await.expect("load expected key material");
+
+        let export_client = LocalKmsClient::new_for_key_export(client.config.clone())
+            .await
+            .expect("open read-only export client");
+        let exported = export_client
+            .decrypt_key_material_for_export(key_id)
+            .await
+            .expect("export plaintext-dev-only key");
+
+        assert_eq!(exported.as_ref(), expected.as_slice());
+        assert!(!LocalKmsClient::master_key_salt_path(&client.config).exists());
+    }
+
     #[tokio::test]
     async fn test_plaintext_dev_only_storage_is_explicit_and_loadable() {
         let (client, _temp_dir) = create_dev_mode_client().await;

crates/kms/src/lib.rs

@@ -89,7 +89,7 @@ pub use error::{KmsError, KmsUnavailableError, Result};
 pub use manager::KmsManager;
 pub use service::{DataKey, ObjectEncryptionService};
 pub use service_manager::{
-    KmsServiceManager, KmsServiceStatus, get_global_encryption_service, get_global_kms_service_manager,
+    KmsServiceManager, KmsServiceStatus, KmsStartOutcome, get_global_encryption_service, get_global_kms_service_manager,
     init_global_kms_service_manager,
 };
 pub use types::*;

crates/kms/src/service_manager.rs

@@ -21,12 +21,13 @@ use crate::manager::KmsManager;
 use crate::service::ObjectEncryptionService;
 use arc_swap::ArcSwap;
 use sha2::{Digest, Sha256};
+use std::future::Future;
 use std::sync::{
     Arc, OnceLock,
     atomic::{AtomicU64, Ordering},
 };
 use subtle::ConstantTimeEq;
-use tokio::sync::{Mutex, RwLock};
+use tokio::sync::Mutex;
 use tracing::{debug, error, info, warn};
 
 const LOG_COMPONENT_KMS: &str = "kms";
@@ -93,6 +94,13 @@ pub enum KmsServiceStatus {
     Error(String),
 }
 
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum KmsStartOutcome {
+    Started,
+    Restarted,
+    AlreadyRunning,
+}
+
 /// Service version information for zero-downtime reconfiguration
 #[derive(Clone)]
 struct ServiceVersion {
@@ -104,16 +112,17 @@ struct ServiceVersion {
     manager: Arc<KmsManager>,
 }
 
+#[derive(Clone)]
+struct RuntimeState {
+    config: Option<KmsConfig>,
+    status: KmsServiceStatus,
+    current_service: Option<ServiceVersion>,
+}
+
 /// Dynamic KMS service manager with versioned services for zero-downtime reconfiguration
 pub struct KmsServiceManager {
-    /// Current service version (if running)
-    /// Uses ArcSwap for atomic, lock-free service switching
-    /// This allows instant atomic updates without blocking readers
-    current_service: ArcSwap<Option<ServiceVersion>>,
-    /// Current configuration
-    config: Arc<RwLock<Option<KmsConfig>>>,
-    /// Current status
-    status: Arc<RwLock<KmsServiceStatus>>,
+    /// Atomically published configuration, status, and current service.
+    state: ArcSwap<RuntimeState>,
     /// Version counter (monotonically increasing)
     version_counter: Arc<AtomicU64>,
     /// Mutex to protect lifecycle operations (start, stop, reconfigure)
@@ -125,9 +134,11 @@ impl KmsServiceManager {
     /// Create a new KMS service manager (not configured)
     pub fn new() -> Self {
         Self {
-            current_service: ArcSwap::from_pointee(None),
-            config: Arc::new(RwLock::new(None)),
-            status: Arc::new(RwLock::new(KmsServiceStatus::NotConfigured)),
+            state: ArcSwap::from_pointee(RuntimeState {
+                config: None,
+                status: KmsServiceStatus::NotConfigured,
+                current_service: None,
+            }),
             version_counter: Arc::new(AtomicU64::new(0)),
             lifecycle_mutex: Arc::new(Mutex::new(())),
         }
@@ -135,44 +146,67 @@ impl KmsServiceManager {
 
     /// Get current service status
     pub async fn get_status(&self) -> KmsServiceStatus {
-        self.status.read().await.clone()
+        self.state.load().status.clone()
     }
 
     /// Get current configuration (if any)
     pub async fn get_config(&self) -> Option<KmsConfig> {
-        self.config.read().await.clone()
+        self.state.load().config.clone()
     }
 
     /// Get configuration for status and management responses without static key material.
     pub async fn get_redacted_config(&self) -> Option<KmsConfig> {
-        let mut config = self.config.read().await.clone()?;
+        let mut config = self.state.load().config.clone()?;
+        Self::redact_config(&mut config);
+        Some(config)
+    }
+
+    /// Get status and redacted configuration from the same published snapshot.
+    pub async fn get_redacted_state(&self) -> (KmsServiceStatus, Option<KmsConfig>) {
+        let state = self.state.load();
+        let mut config = state.config.clone();
+        if let Some(config) = &mut config {
+            Self::redact_config(config);
+        }
+        (state.status.clone(), config)
+    }
+
+    fn redact_config(config: &mut KmsConfig) {
         if let BackendConfig::Static(static_config) = &mut config.backend_config {
             use zeroize::Zeroize;
             static_config.secret_key.zeroize();
         }
-        Some(config)
     }
 
     /// Configure KMS with new configuration
     pub async fn configure(&self, new_config: KmsConfig) -> Result<()> {
-        let _guard = self.lifecycle_mutex.lock().await;
+        self.configure_with_persistence(new_config, || async { Ok(()) }).await
+    }
+
+    /// Configure KMS and publish the in-memory state only after persistence succeeds.
+    ///
+    /// The persistence callback runs under the lifecycle lock and must not call
+    /// another lifecycle method on this manager.
+    pub async fn configure_with_persistence<Persist, PersistFuture>(&self, new_config: KmsConfig, persist: Persist) -> Result<()>
+    where
+        Persist: FnOnce() -> PersistFuture,
+        PersistFuture: Future<Output = Result<()>>,
+    {
         new_config.validate()?;
-        {
-            let config = self.config.read().await;
-            validate_local_transition(config.as_ref(), &new_config)?;
-        }
-
-        // Update configuration
-        {
-            let mut config = self.config.write().await;
-            *config = Some(new_config.clone());
-        }
-
-        // Update status
-        {
-            let mut status = self.status.write().await;
-            *status = KmsServiceStatus::Configured;
+        let _guard = self.lifecycle_mutex.lock().await;
+        let current = self.state.load_full();
+        validate_local_transition(current.config.as_ref(), &new_config)?;
+        if current.current_service.is_some() {
+            return Err(KmsError::configuration_error(
+                "Cannot configure KMS while it is running; use reconfigure instead",
+            ));
         }
+        persist().await?;
+        self.state.store(Arc::new(RuntimeState {
+            config: Some(new_config),
+            status: KmsServiceStatus::Configured,
+            current_service: None,
+        }));
 
         debug!(
             event = EVENT_KMS_SERVICE_STATE,
@@ -190,19 +224,35 @@ impl KmsServiceManager {
         self.start_internal().await
     }
 
+    /// Start or restart KMS with the running-state decision serialized with the lifecycle action.
+    pub async fn start_or_restart(&self, force: bool) -> Result<KmsStartOutcome> {
+        let _guard = self.lifecycle_mutex.lock().await;
+        let running = self.state.load().current_service.is_some();
+        if running && !force {
+            return Ok(KmsStartOutcome::AlreadyRunning);
+        }
+        self.start_internal().await?;
+        Ok(if running {
+            KmsStartOutcome::Restarted
+        } else {
+            KmsStartOutcome::Started
+        })
+    }
+
     /// Internal start implementation (called within lifecycle mutex)
     async fn start_internal(&self) -> Result<()> {
-        let config = {
-            let config_guard = self.config.read().await;
-            match config_guard.as_ref() {
-                Some(config) => config.clone(),
-                None => {
-                    let err_msg = "Cannot start KMS: no configuration provided";
-                    error!("{}", err_msg);
-                    let mut status = self.status.write().await;
-                    *status = KmsServiceStatus::Error(err_msg.to_string());
-                    return Err(KmsError::configuration_error(err_msg));
-                }
+        let state = self.state.load_full();
+        let config = match state.config.as_ref() {
+            Some(config) => config.clone(),
+            None => {
+                let err_msg = "Cannot start KMS: no configuration provided";
+                error!("{}", err_msg);
+                self.state.store(Arc::new(RuntimeState {
+                    config: None,
+                    status: KmsServiceStatus::Error(err_msg.to_string()),
+                    current_service: None,
+                }));
+                return Err(KmsError::configuration_error(err_msg));
             }
         };
 
@@ -215,17 +265,9 @@ impl KmsServiceManager {
             "KMS service starting"
         );
 
-        match self.create_service_version(&config).await {
+        match self.create_healthy_service_version(&config).await {
             Ok(service_version) => {
-                // Atomically update to new service version (lock-free, instant)
-                // ArcSwap::store() is a true atomic operation using CAS
-                self.current_service.store(Arc::new(Some(service_version)));
-
-                // Update status
-                {
-                    let mut status = self.status.write().await;
-                    *status = KmsServiceStatus::Running;
-                }
+                self.publish_running(config, service_version);
 
                 debug!(
                     event = EVENT_KMS_SERVICE_STATE,
@@ -239,13 +281,24 @@ impl KmsServiceManager {
             Err(e) => {
                 let err_msg = format!("Failed to create KMS backend: {e}");
                 error!("{}", err_msg);
-                let mut status = self.status.write().await;
-                *status = KmsServiceStatus::Error(err_msg.clone());
+                if state.current_service.is_none() {
+                    self.state.store(Arc::new(RuntimeState {
+                        config: state.config.clone(),
+                        status: KmsServiceStatus::Error(err_msg.clone()),
+                        current_service: None,
+                    }));
+                }
                 Err(KmsError::backend_error(&err_msg))
             }
         }
     }
 
+    /// Replace the running service without exposing a stopped interval.
+    pub async fn restart(&self) -> Result<()> {
+        let _guard = self.lifecycle_mutex.lock().await;
+        self.start_internal().await
+    }
+
     /// Stop KMS service
     ///
     /// Note: This stops accepting new operations, but existing operations using
@@ -267,15 +320,16 @@ impl KmsServiceManager {
 
         // Atomically clear current service version (lock-free, instant)
         // Note: Existing Arc references will keep the service alive until operations complete
-        self.current_service.store(Arc::new(None));
-
-        // Update status (keep configuration)
-        {
-            let mut status = self.status.write().await;
-            if !matches!(*status, KmsServiceStatus::NotConfigured) {
-                *status = KmsServiceStatus::Configured;
-            }
-        }
+        let state = self.state.load_full();
+        self.state.store(Arc::new(RuntimeState {
+            config: state.config.clone(),
+            status: if state.config.is_some() {
+                KmsServiceStatus::Configured
+            } else {
+                KmsServiceStatus::NotConfigured
+            },
+            current_service: None,
+        }));
 
         debug!(
             event = EVENT_KMS_SERVICE_STATE,
@@ -298,6 +352,22 @@ impl KmsServiceManager {
     /// This ensures zero downtime during reconfiguration, even for long-running
     /// operations like encrypting large files.
     pub async fn reconfigure(&self, new_config: KmsConfig) -> Result<()> {
+        self.reconfigure_with_persistence(new_config, || async { Ok(()) }).await
+    }
+
+    /// Reconfigure KMS after the candidate is healthy and persistence succeeds.
+    ///
+    /// The persistence callback runs under the lifecycle lock and must not call
+    /// another lifecycle method on this manager.
+    pub async fn reconfigure_with_persistence<Persist, PersistFuture>(
+        &self,
+        new_config: KmsConfig,
+        persist: Persist,
+    ) -> Result<()>
+    where
+        Persist: FnOnce() -> PersistFuture,
+        PersistFuture: Future<Output = Result<()>>,
+    {
         let _guard = self.lifecycle_mutex.lock().await;
 
         debug!(
@@ -308,33 +378,18 @@ impl KmsServiceManager {
             "KMS service reconfiguring"
         );
         new_config.validate()?;
-        {
-            let config = self.config.read().await;
-            validate_local_transition(config.as_ref(), &new_config)?;
-        }
+        validate_local_transition(self.state.load().config.as_ref(), &new_config)?;
 
         // Create new service version without stopping old one
         // This allows existing operations to continue while new operations use new service
-        match self.create_service_version(&new_config).await {
+        match self.create_healthy_service_version(&new_config).await {
             Ok(new_service_version) => {
                 // Get old version for logging (lock-free read)
-                let old_version = self.current_service.load().as_ref().as_ref().map(|sv| sv.version);
+                let old_version = self.state.load().current_service.as_ref().map(|sv| sv.version);
 
-                {
-                    let mut config = self.config.write().await;
-                    *config = Some(new_config);
-                }
+                persist().await?;
 
-                // Atomically switch to new service version (lock-free, instant CAS operation)
-                // This is a true atomic operation - no waiting for locks, instant switch
-                // Old service will be dropped when no more Arc references exist
-                self.current_service.store(Arc::new(Some(new_service_version.clone())));
-
-                // Update status
-                {
-                    let mut status = self.status.write().await;
-                    *status = KmsServiceStatus::Running;
-                }
+                self.publish_running(new_config, new_service_version.clone());
 
                 if let Some(old_ver) = old_version {
                     info!(
@@ -371,7 +426,7 @@ impl KmsServiceManager {
     /// Returns the manager from the current service version.
     /// Uses lock-free atomic load for optimal performance.
     pub async fn get_manager(&self) -> Option<Arc<KmsManager>> {
-        self.current_service.load().as_ref().as_ref().map(|sv| sv.manager.clone())
+        self.state.load().current_service.as_ref().map(|sv| sv.manager.clone())
     }
 
     /// Get encryption service (if running)
@@ -381,7 +436,7 @@ impl KmsServiceManager {
     /// This ensures new operations always use the latest service version,
     /// while existing operations continue using their Arc references.
     pub async fn get_encryption_service(&self) -> Option<Arc<ObjectEncryptionService>> {
-        self.current_service.load().as_ref().as_ref().map(|sv| sv.service.clone())
+        self.state.load().current_service.as_ref().map(|sv| sv.service.clone())
     }
 
     /// Get current service version number
@@ -389,14 +444,16 @@ impl KmsServiceManager {
     /// Useful for monitoring and debugging.
     /// Uses lock-free atomic load.
     pub async fn get_service_version(&self) -> Option<u64> {
-        self.current_service.load().as_ref().as_ref().map(|sv| sv.version)
+        self.state.load().current_service.as_ref().map(|sv| sv.version)
     }
 
     /// Health check for the KMS service
     pub async fn health_check(&self) -> Result<bool> {
-        let manager = self.get_manager().await;
-        match manager {
-            Some(manager) => {
+        let checked_state = self.state.load_full();
+        match checked_state.current_service.as_ref() {
+            Some(service_version) => {
+                let manager = service_version.manager.clone();
+                let checked_version = service_version.version;
                 // Perform health check on the backend
                 match manager.health_check().await {
                     Ok(healthy) => {
@@ -407,9 +464,8 @@ impl KmsServiceManager {
                     }
                     Err(e) => {
                         error!("KMS health check error: {}", e);
-                        // Update status to error
-                        let mut status = self.status.write().await;
-                        *status = KmsServiceStatus::Error(format!("Health check failed: {e}"));
+                        let _guard = self.lifecycle_mutex.lock().await;
+                        self.mark_health_error_if_current(checked_version, &e);
                         Err(e)
                     }
                 }
@@ -468,6 +524,33 @@ impl KmsServiceManager {
             manager: kms_manager,
         })
     }
+
+    async fn create_healthy_service_version(&self, config: &KmsConfig) -> Result<ServiceVersion> {
+        let service_version = self.create_service_version(config).await?;
+        if !service_version.manager.health_check().await? {
+            return Err(KmsError::backend_error("KMS backend health check failed"));
+        }
+        Ok(service_version)
+    }
+
+    fn publish_running(&self, config: KmsConfig, service_version: ServiceVersion) {
+        self.state.store(Arc::new(RuntimeState {
+            config: Some(config),
+            status: KmsServiceStatus::Running,
+            current_service: Some(service_version),
+        }));
+    }
+
+    fn mark_health_error_if_current(&self, checked_version: u64, error: &KmsError) {
+        let current = self.state.load_full();
+        if current.current_service.as_ref().map(|version| version.version) == Some(checked_version) {
+            self.state.store(Arc::new(RuntimeState {
+                config: current.config.clone(),
+                status: KmsServiceStatus::Error(format!("Health check failed: {error}")),
+                current_service: current.current_service.clone(),
+            }));
+        }
+    }
 }
 
 impl Default for KmsServiceManager {
@@ -500,6 +583,11 @@ pub async fn get_global_encryption_service() -> Option<Arc<ObjectEncryptionServi
 #[cfg(test)]
 mod tests {
     use super::*;
+    use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
+
+    fn static_config(key_id: &str, fill: u8) -> KmsConfig {
+        KmsConfig::static_kms(key_id.to_string(), BASE64_STANDARD.encode([fill; 32]))
+    }
 
     #[tokio::test]
     async fn configure_rejects_insecure_development_defaults_before_state_update() {
@@ -517,8 +605,6 @@ mod tests {
 
     #[tokio::test]
     async fn redacted_config_omits_static_key_material() {
-        use base64::Engine as _;
-
         let manager = KmsServiceManager::new();
         let encoded_key = base64::engine::general_purpose::STANDARD.encode([0x5au8; 32]);
         manager
@@ -533,6 +619,136 @@ mod tests {
         assert!(static_config.secret_key.is_empty());
     }
 
+    #[tokio::test]
+    async fn configure_persistence_failure_leaves_state_unchanged() {
+        let manager = KmsServiceManager::new();
+
+        let result = manager
+            .configure_with_persistence(static_config("key-a", 0x11), || async { Err(KmsError::backend_error("persist failed")) })
+            .await;
+
+        assert!(result.is_err());
+        assert_eq!(manager.get_status().await, KmsServiceStatus::NotConfigured);
+        assert!(manager.get_config().await.is_none());
+        assert!(manager.get_encryption_service().await.is_none());
+    }
+
+    #[tokio::test]
+    async fn configure_rejects_running_service_without_changing_snapshot() {
+        let manager = KmsServiceManager::new();
+        manager.configure(static_config("key-a", 0x11)).await.expect("configure");
+        manager.start().await.expect("start");
+        let version = manager.get_service_version().await;
+
+        let result = manager.configure(static_config("key-b", 0x22)).await;
+
+        assert!(result.is_err());
+        assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
+        assert_eq!(manager.get_service_version().await, version);
+        assert_eq!(
+            manager.get_config().await.and_then(|config| config.default_key_id),
+            Some("key-a".to_string())
+        );
+    }
+
+    #[tokio::test]
+    async fn reconfigure_persistence_failure_keeps_old_running_snapshot() {
+        let manager = KmsServiceManager::new();
+        manager.configure(static_config("key-a", 0x11)).await.expect("configure");
+        manager.start().await.expect("start");
+        let old_version = manager.get_service_version().await;
+        let old_service = manager.get_encryption_service().await.expect("old service");
+
+        let result = manager
+            .reconfigure_with_persistence(static_config("key-b", 0x22), || async {
+                Err(KmsError::backend_error("persist failed"))
+            })
+            .await;
+
+        assert!(result.is_err());
+        assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
+        assert_eq!(manager.get_service_version().await, old_version);
+        assert_eq!(
+            manager.get_config().await.and_then(|config| config.default_key_id),
+            Some("key-a".to_string())
+        );
+        assert!(Arc::ptr_eq(
+            &old_service,
+            &manager.get_encryption_service().await.expect("old service remains")
+        ));
+    }
+
+    #[tokio::test]
+    async fn reconfigure_candidate_failure_keeps_old_running_snapshot() {
+        let manager = KmsServiceManager::new();
+        manager.configure(static_config("key-a", 0x11)).await.expect("configure");
+        manager.start().await.expect("start");
+        let old_version = manager.get_service_version().await;
+        let invalid_parent = tempfile::NamedTempFile::new().expect("temporary file");
+        let invalid_config = KmsConfig::local(invalid_parent.path().join("keys")).with_insecure_development_defaults();
+
+        let result = manager.reconfigure(invalid_config).await;
+
+        assert!(result.is_err());
+        assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
+        assert_eq!(manager.get_service_version().await, old_version);
+        assert_eq!(
+            manager.get_config().await.and_then(|config| config.default_key_id),
+            Some("key-a".to_string())
+        );
+    }
+
+    #[tokio::test]
+    async fn restart_never_unpublishes_the_running_service() {
+        let manager = Arc::new(KmsServiceManager::new());
+        manager.configure(static_config("key-a", 0x11)).await.expect("configure");
+        manager.start().await.expect("start");
+        let old_version = manager.get_service_version().await.expect("old version");
+        let restarting = {
+            let manager = manager.clone();
+            tokio::spawn(async move { manager.restart().await })
+        };
+
+        while !restarting.is_finished() {
+            assert!(manager.get_encryption_service().await.is_some());
+            tokio::task::yield_now().await;
+        }
+        restarting.await.expect("restart task").expect("restart");
+
+        assert!(manager.get_encryption_service().await.is_some());
+        assert!(manager.get_service_version().await.expect("new version") > old_version);
+        assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
+    }
+
+    #[tokio::test]
+    async fn start_or_restart_decides_under_the_lifecycle_lock() {
+        let manager = KmsServiceManager::new();
+        manager.configure(static_config("key-a", 0x11)).await.expect("configure");
+
+        assert_eq!(manager.start_or_restart(false).await.expect("initial start"), KmsStartOutcome::Started);
+        let first_version = manager.get_service_version().await.expect("first version");
+        assert_eq!(
+            manager.start_or_restart(false).await.expect("already running"),
+            KmsStartOutcome::AlreadyRunning
+        );
+        assert_eq!(manager.get_service_version().await, Some(first_version));
+        assert_eq!(manager.start_or_restart(true).await.expect("forced restart"), KmsStartOutcome::Restarted);
+        assert!(manager.get_service_version().await.expect("restarted version") > first_version);
+    }
+
+    #[tokio::test]
+    async fn stale_health_failure_cannot_poison_new_service_status() {
+        let manager = KmsServiceManager::new();
+        manager.configure(static_config("key-a", 0x11)).await.expect("configure");
+        manager.start().await.expect("start");
+        let old_version = manager.get_service_version().await.expect("old version");
+        manager.restart().await.expect("restart");
+
+        manager.mark_health_error_if_current(old_version, &KmsError::backend_error("stale failure"));
+
+        assert_eq!(manager.get_status().await, KmsServiceStatus::Running);
+    }
+
     #[tokio::test]
     async fn forbidden_local_master_key_change_preserves_running_config_and_service() {
         use crate::types::{CreateKeyRequest, KeyUsage};

crates/obs/src/lib.rs

@@ -64,6 +64,7 @@ mod error;
 mod global;
 mod logging;
 pub mod metrics;
+mod node_identity;
 mod telemetry;
 
 pub use cleaner::*;

crates/obs/src/metrics/collectors/resource.rs

@@ -22,6 +22,7 @@
 
 use crate::metrics::report::PrometheusMetric;
 use crate::metrics::schema::process_resource::*;
+use crate::node_identity::SERVER_LABEL;
 
 /// Resource statistics for metrics collection.
 ///
@@ -30,6 +31,8 @@ use crate::metrics::schema::process_resource::*;
 /// this struct from their available data sources.
 #[derive(Debug, Clone, Default)]
 pub struct ResourceStats {
+    /// Stable local node identity for labeling node-local process resource metrics
+    pub server: String,
     /// CPU usage as a percentage (can exceed 100% on multi-core systems)
     pub cpu_percent: f64,
     /// Resident memory usage in bytes
@@ -43,10 +46,14 @@ pub struct ResourceStats {
 /// Uses the metric descriptors from `metrics_type::process_resource` module.
 /// Returns a vector of Prometheus metrics for resource statistics.
 pub fn collect_resource_metrics(stats: &ResourceStats) -> Vec<PrometheusMetric> {
+    let server_label = stats.server.as_str();
     vec![
-        PrometheusMetric::from_descriptor(&PROCESS_CPU_PERCENT_MD, stats.cpu_percent),
-        PrometheusMetric::from_descriptor(&PROCESS_MEMORY_BYTES_MD, stats.memory_bytes as f64),
-        PrometheusMetric::from_descriptor(&PROCESS_UPTIME_SECONDS_MD, stats.uptime_seconds as f64),
+        PrometheusMetric::from_descriptor(&PROCESS_CPU_PERCENT_MD, stats.cpu_percent)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&PROCESS_MEMORY_BYTES_MD, stats.memory_bytes as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&PROCESS_UPTIME_SECONDS_MD, stats.uptime_seconds as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
     ]
 }
 
@@ -58,6 +65,7 @@ mod tests {
     #[test]
     fn test_collect_resource_metrics() {
         let stats = ResourceStats {
+            server: "node1:9000".to_string(),
             cpu_percent: 45.5,
             memory_bytes: 1024 * 1024 * 256,
             uptime_seconds: 7200,
@@ -67,6 +75,11 @@ mod tests {
         report_metrics(&metrics);
 
         assert_eq!(metrics.len(), 3);
+        assert!(
+            metrics
+                .iter()
+                .all(|m| m.labels.iter().any(|(k, v)| *k == SERVER_LABEL && v == "node1:9000"))
+        );
 
         // Verify CPU metric
         let cpu_metric_name = PROCESS_CPU_PERCENT_MD.get_full_metric_name();
@@ -97,13 +110,16 @@ mod tests {
 
         for metric in &metrics {
             assert_eq!(metric.value, 0.0);
-            assert!(metric.labels.is_empty());
+            assert_eq!(metric.labels.len(), 1);
+            assert_eq!(metric.labels[0].0, SERVER_LABEL);
+            assert!(metric.labels[0].1.is_empty());
         }
     }
 
     #[test]
     fn test_collect_resource_metrics_high_cpu() {
         let stats = ResourceStats {
+            server: "node1:9000".to_string(),
             cpu_percent: 150.0, // Can exceed 100% on multi-core systems
             memory_bytes: 0,
             uptime_seconds: 0,

crates/obs/src/metrics/collectors/system_cpu.rs

@@ -25,11 +25,14 @@
 use crate::metrics::report::PrometheusMetric;
 use crate::metrics::schema::system_cpu::*;
 use crate::metrics::schema::system_process::{PROCESS_CPU_USAGE_MD, PROCESS_CPU_UTILIZATION_MD};
+use crate::node_identity::SERVER_LABEL;
 use std::borrow::Cow;
 
 /// System CPU statistics.
 #[derive(Debug, Clone, Default)]
 pub struct CpuStats {
+    /// Stable local node identity for labeling node-local CPU metrics
+    pub server: String,
     /// Average CPU idle time (percentage, 0-100)
     pub avg_idle: f64,
     /// CPU load average over 1 minute
@@ -56,11 +59,16 @@ pub struct ProcessCpuStats {
 /// Uses the metric descriptors from `metrics_type::system_cpu` module.
 /// Returns a vector of Prometheus metrics for CPU statistics.
 pub fn collect_cpu_metrics(stats: &CpuStats) -> Vec<PrometheusMetric> {
+    let server_label = stats.server.as_str();
     vec![
-        PrometheusMetric::from_descriptor(&SYS_CPU_AVG_IDLE_MD, stats.avg_idle),
-        PrometheusMetric::from_descriptor(&SYS_CPU_LOAD_MD, stats.load_avg),
-        PrometheusMetric::from_descriptor(&SYS_CPU_LOAD_PERC_MD, stats.load_avg_perc),
-        PrometheusMetric::from_descriptor(&SYS_CPU_USAGE_PERC_MD, stats.usage_perc),
+        PrometheusMetric::from_descriptor(&SYS_CPU_AVG_IDLE_MD, stats.avg_idle)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&SYS_CPU_LOAD_MD, stats.load_avg)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&SYS_CPU_LOAD_PERC_MD, stats.load_avg_perc)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&SYS_CPU_USAGE_PERC_MD, stats.usage_perc)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
     ]
 }
 
@@ -91,10 +99,12 @@ pub fn collect_process_cpu_metrics(
 mod tests {
     use super::*;
     use crate::metrics::report::report_metrics;
+    use crate::metrics::schema::system_process::{PROCESS_EXECUTABLE_NAME_LABEL, PROCESS_PID_LABEL};
 
     #[test]
     fn test_collect_cpu_metrics() {
         let stats = CpuStats {
+            server: "node1:9000".to_string(),
             avg_idle: 75.5,
             load_avg: 1.5,
             load_avg_perc: 37.5,
@@ -108,6 +118,11 @@ mod tests {
 
         // Verify that metric names are properly generated from descriptors
         assert!(metrics.iter().all(|m| m.name.starts_with("rustfs_system_cpu_")));
+        assert!(
+            metrics
+                .iter()
+                .all(|m| m.labels.iter().any(|(k, v)| *k == SERVER_LABEL && v == "node1:9000"))
+        );
     }
 
     #[test]
@@ -118,13 +133,16 @@ mod tests {
         assert_eq!(metrics.len(), 4);
         for metric in &metrics {
             assert_eq!(metric.value, 0.0);
-            assert!(metric.labels.is_empty());
+            assert_eq!(metric.labels.len(), 1);
+            assert_eq!(metric.labels[0].0, SERVER_LABEL);
+            assert!(metric.labels[0].1.is_empty());
         }
     }
 
     #[test]
     fn system_cpu_metrics_export_total_usage_under_honest_name() {
         let stats = CpuStats {
+            server: "node1:9000".to_string(),
             avg_idle: 40.0,
             load_avg: 1.0,
             load_avg_perc: 25.0,
@@ -171,8 +189,9 @@ mod tests {
         };
 
         let labels = vec![
-            ("process_pid", Cow::Borrowed("12345")),
-            ("process_executable_name", Cow::Borrowed("rustfs")),
+            (SERVER_LABEL, Cow::Borrowed("node1:9000")),
+            (PROCESS_PID_LABEL, Cow::Borrowed("12345")),
+            (PROCESS_EXECUTABLE_NAME_LABEL, Cow::Borrowed("rustfs")),
         ];
 
         let metrics = collect_process_cpu_metrics(&stats, Some(&labels));
@@ -180,7 +199,8 @@ mod tests {
 
         // All metrics should have the labels
         for metric in &metrics {
-            assert_eq!(metric.labels.len(), 2);
+            assert_eq!(metric.labels.len(), 3);
+            assert!(metric.labels.iter().any(|(k, v)| *k == SERVER_LABEL && v == "node1:9000"));
         }
     }
 }

crates/obs/src/metrics/collectors/system_drive.rs

@@ -24,7 +24,7 @@
 
 use crate::metrics::report::PrometheusMetric;
 use crate::metrics::schema::system_drive::*;
-use crate::metrics::schema::system_process::PROCESS_DISK_IO_MD;
+use crate::metrics::schema::system_process::{DIRECTION_LABEL, PROCESS_DISK_IO_MD};
 use std::borrow::Cow;
 
 /// Detailed drive statistics for a single drive.
@@ -223,8 +223,8 @@ pub fn collect_process_disk_metrics(
     let mut read_metric = PrometheusMetric::from_descriptor(&PROCESS_DISK_IO_MD, stats.read_bytes as f64);
     let mut write_metric = PrometheusMetric::from_descriptor(&PROCESS_DISK_IO_MD, stats.written_bytes as f64);
 
-    read_metric.labels.push(("direction", Cow::Borrowed("read")));
-    write_metric.labels.push(("direction", Cow::Borrowed("write")));
+    read_metric.labels.push((DIRECTION_LABEL, Cow::Borrowed("read")));
+    write_metric.labels.push((DIRECTION_LABEL, Cow::Borrowed("write")));
 
     if let Some(l) = labels {
         read_metric.labels.extend(l.iter().map(|(k, v)| (*k, v.clone())));
@@ -238,6 +238,7 @@ pub fn collect_process_disk_metrics(
 mod tests {
     use super::*;
     use crate::metrics::report::report_metrics;
+    use crate::metrics::schema::system_process::{PROCESS_EXECUTABLE_NAME_LABEL, PROCESS_PID_LABEL};
     use std::collections::BTreeSet;
 
     fn assert_metric_label_keys(
@@ -371,4 +372,32 @@ mod tests {
         assert!(offline.is_some());
         assert_eq!(offline.map(|m| m.value), Some(2.0));
     }
+
+    #[test]
+    fn test_collect_process_disk_metrics_with_node_and_process_labels() {
+        let stats = ProcessDiskStats {
+            read_bytes: 1024,
+            written_bytes: 2048,
+        };
+        let labels = vec![
+            (SERVER_LABEL, Cow::Borrowed("node1:9000")),
+            (PROCESS_PID_LABEL, Cow::Borrowed("12345")),
+            (PROCESS_EXECUTABLE_NAME_LABEL, Cow::Borrowed("rustfs")),
+        ];
+
+        let metrics = collect_process_disk_metrics(&stats, Some(&labels));
+
+        assert_eq!(metrics.len(), 2);
+        assert_metric_label_keys(
+            &metrics,
+            &PROCESS_DISK_IO_MD,
+            1024.0,
+            &[
+                DIRECTION_LABEL,
+                SERVER_LABEL,
+                PROCESS_PID_LABEL,
+                PROCESS_EXECUTABLE_NAME_LABEL,
+            ],
+        );
+    }
 }

crates/obs/src/metrics/collectors/system_memory.rs

@@ -25,11 +25,14 @@
 use crate::metrics::report::PrometheusMetric;
 use crate::metrics::schema::system_memory::*;
 use crate::metrics::schema::system_process::{PROCESS_RESIDENT_MEMORY_BYTES_MD, PROCESS_VIRTUAL_MEMORY_BYTES_MD};
+use crate::node_identity::SERVER_LABEL;
 use std::borrow::Cow;
 
 /// System memory statistics.
 #[derive(Debug, Clone, Default)]
 pub struct MemoryStats {
+    /// Stable local node identity for labeling node-local memory metrics
+    pub server: String,
     /// Total memory in bytes
     pub total: u64,
     /// Used memory in bytes
@@ -64,15 +67,24 @@ pub struct ProcessMemoryStats {
 /// Uses the metric descriptors from `metrics_type::system_memory` module.
 /// Returns a vector of Prometheus metrics for memory statistics.
 pub fn collect_memory_metrics(stats: &MemoryStats) -> Vec<PrometheusMetric> {
+    let server_label = stats.server.as_str();
     vec![
-        PrometheusMetric::from_descriptor(&MEM_TOTAL_MD, stats.total as f64),
-        PrometheusMetric::from_descriptor(&MEM_USED_MD, stats.used as f64),
-        PrometheusMetric::from_descriptor(&MEM_USED_PERC_MD, stats.used_perc),
-        PrometheusMetric::from_descriptor(&MEM_FREE_MD, stats.free as f64),
-        PrometheusMetric::from_descriptor(&MEM_BUFFERS_MD, stats.buffers as f64),
-        PrometheusMetric::from_descriptor(&MEM_CACHE_MD, stats.cache as f64),
-        PrometheusMetric::from_descriptor(&MEM_SHARED_MD, stats.shared as f64),
-        PrometheusMetric::from_descriptor(&MEM_AVAILABLE_MD, stats.available as f64),
+        PrometheusMetric::from_descriptor(&MEM_TOTAL_MD, stats.total as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&MEM_USED_MD, stats.used as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&MEM_USED_PERC_MD, stats.used_perc)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&MEM_FREE_MD, stats.free as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&MEM_BUFFERS_MD, stats.buffers as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&MEM_CACHE_MD, stats.cache as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&MEM_SHARED_MD, stats.shared as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&MEM_AVAILABLE_MD, stats.available as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
     ]
 }
 
@@ -104,10 +116,12 @@ pub fn collect_process_memory_metrics(
 mod tests {
     use super::*;
     use crate::metrics::report::report_metrics;
+    use crate::metrics::schema::system_process::{PROCESS_EXECUTABLE_NAME_LABEL, PROCESS_PID_LABEL};
 
     #[test]
     fn test_collect_memory_metrics() {
         let stats = MemoryStats {
+            server: "node1:9000".to_string(),
             total: 16 * 1024 * 1024 * 1024, // 16 GB
             used: 8 * 1024 * 1024 * 1024,   // 8 GB
             used_perc: 50.0,
@@ -123,6 +137,11 @@ mod tests {
 
         assert_eq!(metrics.len(), 8);
         assert!(metrics.iter().all(|m| m.name.starts_with("rustfs_system_memory_")));
+        assert!(
+            metrics
+                .iter()
+                .all(|m| m.labels.iter().any(|(k, v)| *k == SERVER_LABEL && v == "node1:9000"))
+        );
     }
 
     #[test]
@@ -133,7 +152,9 @@ mod tests {
         assert_eq!(metrics.len(), 8);
         for metric in &metrics {
             assert_eq!(metric.value, 0.0);
-            assert!(metric.labels.is_empty());
+            assert_eq!(metric.labels.len(), 1);
+            assert_eq!(metric.labels[0].0, SERVER_LABEL);
+            assert!(metric.labels[0].1.is_empty());
         }
     }
 
@@ -157,13 +178,18 @@ mod tests {
             virtual_mem: 1024 * 1024 * 1024,
         };
 
-        let labels = vec![("process_pid", Cow::Borrowed("12345"))];
+        let labels = vec![
+            (SERVER_LABEL, Cow::Borrowed("node1:9000")),
+            (PROCESS_PID_LABEL, Cow::Borrowed("12345")),
+            (PROCESS_EXECUTABLE_NAME_LABEL, Cow::Borrowed("rustfs")),
+        ];
 
         let metrics = collect_process_memory_metrics(&stats, Some(&labels));
         assert_eq!(metrics.len(), 2);
 
         for metric in &metrics {
-            assert_eq!(metric.labels.len(), 1);
+            assert_eq!(metric.labels.len(), 3);
+            assert!(metric.labels.iter().any(|(k, v)| *k == SERVER_LABEL && v == "node1:9000"));
         }
     }
 }

crates/obs/src/metrics/collectors/system_network.rs

@@ -23,10 +23,13 @@
 
 use crate::metrics::report::PrometheusMetric;
 use crate::metrics::schema::system_network::*;
+use crate::node_identity::SERVER_LABEL;
 
 /// Network statistics for internode communication.
 #[derive(Debug, Clone, Default)]
 pub struct NetworkStats {
+    /// Stable local node identity for labeling node-local internode metrics
+    pub server: String,
     /// Total number of failed internode calls
     pub internode_errors_total: u64,
     /// Total number of TCP dial timeouts and errors
@@ -44,12 +47,18 @@ pub struct NetworkStats {
 /// Uses the metric descriptors from `metrics_type::system_network` module.
 /// Returns a vector of Prometheus metrics for network statistics.
 pub fn collect_network_metrics(stats: &NetworkStats) -> Vec<PrometheusMetric> {
+    let server_label = stats.server.as_str();
     vec![
-        PrometheusMetric::from_descriptor(&INTERNODE_ERRORS_TOTAL_MD, stats.internode_errors_total as f64),
-        PrometheusMetric::from_descriptor(&INTERNODE_DIAL_ERRORS_TOTAL_MD, stats.internode_dial_errors_total as f64),
-        PrometheusMetric::from_descriptor(&INTERNODE_DIAL_AVG_TIME_NANOS_MD, stats.internode_dial_avg_time_nanos as f64),
-        PrometheusMetric::from_descriptor(&INTERNODE_SENT_BYTES_TOTAL_MD, stats.internode_sent_bytes_total as f64),
-        PrometheusMetric::from_descriptor(&INTERNODE_RECV_BYTES_TOTAL_MD, stats.internode_recv_bytes_total as f64),
+        PrometheusMetric::from_descriptor(&INTERNODE_ERRORS_TOTAL_MD, stats.internode_errors_total as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&INTERNODE_DIAL_ERRORS_TOTAL_MD, stats.internode_dial_errors_total as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&INTERNODE_DIAL_AVG_TIME_NANOS_MD, stats.internode_dial_avg_time_nanos as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&INTERNODE_SENT_BYTES_TOTAL_MD, stats.internode_sent_bytes_total as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
+        PrometheusMetric::from_descriptor(&INTERNODE_RECV_BYTES_TOTAL_MD, stats.internode_recv_bytes_total as f64)
+            .with_label_owned(SERVER_LABEL, server_label.to_string()),
     ]
 }
 
@@ -61,6 +70,7 @@ mod tests {
     #[test]
     fn test_collect_network_metrics() {
         let stats = NetworkStats {
+            server: "node1:9000".to_string(),
             internode_errors_total: 10,
             internode_dial_errors_total: 5,
             internode_dial_avg_time_nanos: 1_500_000,      // 1.5ms
@@ -73,6 +83,11 @@ mod tests {
 
         assert_eq!(metrics.len(), 5);
         assert!(metrics.iter().all(|m| m.name.contains("internode")));
+        assert!(
+            metrics
+                .iter()
+                .all(|m| m.labels.iter().any(|(k, v)| *k == SERVER_LABEL && v == "node1:9000"))
+        );
     }
 
     #[test]
@@ -83,7 +98,9 @@ mod tests {
         assert_eq!(metrics.len(), 5);
         for metric in &metrics {
             assert_eq!(metric.value, 0.0);
-            assert!(metric.labels.is_empty());
+            assert_eq!(metric.labels.len(), 1);
+            assert_eq!(metric.labels[0].0, SERVER_LABEL);
+            assert!(metric.labels[0].1.is_empty());
         }
     }
 }

crates/obs/src/metrics/collectors/system_network_host.rs

@@ -18,6 +18,7 @@ use crate::metrics::report::PrometheusMetric;
 use crate::metrics::schema::system_network_host::{
     DIRECTION_LABEL, HOST_NETWORK_IO_MD, HOST_NETWORK_IO_PER_INTERFACE_MD, INTERFACE_LABEL,
 };
+use crate::node_identity::SERVER_LABEL;
 use std::borrow::Cow;
 
 /// Network I/O statistics.
@@ -25,6 +26,8 @@ use std::borrow::Cow;
 /// Contains host-wide network I/O totals and per-interface counters.
 #[derive(Debug, Clone, Default)]
 pub struct HostNetworkStats {
+    /// Stable local node identity for labeling host-wide network metrics.
+    pub server: String,
     /// Total bytes received across observed host interfaces.
     pub total_received: u64,
     /// Total bytes transmitted across observed host interfaces.
@@ -47,7 +50,11 @@ pub fn collect_host_network_metrics(
     let mut received_metric = PrometheusMetric::from_descriptor(&HOST_NETWORK_IO_MD, stats.total_received as f64);
     let mut transmitted_metric = PrometheusMetric::from_descriptor(&HOST_NETWORK_IO_MD, stats.total_transmitted as f64);
 
+    received_metric.labels.push((SERVER_LABEL, Cow::Owned(stats.server.clone())));
     received_metric.labels.push((DIRECTION_LABEL, Cow::Borrowed("received")));
+    transmitted_metric
+        .labels
+        .push((SERVER_LABEL, Cow::Owned(stats.server.clone())));
     transmitted_metric
         .labels
         .push((DIRECTION_LABEL, Cow::Borrowed("transmitted")));
@@ -64,9 +71,13 @@ pub fn collect_host_network_metrics(
         let mut iface_received = PrometheusMetric::from_descriptor(&HOST_NETWORK_IO_PER_INTERFACE_MD, *received as f64);
         let mut iface_transmitted = PrometheusMetric::from_descriptor(&HOST_NETWORK_IO_PER_INTERFACE_MD, *transmitted as f64);
 
+        iface_received.labels.push((SERVER_LABEL, Cow::Owned(stats.server.clone())));
         iface_received.labels.push((INTERFACE_LABEL, Cow::Owned(interface.clone())));
         iface_received.labels.push((DIRECTION_LABEL, Cow::Borrowed("received")));
 
+        iface_transmitted
+            .labels
+            .push((SERVER_LABEL, Cow::Owned(stats.server.clone())));
         iface_transmitted
             .labels
             .push((INTERFACE_LABEL, Cow::Owned(interface.clone())));
@@ -92,6 +103,7 @@ mod tests {
     #[test]
     fn host_network_metrics_use_dedicated_network_host_prefix() {
         let stats = HostNetworkStats {
+            server: "node1:9000".to_string(),
             total_received: 1024,
             total_transmitted: 2048,
             per_interface: vec![("eth0".to_string(), 512, 256)],
@@ -107,9 +119,9 @@ mod tests {
         );
 
         let total_keys: BTreeSet<&str> = metrics[0].labels.iter().map(|(key, _)| *key).collect();
-        assert_eq!(total_keys, BTreeSet::from([DIRECTION_LABEL]));
+        assert_eq!(total_keys, BTreeSet::from([SERVER_LABEL, DIRECTION_LABEL]));
 
         let per_interface_keys: BTreeSet<&str> = metrics[2].labels.iter().map(|(key, _)| *key).collect();
-        assert_eq!(per_interface_keys, BTreeSet::from([DIRECTION_LABEL, INTERFACE_LABEL]));
+        assert_eq!(per_interface_keys, BTreeSet::from([SERVER_LABEL, DIRECTION_LABEL, INTERFACE_LABEL]));
     }
 }

crates/obs/src/metrics/collectors/system_process.rs

@@ -24,6 +24,7 @@
 
 use crate::metrics::report::PrometheusMetric;
 use crate::metrics::schema::system_process::*;
+use crate::node_identity::SERVER_LABEL;
 use std::borrow::Cow;
 use sysinfo::{Pid, ProcessStatus, System};
 
@@ -146,6 +147,8 @@ impl From<ProcessStatus> for ProcessStatusType {
 /// Process statistics for the RustFS server process.
 #[derive(Debug, Clone, Default)]
 pub struct ProcessStats {
+    /// Stable local node identity for labeling node-local process metrics
+    pub server: String,
     /// Total read locks held
     pub locks_read_total: u64,
     /// Total write locks held
@@ -190,6 +193,7 @@ pub struct ProcessStats {
 ///
 /// Returns a vector of Prometheus metrics for process statistics.
 pub fn collect_process_metrics(stats: &ProcessStats) -> Vec<PrometheusMetric> {
+    let server_label = stats.server.as_str();
     let mut metrics = vec![
         PrometheusMetric::from_descriptor(&PROCESS_LOCKS_READ_TOTAL_MD, stats.locks_read_total as f64),
         PrometheusMetric::from_descriptor(&PROCESS_LOCKS_WRITE_TOTAL_MD, stats.locks_write_total as f64),
@@ -209,12 +213,18 @@ pub fn collect_process_metrics(stats: &ProcessStats) -> Vec<PrometheusMetric> {
         PrometheusMetric::from_descriptor(&PROCESS_VIRTUAL_MEMORY_BYTES_MD, stats.virtual_memory_bytes as f64),
         PrometheusMetric::from_descriptor(&PROCESS_VIRTUAL_MEMORY_MAX_BYTES_MD, stats.virtual_memory_max_bytes as f64),
     ];
+    for metric in &mut metrics {
+        metric.labels.push((SERVER_LABEL, Cow::Owned(server_label.to_string())));
+    }
 
     // Add process status metric
     let mut status_metric = PrometheusMetric::from_descriptor(&PROCESS_STATUS_MD, stats.status_value as f64);
     status_metric
         .labels
-        .push(("status", Cow::Owned(format!("{:?}", stats.status))));
+        .push((SERVER_LABEL, Cow::Owned(server_label.to_string())));
+    status_metric
+        .labels
+        .push((STATUS_LABEL, Cow::Owned(format!("{:?}", stats.status))));
     metrics.push(status_metric);
 
     metrics
@@ -235,6 +245,7 @@ mod tests {
     #[test]
     fn test_collect_process_metrics() {
         let stats = ProcessStats {
+            server: "node1:9000".to_string(),
             locks_read_total: 100,
             locks_write_total: 50,
             cpu_total_seconds: 1234.56,
@@ -261,6 +272,11 @@ mod tests {
 
         // 17 original metrics + 1 status metric = 18
         assert_eq!(metrics.len(), 18);
+        assert!(
+            metrics
+                .iter()
+                .all(|m| m.labels.iter().any(|(k, v)| *k == SERVER_LABEL && v == "node1:9000"))
+        );
 
         // Verify uptime
         let uptime_name = PROCESS_UPTIME_SECONDS_MD.get_full_metric_name();
@@ -288,6 +304,7 @@ mod tests {
 
         // 17 original metrics + 1 status metric = 18
         assert_eq!(metrics.len(), 18);
+        assert!(metrics.iter().all(|m| m.labels.iter().any(|(k, _)| *k == SERVER_LABEL)));
     }
 
     #[test]
@@ -296,7 +313,7 @@ mod tests {
         let result = collect_process_attributes();
         assert!(result.is_ok());
 
-        let attrs = result.unwrap();
+        let attrs = result.expect("current process attributes should be collectable");
         assert!(attrs.pid > 0);
         assert!(!attrs.executable_name.is_empty());
     }
@@ -319,7 +336,7 @@ mod tests {
 
         let labels = attrs.to_labels();
         assert_eq!(labels.len(), 4);
-        assert_eq!(labels[0].0, "process_pid");
+        assert_eq!(labels[0].0, PROCESS_PID_LABEL);
         assert_eq!(labels[0].1, "12345");
     }
 }

crates/obs/src/metrics/scheduler.rs

@@ -92,6 +92,7 @@ use crate::metrics::schema::notification_target::{
     NOTIFICATION_TARGET_FAILED_MESSAGES_MD, NOTIFICATION_TARGET_QUEUE_LENGTH_MD, NOTIFICATION_TARGET_TOTAL_MESSAGES_MD,
     TARGET_ID as NOTIFICATION_TARGET_ID_LABEL, TARGET_TYPE as NOTIFICATION_TARGET_TYPE_LABEL,
 };
+use crate::metrics::schema::system_process::{PROCESS_EXECUTABLE_NAME_LABEL, PROCESS_PID_LABEL};
 use crate::metrics::stats_collector::{
     ProcessMetricBundle, collect_bucket_replication_bandwidth_stats, collect_bucket_replication_detail_stats,
     collect_bucket_stats, collect_cluster_and_health_stats, collect_cluster_config_stats, collect_cluster_usage_metric_stats,
@@ -100,6 +101,7 @@ use crate::metrics::stats_collector::{
     collect_process_metric_bundle_with, collect_replication_stats, collect_scanner_metric_stats,
     collect_system_cpu_and_memory_stats_with,
 };
+use crate::node_identity::{SERVER_LABEL, current_local_node_identity};
 use crate::telemetry::retire_metric_series;
 use futures_util::FutureExt;
 use rustfs_audit::audit_target_metrics;
@@ -1509,7 +1511,7 @@ pub fn init_metrics_runtime(token: CancellationToken) {
 
     let token_clone = token.clone();
     tokio::spawn(async move {
-        let labels = current_process_metric_labels();
+        let process_attribute_labels = current_process_attribute_labels();
         let mut host_system = System::new_all();
         let mut host_networks = Networks::new();
         let mut process_sampler = ProcessSampler::new();
@@ -1560,6 +1562,7 @@ pub fn init_metrics_runtime(token: CancellationToken) {
                         }
 
                         if now >= next_system_run {
+                            let labels = current_process_metric_labels(&process_attribute_labels);
                             #[cfg(feature = "gpu")]
                             let mut metrics =
                                 collect_system_monitoring_metrics(&bundle, &labels, &mut host_system, &mut host_networks);
@@ -1686,24 +1689,33 @@ fn advance_deadline(deadline: &mut Instant, interval: Duration, now: Instant) {
     }
 }
 
-fn current_process_metric_labels() -> Vec<(&'static str, Cow<'static, str>)> {
+fn current_process_attribute_labels() -> Vec<(&'static str, Cow<'static, str>)> {
     match collect_process_attributes() {
         Ok(attrs) => vec![
-            ("process_pid", Cow::Owned(attrs.pid.to_string())),
-            ("process_executable_name", Cow::Owned(attrs.executable_name)),
+            (PROCESS_PID_LABEL, Cow::Owned(attrs.pid.to_string())),
+            (PROCESS_EXECUTABLE_NAME_LABEL, Cow::Owned(attrs.executable_name)),
         ],
-        Err(err) => fallback_process_metric_labels(err),
+        Err(err) => fallback_process_attribute_labels(err),
     }
 }
 
-fn fallback_process_metric_labels(err: ProcessAttributeError) -> Vec<(&'static str, Cow<'static, str>)> {
+fn fallback_process_attribute_labels(err: ProcessAttributeError) -> Vec<(&'static str, Cow<'static, str>)> {
     warn!(event = EVENT_METRICS_RUNTIME_STATE, component = LOG_COMPONENT_OBS, subsystem = LOG_SUBSYSTEM_METRICS_RUNTIME, collector = "process_metric_labels", result = "collect_failed", error = %err, "metrics runtime state changed");
     vec![
-        ("process_pid", Cow::Owned(std::process::id().to_string())),
-        ("process_executable_name", Cow::Borrowed("unknown")),
+        (PROCESS_PID_LABEL, Cow::Owned(std::process::id().to_string())),
+        (PROCESS_EXECUTABLE_NAME_LABEL, Cow::Borrowed("unknown")),
     ]
 }
 
+fn current_process_metric_labels(
+    process_attribute_labels: &[(&'static str, Cow<'static, str>)],
+) -> Vec<(&'static str, Cow<'static, str>)> {
+    let mut labels = Vec::with_capacity(process_attribute_labels.len() + 1);
+    labels.push((SERVER_LABEL, Cow::Owned(current_local_node_identity())));
+    labels.extend(process_attribute_labels.iter().map(|(key, value)| (*key, value.clone())));
+    labels
+}
+
 fn collect_system_monitoring_metrics(
     bundle: &ProcessMetricBundle,
     labels: &[(&'static str, Cow<'static, str>)],

crates/obs/src/metrics/schema/process_resource.rs

@@ -14,6 +14,7 @@
 
 #![allow(dead_code)]
 
+use crate::node_identity::SERVER_LABEL;
 use crate::{MetricDescriptor, MetricName, MetricSubsystem, new_gauge_md};
 use std::sync::LazyLock;
 
@@ -22,7 +23,7 @@ pub static PROCESS_CPU_PERCENT_MD: LazyLock<MetricDescriptor> = LazyLock::new(||
     new_gauge_md(
         MetricName::Custom("cpu_percent".to_string()),
         "CPU usage of the RustFS process as a percentage",
-        &[],
+        &[SERVER_LABEL],
         MetricSubsystem::new("/process"),
     )
 });
@@ -32,7 +33,7 @@ pub static PROCESS_MEMORY_BYTES_MD: LazyLock<MetricDescriptor> = LazyLock::new(|
     new_gauge_md(
         MetricName::Custom("memory_bytes".to_string()),
         "Resident memory usage of the RustFS process in bytes",
-        &[],
+        &[SERVER_LABEL],
         MetricSubsystem::new("/process"),
     )
 });
@@ -42,7 +43,7 @@ pub static PROCESS_UPTIME_SECONDS_MD: LazyLock<MetricDescriptor> = LazyLock::new
     new_gauge_md(
         MetricName::Custom("uptime_seconds".to_string()),
         "Uptime of the RustFS process in seconds",
-        &[],
+        &[SERVER_LABEL],
         MetricSubsystem::new("/process"),
     )
 });

crates/obs/src/metrics/schema/system_cpu.rs

@@ -14,24 +14,37 @@
 
 #![allow(dead_code)]
 
+use crate::node_identity::SERVER_LABEL;
 use crate::{MetricDescriptor, MetricName, new_gauge_md, subsystems};
 /// CPU system-related metric descriptors
 use std::sync::LazyLock;
 
-pub static SYS_CPU_AVG_IDLE_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::SysCPUAvgIdle, "Average CPU idle time", &[], subsystems::SYSTEM_CPU));
+pub static SYS_CPU_AVG_IDLE_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
+    new_gauge_md(
+        MetricName::SysCPUAvgIdle,
+        "Average CPU idle time",
+        &[SERVER_LABEL],
+        subsystems::SYSTEM_CPU,
+    )
+});
 
-pub static SYS_CPU_AVG_IOWAIT_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::SysCPUAvgIOWait, "Average CPU IOWait time", &[], subsystems::SYSTEM_CPU));
+pub static SYS_CPU_AVG_IOWAIT_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
+    new_gauge_md(
+        MetricName::SysCPUAvgIOWait,
+        "Average CPU IOWait time",
+        &[SERVER_LABEL],
+        subsystems::SYSTEM_CPU,
+    )
+});
 
 pub static SYS_CPU_LOAD_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::SysCPULoad, "CPU load average 1min", &[], subsystems::SYSTEM_CPU));
+    LazyLock::new(|| new_gauge_md(MetricName::SysCPULoad, "CPU load average 1min", &[SERVER_LABEL], subsystems::SYSTEM_CPU));
 
 pub static SYS_CPU_LOAD_PERC_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
     new_gauge_md(
         MetricName::SysCPULoadPerc,
         "CPU load average 1min (percentage)",
-        &[],
+        &[SERVER_LABEL],
         subsystems::SYSTEM_CPU,
     )
 });
@@ -40,19 +53,19 @@ pub static SYS_CPU_USAGE_PERC_MD: LazyLock<MetricDescriptor> = LazyLock::new(||
     new_gauge_md(
         MetricName::Custom("usage_perc".to_string()),
         "Total CPU usage percentage across all measured CPU time categories",
-        &[],
+        &[SERVER_LABEL],
         subsystems::SYSTEM_CPU,
     )
 });
 
 pub static SYS_CPU_NICE_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::SysCPUNice, "CPU nice time", &[], subsystems::SYSTEM_CPU));
+    LazyLock::new(|| new_gauge_md(MetricName::SysCPUNice, "CPU nice time", &[SERVER_LABEL], subsystems::SYSTEM_CPU));
 
 pub static SYS_CPU_STEAL_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::SysCPUSteal, "CPU steal time", &[], subsystems::SYSTEM_CPU));
+    LazyLock::new(|| new_gauge_md(MetricName::SysCPUSteal, "CPU steal time", &[SERVER_LABEL], subsystems::SYSTEM_CPU));
 
 pub static SYS_CPU_SYSTEM_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::SysCPUSystem, "CPU system time", &[], subsystems::SYSTEM_CPU));
+    LazyLock::new(|| new_gauge_md(MetricName::SysCPUSystem, "CPU system time", &[SERVER_LABEL], subsystems::SYSTEM_CPU));
 
 pub static SYS_CPU_USER_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::SysCPUUser, "CPU user time", &[], subsystems::SYSTEM_CPU));
+    LazyLock::new(|| new_gauge_md(MetricName::SysCPUUser, "CPU user time", &[SERVER_LABEL], subsystems::SYSTEM_CPU));

crates/obs/src/metrics/schema/system_memory.rs

@@ -14,43 +14,74 @@
 
 #![allow(dead_code)]
 
+use crate::node_identity::SERVER_LABEL;
 use crate::{MetricDescriptor, MetricName, new_gauge_md, subsystems};
 use std::sync::LazyLock;
 
 /// Total memory available on the node
-pub static MEM_TOTAL_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::MemTotal, "Total memory on the node", &[], subsystems::SYSTEM_MEMORY));
+pub static MEM_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
+    new_gauge_md(
+        MetricName::MemTotal,
+        "Total memory on the node",
+        &[SERVER_LABEL],
+        subsystems::SYSTEM_MEMORY,
+    )
+});
 
 /// Memory currently in use on the node
 pub static MEM_USED_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::MemUsed, "Used memory on the node", &[], subsystems::SYSTEM_MEMORY));
+    LazyLock::new(|| new_gauge_md(MetricName::MemUsed, "Used memory on the node", &[SERVER_LABEL], subsystems::SYSTEM_MEMORY));
 
 /// Percentage of total memory currently in use
 pub static MEM_USED_PERC_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
     new_gauge_md(
         MetricName::MemUsedPerc,
         "Used memory percentage on the node",
-        &[],
+        &[SERVER_LABEL],
         subsystems::SYSTEM_MEMORY,
     )
 });
 
 /// Memory not currently in use and available for allocation
 pub static MEM_FREE_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::MemFree, "Free memory on the node", &[], subsystems::SYSTEM_MEMORY));
+    LazyLock::new(|| new_gauge_md(MetricName::MemFree, "Free memory on the node", &[SERVER_LABEL], subsystems::SYSTEM_MEMORY));
 
 /// Memory used for file buffers by the kernel
-pub static MEM_BUFFERS_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::MemBuffers, "Buffers memory on the node", &[], subsystems::SYSTEM_MEMORY));
+pub static MEM_BUFFERS_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
+    new_gauge_md(
+        MetricName::MemBuffers,
+        "Buffers memory on the node",
+        &[SERVER_LABEL],
+        subsystems::SYSTEM_MEMORY,
+    )
+});
 
 /// Memory used for caching file data by the kernel
-pub static MEM_CACHE_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::MemCache, "Cache memory on the node", &[], subsystems::SYSTEM_MEMORY));
+pub static MEM_CACHE_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
+    new_gauge_md(
+        MetricName::MemCache,
+        "Cache memory on the node",
+        &[SERVER_LABEL],
+        subsystems::SYSTEM_MEMORY,
+    )
+});
 
 /// Memory shared between multiple processes
-pub static MEM_SHARED_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::MemShared, "Shared memory on the node", &[], subsystems::SYSTEM_MEMORY));
+pub static MEM_SHARED_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
+    new_gauge_md(
+        MetricName::MemShared,
+        "Shared memory on the node",
+        &[SERVER_LABEL],
+        subsystems::SYSTEM_MEMORY,
+    )
+});
 
 /// Estimate of memory available for new applications without swapping
-pub static MEM_AVAILABLE_MD: LazyLock<MetricDescriptor> =
-    LazyLock::new(|| new_gauge_md(MetricName::MemAvailable, "Available memory on the node", &[], subsystems::SYSTEM_MEMORY));
+pub static MEM_AVAILABLE_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
+    new_gauge_md(
+        MetricName::MemAvailable,
+        "Available memory on the node",
+        &[SERVER_LABEL],
+        subsystems::SYSTEM_MEMORY,
+    )
+});

crates/obs/src/metrics/schema/system_network.rs

@@ -14,6 +14,7 @@
 
 #![allow(dead_code)]
 
+use crate::node_identity::SERVER_LABEL;
 use crate::{MetricDescriptor, MetricName, new_counter_md, new_gauge_md, subsystems};
 use std::sync::LazyLock;
 
@@ -22,7 +23,7 @@ pub static INTERNODE_ERRORS_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock::new
     new_counter_md(
         MetricName::InternodeErrorsTotal,
         "Total number of failed internode calls",
-        &[],
+        &[SERVER_LABEL],
         subsystems::SYSTEM_NETWORK_INTERNODE,
     )
 });
@@ -32,7 +33,7 @@ pub static INTERNODE_DIAL_ERRORS_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock
     new_counter_md(
         MetricName::InternodeDialErrorsTotal,
         "Total number of internode TCP dial timeouts and errors",
-        &[],
+        &[SERVER_LABEL],
         subsystems::SYSTEM_NETWORK_INTERNODE,
     )
 });
@@ -42,7 +43,7 @@ pub static INTERNODE_DIAL_AVG_TIME_NANOS_MD: LazyLock<MetricDescriptor> = LazyLo
     new_gauge_md(
         MetricName::InternodeDialAvgTimeNanos,
         "Average dial time of internode TCP calls in nanoseconds",
-        &[],
+        &[SERVER_LABEL],
         subsystems::SYSTEM_NETWORK_INTERNODE,
     )
 });
@@ -52,7 +53,7 @@ pub static INTERNODE_SENT_BYTES_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock:
     new_counter_md(
         MetricName::InternodeSentBytesTotal,
         "Total number of bytes sent to other peer nodes",
-        &[],
+        &[SERVER_LABEL],
         subsystems::SYSTEM_NETWORK_INTERNODE,
     )
 });
@@ -62,7 +63,7 @@ pub static INTERNODE_RECV_BYTES_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock:
     new_counter_md(
         MetricName::InternodeRecvBytesTotal,
         "Total number of bytes received from other peer nodes",
-        &[],
+        &[SERVER_LABEL],
         subsystems::SYSTEM_NETWORK_INTERNODE,
     )
 });

crates/obs/src/metrics/schema/system_network_host.rs

@@ -14,6 +14,7 @@
 
 #![allow(dead_code)]
 
+use crate::node_identity::SERVER_LABEL;
 use crate::{MetricDescriptor, MetricName, new_counter_md, subsystems};
 use std::sync::LazyLock;
 
@@ -25,7 +26,7 @@ pub static HOST_NETWORK_IO_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
     new_counter_md(
         MetricName::HostNetworkIO,
         "Network bytes transferred across system network interfaces",
-        &[DIRECTION_LABEL],
+        &[SERVER_LABEL, DIRECTION_LABEL],
         subsystems::SYSTEM_NETWORK_HOST,
     )
 });
@@ -35,7 +36,7 @@ pub static HOST_NETWORK_IO_PER_INTERFACE_MD: LazyLock<MetricDescriptor> = LazyLo
     new_counter_md(
         MetricName::HostNetworkIOPerInterface,
         "Network bytes transferred across system network interfaces (per interface)",
-        &[INTERFACE_LABEL, DIRECTION_LABEL],
+        &[SERVER_LABEL, INTERFACE_LABEL, DIRECTION_LABEL],
         subsystems::SYSTEM_NETWORK_HOST,
     )
 });
@@ -48,12 +49,19 @@ mod tests {
     #[test]
     fn host_network_descriptors_export_counter_labels() {
         assert_eq!(HOST_NETWORK_IO_MD.metric_type, MetricType::Counter);
-        assert_eq!(HOST_NETWORK_IO_MD.variable_labels, vec![DIRECTION_LABEL.to_string()]);
+        assert_eq!(
+            HOST_NETWORK_IO_MD.variable_labels,
+            vec![SERVER_LABEL.to_string(), DIRECTION_LABEL.to_string()]
+        );
 
         assert_eq!(HOST_NETWORK_IO_PER_INTERFACE_MD.metric_type, MetricType::Counter);
         assert_eq!(
             HOST_NETWORK_IO_PER_INTERFACE_MD.variable_labels,
-            vec![INTERFACE_LABEL.to_string(), DIRECTION_LABEL.to_string()]
+            vec![
+                SERVER_LABEL.to_string(),
+                INTERFACE_LABEL.to_string(),
+                DIRECTION_LABEL.to_string()
+            ]
         );
     }
 }

crates/obs/src/metrics/schema/system_process.rs

@@ -14,15 +14,31 @@
 
 #![allow(dead_code)]
 
+use crate::node_identity::SERVER_LABEL;
 use crate::{MetricDescriptor, MetricName, new_counter_md, new_gauge_md, subsystems};
 use std::sync::LazyLock;
 
+pub const PROCESS_PID_LABEL: &str = "process_pid";
+pub const PROCESS_EXECUTABLE_NAME_LABEL: &str = "process_executable_name";
+pub const DIRECTION_LABEL: &str = "direction";
+pub const STATUS_LABEL: &str = "status";
+
+const PROCESS_LABELS: &[&str] = &[SERVER_LABEL];
+const PROCESS_WITH_ATTRIBUTES_LABELS: &[&str] = &[SERVER_LABEL, PROCESS_PID_LABEL, PROCESS_EXECUTABLE_NAME_LABEL];
+const PROCESS_DISK_IO_LABELS: &[&str] = &[
+    DIRECTION_LABEL,
+    SERVER_LABEL,
+    PROCESS_PID_LABEL,
+    PROCESS_EXECUTABLE_NAME_LABEL,
+];
+const PROCESS_STATUS_LABELS: &[&str] = &[SERVER_LABEL, STATUS_LABEL];
+
 /// Number of current READ locks on this peer
 pub static PROCESS_LOCKS_READ_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
     new_gauge_md(
         MetricName::ProcessLocksReadTotal,
         "Number of current READ locks on this peer",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -32,7 +48,7 @@ pub static PROCESS_LOCKS_WRITE_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock::
     new_gauge_md(
         MetricName::ProcessLocksWriteTotal,
         "Number of current WRITE locks on this peer",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -42,7 +58,7 @@ pub static PROCESS_CPU_TOTAL_SECONDS_MD: LazyLock<MetricDescriptor> = LazyLock::
     new_counter_md(
         MetricName::ProcessCPUTotalSeconds,
         "Total user and system CPU time spent in seconds",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -52,7 +68,7 @@ pub static PROCESS_GO_ROUTINE_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock::n
     new_gauge_md(
         MetricName::ProcessGoRoutineTotal,
         "Total number of go routines running",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -62,7 +78,7 @@ pub static PROCESS_IO_RCHAR_BYTES_MD: LazyLock<MetricDescriptor> = LazyLock::new
     new_counter_md(
         MetricName::ProcessIORCharBytes,
         "Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -72,7 +88,7 @@ pub static PROCESS_IO_READ_BYTES_MD: LazyLock<MetricDescriptor> = LazyLock::new(
     new_counter_md(
         MetricName::ProcessIOReadBytes,
         "Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -82,7 +98,7 @@ pub static PROCESS_IO_WCHAR_BYTES_MD: LazyLock<MetricDescriptor> = LazyLock::new
     new_counter_md(
         MetricName::ProcessIOWCharBytes,
         "Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -92,7 +108,7 @@ pub static PROCESS_IO_WRITE_BYTES_MD: LazyLock<MetricDescriptor> = LazyLock::new
     new_counter_md(
         MetricName::ProcessIOWriteBytes,
         "Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -102,7 +118,7 @@ pub static PROCESS_START_TIME_SECONDS_MD: LazyLock<MetricDescriptor> = LazyLock:
     new_gauge_md(
         MetricName::ProcessStartTimeSeconds,
         "Start time for RustFS process in seconds since Unix epoch",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -112,7 +128,7 @@ pub static PROCESS_UPTIME_SECONDS_MD: LazyLock<MetricDescriptor> = LazyLock::new
     new_gauge_md(
         MetricName::ProcessUptimeSeconds,
         "Uptime for RustFS process in seconds",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -122,7 +138,7 @@ pub static PROCESS_FILE_DESCRIPTOR_LIMIT_TOTAL_MD: LazyLock<MetricDescriptor> =
     new_gauge_md(
         MetricName::ProcessFileDescriptorLimitTotal,
         "Limit on total number of open file descriptors for the RustFS Server process",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -132,7 +148,7 @@ pub static PROCESS_FILE_DESCRIPTOR_OPEN_TOTAL_MD: LazyLock<MetricDescriptor> = L
     new_gauge_md(
         MetricName::ProcessFileDescriptorOpenTotal,
         "Total number of open file descriptors by the RustFS Server process",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -142,7 +158,7 @@ pub static PROCESS_SYSCALL_READ_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock:
     new_counter_md(
         MetricName::ProcessSyscallReadTotal,
         "Total read SysCalls to the kernel. /proc/[pid]/io syscr",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -152,7 +168,7 @@ pub static PROCESS_SYSCALL_WRITE_TOTAL_MD: LazyLock<MetricDescriptor> = LazyLock
     new_counter_md(
         MetricName::ProcessSyscallWriteTotal,
         "Total write SysCalls to the kernel. /proc/[pid]/io syscw",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -162,7 +178,7 @@ pub static PROCESS_RESIDENT_MEMORY_BYTES_MD: LazyLock<MetricDescriptor> = LazyLo
     new_gauge_md(
         MetricName::ProcessResidentMemoryBytes,
         "Resident memory size in bytes",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -172,7 +188,7 @@ pub static PROCESS_VIRTUAL_MEMORY_BYTES_MD: LazyLock<MetricDescriptor> = LazyLoc
     new_gauge_md(
         MetricName::ProcessVirtualMemoryBytes,
         "Virtual memory size in bytes",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -182,7 +198,7 @@ pub static PROCESS_VIRTUAL_MEMORY_MAX_BYTES_MD: LazyLock<MetricDescriptor> = Laz
     new_gauge_md(
         MetricName::ProcessVirtualMemoryMaxBytes,
         "Maximum virtual memory size in bytes",
-        &[],
+        PROCESS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -196,7 +212,7 @@ pub static PROCESS_CPU_USAGE_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
     new_gauge_md(
         MetricName::ProcessCPUUsage,
         "The percentage of CPU in use by the process",
-        &[],
+        PROCESS_WITH_ATTRIBUTES_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -206,7 +222,7 @@ pub static PROCESS_CPU_UTILIZATION_MD: LazyLock<MetricDescriptor> = LazyLock::ne
     new_gauge_md(
         MetricName::ProcessCPUUtilization,
         "The amount of CPU in use by the process (considering multiple cores)",
-        &[],
+        PROCESS_WITH_ATTRIBUTES_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -216,7 +232,7 @@ pub static PROCESS_DISK_IO_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
     new_gauge_md(
         MetricName::ProcessDiskIO,
         "Disk bytes transferred by the process",
-        &[],
+        PROCESS_DISK_IO_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });
@@ -226,7 +242,7 @@ pub static PROCESS_STATUS_MD: LazyLock<MetricDescriptor> = LazyLock::new(|| {
     new_gauge_md(
         MetricName::ProcessStatus,
         "Process status (0: Running, 1: Sleeping, 2: Zombie, 3: Other)",
-        &[],
+        PROCESS_STATUS_LABELS,
         subsystems::SYSTEM_PROCESS,
     )
 });

crates/obs/src/metrics/stats_collector.rs

@@ -33,6 +33,7 @@ use crate::metrics::{
     obs_load_compression_total_from_memory, obs_load_data_usage_from_backend, obs_replication_site_stats_snapshot,
     obs_resolve_object_store_handle,
 };
+use crate::node_identity::current_local_node_identity;
 use chrono::Utc;
 use rustfs_common::heal_channel::HealScanMode;
 use rustfs_common::metrics::{ScannerMetricsReport, global_metrics};
@@ -539,12 +540,13 @@ pub async fn collect_disk_stats() -> Vec<DiskStats> {
     disk_stats
 }
 
-fn build_system_cpu_stats(system: &System) -> CpuStats {
+fn build_system_cpu_stats(system: &System, server: &str) -> CpuStats {
     let cpu_usage = system.global_cpu_usage() as f64;
     let cpu_count = system.cpus().len().max(1) as f64;
     let load_avg = System::load_average().one;
 
     CpuStats {
+        server: server.to_string(),
         avg_idle: (100.0 - cpu_usage).max(0.0),
         load_avg,
         load_avg_perc: (load_avg / cpu_count) * 100.0,
@@ -552,11 +554,12 @@ fn build_system_cpu_stats(system: &System) -> CpuStats {
     }
 }
 
-fn build_system_memory_stats(system: &System) -> MemoryStats {
+fn build_system_memory_stats(system: &System, server: &str) -> MemoryStats {
     let total = system.total_memory();
     let used = system.used_memory();
 
     MemoryStats {
+        server: server.to_string(),
         total,
         used,
         used_perc: if total > 0 {
@@ -582,7 +585,8 @@ pub fn collect_system_cpu_and_memory_stats() -> (CpuStats, MemoryStats) {
 pub fn collect_system_cpu_and_memory_stats_with(system: &mut System) -> (CpuStats, MemoryStats) {
     system.refresh_cpu_all();
     system.refresh_memory();
-    (build_system_cpu_stats(system), build_system_memory_stats(system))
+    let server = current_local_node_identity();
+    (build_system_cpu_stats(system, &server), build_system_memory_stats(system, &server))
 }
 
 /// Collect system CPU statistics from the current host.
@@ -692,6 +696,7 @@ fn process_metric_bundle_from_snapshots(
     resource_snapshot: ProcessResourceSnapshot,
     process_snapshot: ProcessSystemSnapshot,
 ) -> ProcessMetricBundle {
+    let server = current_local_node_identity();
     let status = match process_snapshot.status {
         ProcessStatusSnapshot::Running => ProcessStatusType::Running,
         ProcessStatusSnapshot::Sleeping => ProcessStatusType::Sleeping,
@@ -700,11 +705,13 @@ fn process_metric_bundle_from_snapshots(
     };
 
     let resource_stats = ResourceStats {
+        server: server.clone(),
         cpu_percent: resource_snapshot.cpu_percent,
         memory_bytes: resource_snapshot.memory_bytes,
         uptime_seconds: resource_snapshot.uptime_seconds,
     };
     let process_stats = ProcessStats {
+        server,
         locks_read_total: process_snapshot.locks_read_total,
         locks_write_total: process_snapshot.locks_write_total,
         cpu_total_seconds: process_snapshot.cpu_total_seconds,
@@ -770,6 +777,7 @@ pub fn collect_host_network_stats_with(networks: &Networks) -> HostNetworkStats
     }
 
     HostNetworkStats {
+        server: current_local_node_identity(),
         total_received,
         total_transmitted,
         per_interface,
@@ -794,6 +802,7 @@ pub fn collect_internode_network_stats() -> Option<NetworkStats> {
     let snapshot = global_internode_metrics().snapshot();
 
     Some(NetworkStats {
+        server: current_local_node_identity(),
         internode_errors_total: snapshot.errors_total,
         internode_dial_errors_total: snapshot.dial_errors_total,
         internode_dial_avg_time_nanos: snapshot.dial_avg_time_nanos,
@@ -1307,6 +1316,27 @@ mod tests {
         assert!(cluster_config_stats_from_backend_parities(Some(1), Some(overflow)).is_none());
     }
 
+    #[tokio::test]
+    async fn node_local_resource_stats_use_stable_local_node_identity() {
+        let _guard = crate::node_identity::local_node_identity_test_guard().await;
+        let previous = rustfs_common::get_global_local_node_name().await;
+        rustfs_common::set_global_local_node_name("node1:9000").await;
+
+        let mut system = System::new_all();
+        let (cpu, memory) = collect_system_cpu_and_memory_stats_with(&mut system);
+        let host_network = collect_host_network_stats_with(&Networks::new());
+        let process_bundle =
+            process_metric_bundle_from_snapshots(ProcessResourceSnapshot::default(), ProcessSystemSnapshot::default());
+
+        assert_eq!(cpu.server, "node1:9000");
+        assert_eq!(memory.server, "node1:9000");
+        assert_eq!(host_network.server, "node1:9000");
+        assert_eq!(process_bundle.resource.server, "node1:9000");
+        assert_eq!(process_bundle.process.server, "node1:9000");
+
+        rustfs_common::set_global_local_node_name(&previous).await;
+    }
+
     #[test]
     fn erasure_set_stats_skip_unknown_backend_layout() {
         let storage_info = storage_info_with_one_online_disk();

crates/obs/src/node_identity.rs

@@ -0,0 +1,34 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+pub(crate) const RUSTFS_NODE_ATTRIBUTE: &str = "rustfs.node";
+pub(crate) const SERVER_LABEL: &str = "server";
+
+#[cfg(test)]
+static LOCAL_NODE_IDENTITY_TEST_LOCK: std::sync::LazyLock<tokio::sync::Mutex<()>> =
+    std::sync::LazyLock::new(|| tokio::sync::Mutex::new(()));
+
+pub(crate) fn local_node_identity(local_ip: &str) -> String {
+    rustfs_common::try_get_global_local_node_name().unwrap_or_else(|| local_ip.to_string())
+}
+
+pub(crate) fn current_local_node_identity() -> String {
+    let local_ip = rustfs_utils::get_local_ip_with_default();
+    local_node_identity(&local_ip)
+}
+
+#[cfg(test)]
+pub(crate) async fn local_node_identity_test_guard() -> tokio::sync::MutexGuard<'static, ()> {
+    LOCAL_NODE_IDENTITY_TEST_LOCK.lock().await
+}

crates/obs/src/telemetry/resource.rs

@@ -16,15 +16,20 @@
 //!
 //! A `Resource` describes the entity producing telemetry data.  The resource
 //! built here includes the service name, service version, deployment
-//! environment, and the local machine IP address so that data can be
-//! correlated across services in a distributed system.
+//! environment, the stable RustFS node identity, and the local machine IP
+//! address so that data can be correlated across services in a distributed
+//! system.
 
 use crate::config::OtelConfig;
+use crate::node_identity::{RUSTFS_NODE_ATTRIBUTE, local_node_identity};
 use opentelemetry::KeyValue;
 use opentelemetry_sdk::Resource;
 use opentelemetry_semantic_conventions::{
     SCHEMA_URL,
-    attribute::{DEPLOYMENT_ENVIRONMENT_NAME, NETWORK_LOCAL_ADDRESS, SERVICE_VERSION as OTEL_SERVICE_VERSION},
+    attribute::{
+        DEPLOYMENT_ENVIRONMENT_NAME, NETWORK_LOCAL_ADDRESS, SERVICE_INSTANCE_ID as OTEL_SERVICE_INSTANCE_ID,
+        SERVICE_VERSION as OTEL_SERVICE_VERSION,
+    },
 };
 use rustfs_config::{APP_NAME, ENVIRONMENT, SERVICE_VERSION};
 use rustfs_utils::get_local_ip_with_default;
@@ -38,12 +43,18 @@ use std::borrow::Cow;
 ///   [`SERVICE_VERSION`].
 /// - `deployment.environment` — from `config.environment`, defaulting to
 ///   [`ENVIRONMENT`].
+/// - `rustfs.node` / `service.instance.id` — the stable RustFS local node name
+///   when available, falling back to the local IP during early startup.
 /// - `network.local.address` — the primary local IP of the current host,
-///   useful for identifying individual nodes in a cluster.
+///   useful as an operational fallback when the stable node name is not yet
+///   initialized.
 ///
 /// All attributes are attached to the resource using the semantic conventions
 /// schema URL to ensure compatibility with standard OTLP backends.
 pub(super) fn build_resource(config: &OtelConfig) -> Resource {
+    let local_ip = get_local_ip_with_default();
+    let node_identity = local_node_identity(&local_ip);
+
     Resource::builder()
         .with_service_name(Cow::Borrowed(config.service_name.as_deref().unwrap_or(APP_NAME)).to_string())
         .with_schema_url(
@@ -56,9 +67,41 @@ pub(super) fn build_resource(config: &OtelConfig) -> Resource {
                     DEPLOYMENT_ENVIRONMENT_NAME,
                     Cow::Borrowed(config.environment.as_deref().unwrap_or(ENVIRONMENT)).to_string(),
                 ),
-                KeyValue::new(NETWORK_LOCAL_ADDRESS, get_local_ip_with_default()),
+                KeyValue::new(RUSTFS_NODE_ATTRIBUTE, node_identity.clone()),
+                KeyValue::new(OTEL_SERVICE_INSTANCE_ID, node_identity),
+                KeyValue::new(NETWORK_LOCAL_ADDRESS, local_ip),
             ],
             SCHEMA_URL,
         )
         .build()
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use opentelemetry::Key;
+
+    #[tokio::test]
+    async fn build_resource_uses_stable_local_node_identity() {
+        let _guard = crate::node_identity::local_node_identity_test_guard().await;
+        let previous = rustfs_common::get_global_local_node_name().await;
+        rustfs_common::set_global_local_node_name("node1:9000").await;
+
+        let resource = build_resource(&OtelConfig::default());
+
+        assert_eq!(
+            resource
+                .get(&Key::from_static_str(RUSTFS_NODE_ATTRIBUTE))
+                .map(|value| value.to_string()),
+            Some("node1:9000".to_string())
+        );
+        assert_eq!(
+            resource
+                .get(&Key::from_static_str(OTEL_SERVICE_INSTANCE_ID))
+                .map(|value| value.to_string()),
+            Some("node1:9000".to_string())
+        );
+
+        rustfs_common::set_global_local_node_name(&previous).await;
+    }
+}

crates/replication/src/config.rs

@@ -51,28 +51,6 @@ pub enum ReplicationTargetValidationError {
     StaleTarget,
 }
 
-pub fn unsupported_replication_config_field(config: &ReplicationConfiguration) -> Option<&'static str> {
-    for rule in &config.rules {
-        if rule
-            .source_selection_criteria
-            .as_ref()
-            .is_some_and(|criteria| criteria.sse_kms_encrypted_objects.is_some())
-        {
-            return Some("SourceSelectionCriteria.SseKmsEncryptedObjects");
-        }
-        if rule.destination.encryption_configuration.is_some() {
-            return Some("Destination.EncryptionConfiguration");
-        }
-        if rule.destination.metrics.is_some() {
-            return Some("Destination.Metrics");
-        }
-        if rule.destination.replication_time.is_some() {
-            return Some("Destination.ReplicationTime");
-        }
-    }
-    None
-}
-
 pub fn active_replication_rule_destination_arns(config: &ReplicationConfiguration) -> HashSet<String> {
     let mut arns = HashSet::new();
 
@@ -239,6 +217,11 @@ impl ReplicationConfigurationExt for ReplicationConfiguration {
                 }
 
                 if obj.version_id.is_some() {
+                    if obj.delete_marker {
+                        return rule.delete_marker_replication.clone().is_some_and(|d| {
+                            d.status == Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED))
+                        });
+                    }
                     return rule
                         .delete_replication
                         .clone()
@@ -322,11 +305,7 @@ impl ReplicationConfigurationExt for ReplicationConfiguration {
 #[cfg(test)]
 mod tests {
     use super::*;
-    use s3s::dto::{
-        DeleteMarkerReplication, DeleteReplication, Destination, EncryptionConfiguration, ExistingObjectReplication, Metrics,
-        MetricsStatus, ReplicationRule, ReplicationTime, ReplicationTimeStatus, ReplicationTimeValue, SourceSelectionCriteria,
-        SseKmsEncryptedObjects, SseKmsEncryptedObjectsStatus,
-    };
+    use s3s::dto::{DeleteMarkerReplication, Destination, ExistingObjectReplication, ReplicationRule};
 
     fn replication_rule(id: &str, arn: &str) -> ReplicationRule {
         ReplicationRule {
@@ -626,99 +605,4 @@ mod tests {
             "highest-priority rule disables delete-marker replication, so the delete marker must not replicate"
         );
     }
-
-    #[test]
-    fn version_purge_uses_delete_replication_for_object_and_marker_versions() {
-        let arn = "arn:rustfs:replication:us-east-1:target:bucket";
-        let mut rule = replication_rule("delete", arn);
-        rule.delete_marker_replication = Some(DeleteMarkerReplication {
-            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
-        });
-        rule.delete_replication = Some(DeleteReplication {
-            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
-        });
-        let mut config = ReplicationConfiguration {
-            role: String::new(),
-            rules: vec![rule],
-        };
-        let version_id = Some(Uuid::new_v4());
-
-        for delete_marker in [false, true] {
-            assert!(config.replicate(&ObjectOpts {
-                name: "object".to_string(),
-                version_id,
-                delete_marker,
-                op_type: ReplicationType::Delete,
-                ..Default::default()
-            }));
-        }
-        assert!(!config.replicate(&ObjectOpts {
-            name: "object".to_string(),
-            delete_marker: true,
-            op_type: ReplicationType::Delete,
-            ..Default::default()
-        }));
-
-        let rule = &mut config.rules[0];
-        rule.delete_marker_replication = Some(DeleteMarkerReplication {
-            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
-        });
-        rule.delete_replication = Some(DeleteReplication {
-            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
-        });
-
-        for delete_marker in [false, true] {
-            assert!(!config.replicate(&ObjectOpts {
-                name: "object".to_string(),
-                version_id,
-                delete_marker,
-                op_type: ReplicationType::Delete,
-                ..Default::default()
-            }));
-        }
-        assert!(config.replicate(&ObjectOpts {
-            name: "object".to_string(),
-            delete_marker: true,
-            op_type: ReplicationType::Delete,
-            ..Default::default()
-        }));
-    }
-
-    #[test]
-    fn unsupported_replication_fields_are_reported_before_persistence() {
-        let arn = "arn:rustfs:replication:us-east-1:target:bucket";
-        let mut config = ReplicationConfiguration {
-            role: String::new(),
-            rules: vec![replication_rule("unsupported", arn)],
-        };
-
-        config.rules[0].source_selection_criteria = Some(SourceSelectionCriteria {
-            replica_modifications: None,
-            sse_kms_encrypted_objects: Some(SseKmsEncryptedObjects {
-                status: SseKmsEncryptedObjectsStatus::from_static(SseKmsEncryptedObjectsStatus::ENABLED),
-            }),
-        });
-        assert_eq!(
-            unsupported_replication_config_field(&config),
-            Some("SourceSelectionCriteria.SseKmsEncryptedObjects")
-        );
-
-        config.rules[0].source_selection_criteria = None;
-        config.rules[0].destination.encryption_configuration = Some(EncryptionConfiguration::default());
-        assert_eq!(unsupported_replication_config_field(&config), Some("Destination.EncryptionConfiguration"));
-
-        config.rules[0].destination.encryption_configuration = None;
-        config.rules[0].destination.metrics = Some(Metrics {
-            event_threshold: None,
-            status: MetricsStatus::from_static(MetricsStatus::ENABLED),
-        });
-        assert_eq!(unsupported_replication_config_field(&config), Some("Destination.Metrics"));
-
-        config.rules[0].destination.metrics = None;
-        config.rules[0].destination.replication_time = Some(ReplicationTime {
-            status: ReplicationTimeStatus::from_static(ReplicationTimeStatus::ENABLED),
-            time: ReplicationTimeValue { minutes: Some(15) },
-        });
-        assert_eq!(unsupported_replication_config_field(&config), Some("Destination.ReplicationTime"));
-    }
 }

crates/replication/src/lib.rs

@@ -30,8 +30,7 @@ pub mod tagging;
 
 pub use config::{
     ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, active_replication_rule_destination_arns,
-    replication_target_arns, should_remove_replication_target, unsupported_replication_config_field,
-    validate_replication_config_target_arns,
+    replication_target_arns, should_remove_replication_target, validate_replication_config_target_arns,
 };
 pub use delete::{
     DeletedObjectReplicationInfo, is_retryable_delete_replication_head_error, is_version_delete_replication,

crates/replication/src/operation.rs

@@ -322,9 +322,9 @@ mod tests {
     use crate::storage_api::ObjectToDelete;
     use crate::{ReplicationStatusType, ReplicationType, VersionPurgeStatusType, target_reset_header};
     use s3s::dto::{
-        DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
-        ExistingObjectReplication, ExistingObjectReplicationStatus, ReplicaModifications, ReplicaModificationsStatus,
-        ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus, SourceSelectionCriteria,
+        DeleteMarkerReplication, DeleteMarkerReplicationStatus, Destination, ExistingObjectReplication,
+        ExistingObjectReplicationStatus, ReplicaModifications, ReplicaModificationsStatus, ReplicationConfiguration,
+        ReplicationRule, ReplicationRuleStatus, SourceSelectionCriteria,
     };
     use std::collections::HashMap;
     use time::{Duration, OffsetDateTime};
@@ -433,9 +433,7 @@ mod tests {
             delete_marker_replication: Some(DeleteMarkerReplication {
                 status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
             }),
-            delete_replication: Some(DeleteReplication {
-                status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
-            }),
+            delete_replication: None,
             destination: Destination {
                 bucket: arn.to_string(),
                 ..Default::default()
@@ -515,7 +513,7 @@ mod tests {
         };
 
         let state = delete_replication_state_from_config(&config, &source)
-            .expect("delete-marker version purge should honor delete replication rules");
+            .expect("delete-marker version purge should honor delete-marker replication rules");
         let pending = format!("{arn}=PENDING;");
 
         assert_eq!(state.version_purge_status_internal.as_deref(), Some(pending.as_str()));

crates/rio-v2/tests/minio_fixture_lab/README.md

@@ -137,7 +137,7 @@ tests read):
 ./capture_via_docker.sh
 
 RUSTFS_MINIO_STATIC_KMS_KEY_B64=IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= \
-  cargo test -p rustfs-ecstore --features rio-v2 --test minio_generated_read_test -- --ignored
+  cargo test -p rustfs --features rio-v2 storage::minio_generated_read_test --lib -- --ignored
 ```
 
 This is exactly what the nightly `minio-interop` GitHub Actions workflow runs

crates/utils/src/http/header_compat.rs

@@ -25,6 +25,11 @@ const RUSTFS_PREFIX: &str = "x-rustfs-";
 const MINIO_PREFIX: &str = "x-minio-";
 const MINIO_ENCRYPTION_PREFIX: &str = "x-minio-encryption-";
 const RUSTFS_ENCRYPTION_PREFIX: &str = "x-rustfs-encryption-";
+const MINIO_INTERNAL_ENCRYPTION_PREFIX: &str = "x-minio-internal-server-side-encryption-";
+const MINIO_INTERNAL_ENCRYPTED_MULTIPART: &str = "x-minio-internal-encrypted-multipart";
+const RUSTFS_ENCRYPTION_ORIGINAL_SIZE: &str = "x-rustfs-encryption-original-size";
+const MINIO_ENCRYPTION_ORIGINAL_SIZE: &str = "x-minio-encryption-original-size";
+const SSEC_ORIGINAL_SIZE: &str = "x-amz-server-side-encryption-customer-original-size";
 
 // Suffix constants (part after x-rustfs- or x-minio-). Use with get_header/insert_header.
 pub const SUFFIX_FORCE_DELETE: &str = "force-delete";
@@ -40,11 +45,49 @@ pub const SUFFIX_SOURCE_REPLICATION_REQUEST: &str = "source-replication-request"
 pub const SUFFIX_SOURCE_REPLICATION_CHECK: &str = "source-replication-check";
 pub const SUFFIX_REPLICATION_SSEC_CRC: &str = "replication-ssec-crc";
 
-/// Returns true if the key is an internal encryption metadata key (x-rustfs-encryption-* or
-/// x-minio-encryption-*). Case-insensitive for metadata filtering.
+/// Returns true if the key is object-encryption metadata understood by RustFS or MinIO.
+/// Case-insensitive for metadata filtering.
 pub fn is_encryption_metadata_key(key: &str) -> bool {
     let lower = key.to_lowercase();
-    lower.starts_with(RUSTFS_ENCRYPTION_PREFIX) || lower.starts_with(MINIO_ENCRYPTION_PREFIX)
+    lower.starts_with(RUSTFS_ENCRYPTION_PREFIX)
+        || lower.starts_with(MINIO_ENCRYPTION_PREFIX)
+        || lower.starts_with(MINIO_INTERNAL_ENCRYPTION_PREFIX)
+        || lower == MINIO_INTERNAL_ENCRYPTED_MULTIPART
+}
+
+/// Returns true when a metadata key proves that object data is encrypted.
+///
+/// Original-size metadata alone is not proof: older plaintext objects can
+/// retain that compatibility field after metadata migration.
+pub fn is_object_encryption_marker(key: &str) -> bool {
+    (is_encryption_metadata_key(key)
+        && !key.eq_ignore_ascii_case(RUSTFS_ENCRYPTION_ORIGINAL_SIZE)
+        && !key.eq_ignore_ascii_case(MINIO_ENCRYPTION_ORIGINAL_SIZE))
+        || super::is_sse_header(key)
+}
+
+/// Reads the logical object size recorded by encryption metadata.
+pub fn get_object_encryption_original_size(metadata: &std::collections::HashMap<String, String>) -> std::io::Result<Option<i64>> {
+    let actual_size = super::get_str(metadata, super::SUFFIX_ACTUAL_SIZE);
+    let size = get_case_insensitive(metadata, RUSTFS_ENCRYPTION_ORIGINAL_SIZE)
+        .or_else(|| get_case_insensitive(metadata, SSEC_ORIGINAL_SIZE))
+        .or(actual_size.as_deref());
+
+    let Some(size) = size.filter(|size| !size.is_empty()) else {
+        return Ok(None);
+    };
+    size.parse::<i64>()
+        .map(Some)
+        .map_err(|error| std::io::Error::other(format!("Failed to parse encryption original size: {error}")))
+}
+
+fn get_case_insensitive<'a>(metadata: &'a std::collections::HashMap<String, String>, key: &str) -> Option<&'a str> {
+    metadata.get(key).map(String::as_str).or_else(|| {
+        metadata
+            .iter()
+            .find(|(candidate, _)| candidate.eq_ignore_ascii_case(key))
+            .map(|(_, value)| value.as_str())
+    })
 }
 
 fn rustfs_key(suffix: &str) -> String {
@@ -106,10 +149,37 @@ mod tests {
         assert!(is_encryption_metadata_key("x-rustfs-encryption-iv"));
         assert!(is_encryption_metadata_key("X-Rustfs-Encryption-Key"));
         assert!(is_encryption_metadata_key("x-minio-encryption-iv"));
+        assert!(is_encryption_metadata_key("X-Minio-Internal-Server-Side-Encryption-Sealed-Key"));
+        assert!(is_encryption_metadata_key("X-Minio-Internal-Encrypted-Multipart"));
         assert!(!is_encryption_metadata_key("x-amz-meta-custom"));
         assert!(!is_encryption_metadata_key("x-rustfs-internal-healing"));
     }
 
+    #[test]
+    fn object_encryption_marker_excludes_size_only_metadata() {
+        assert!(!is_object_encryption_marker(RUSTFS_ENCRYPTION_ORIGINAL_SIZE));
+        assert!(is_object_encryption_marker("X-Minio-Internal-Server-Side-Encryption-Sealed-Key"));
+        assert!(is_object_encryption_marker("x-amz-server-side-encryption"));
+    }
+
+    #[test]
+    fn object_encryption_original_size_is_case_insensitive() {
+        let metadata = std::collections::HashMap::from([(
+            "X-Amz-Server-Side-Encryption-Customer-Original-Size".to_string(),
+            "42".to_string(),
+        )]);
+        assert_eq!(get_object_encryption_original_size(&metadata).expect("valid size"), Some(42));
+    }
+
+    #[test]
+    fn object_encryption_original_size_prefers_rustfs_metadata() {
+        let metadata = std::collections::HashMap::from([
+            (SSEC_ORIGINAL_SIZE.to_string(), "21".to_string()),
+            (RUSTFS_ENCRYPTION_ORIGINAL_SIZE.to_string(), "42".to_string()),
+        ]);
+        assert_eq!(get_object_encryption_original_size(&metadata).expect("valid size"), Some(42));
+    }
+
     #[test]
     fn test_get_header() {
         let mut headers = HeaderMap::new();

crates/utils/src/net.rs

@@ -92,7 +92,6 @@ fn resolve_domain(domain: &str) -> std::io::Result<HashSet<IpAddr>> {
     (domain, 0)
         .to_socket_addrs()
         .map(|v| v.map(|v| v.ip()).collect::<HashSet<_>>())
-        .map_err(Error::other)
 }
 
 #[cfg(test)]
@@ -284,7 +283,7 @@ pub async fn get_host_ip(host: Host<&str>) -> std::io::Result<HashSet<IpAddr>> {
                 }
                 Err(err) => {
                     error!("Failed to resolve domain {domain} using system resolver, err: {err}");
-                    Err(Error::other(err))
+                    Err(err)
                 }
             }
         }
@@ -608,6 +607,27 @@ mod test {
         assert!(get_host_ip(invalid_host).await.is_err());
     }
 
+    #[tokio::test]
+    async fn test_get_host_ip_preserves_resolver_error_provenance() {
+        let _resolver_guard = set_mock_dns_resolver(|_| Err(IoError::from_raw_os_error(-3)));
+
+        let err = get_host_ip(Host::Domain("temporarily-unavailable.example"))
+            .await
+            .unwrap_err();
+
+        assert_eq!(err.raw_os_error(), Some(-3));
+    }
+
+    #[test]
+    fn test_resolve_domain_preserves_system_resolver_error_provenance() {
+        let _resolver_lock = DNS_RESOLVER_TEST_LOCK.lock().unwrap();
+        reset_dns_resolver_inner();
+
+        let err = resolve_domain("rustfs-resolver-provenance.invalid").unwrap_err();
+
+        assert_ne!(err.kind(), std::io::ErrorKind::Other, "system resolver error was wrapped: {err}");
+    }
+
     #[test]
     fn test_dns_cache_entry_expiry() {
         let ips: HashSet<IpAddr> = [IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1))].into_iter().collect();

crates/utils/src/string.rs

@@ -360,7 +360,7 @@ pub fn find_ellipses_patterns(arg: &str) -> Result<ArgPattern> {
         Some(caps) => caps,
         None => {
             return Err(Error::other(format!(
-                "Invalid ellipsis format in ({arg}), Ellipsis range must be provided in format {{N...M}} where N and M are decimal or hexadecimal positive integers, M must be greater than N, with a maximum expanded range size of {MAX_ELLIPSES_RANGE_SIZE}"
+                "Invalid ellipsis format. Ellipsis range must be provided in format {{N...M}} where N and M are decimal or hexadecimal positive integers, M must be greater than N, with a maximum expanded range size of {MAX_ELLIPSES_RANGE_SIZE}"
             )));
         }
     };
@@ -399,7 +399,7 @@ pub fn find_ellipses_patterns(arg: &str) -> Result<ArgPattern> {
             || p.suffix.contains(CLOSE_BRACES)
         {
             return Err(Error::other(format!(
-                "Invalid ellipsis format in ({arg}), Ellipsis range must be provided in format {{N...M}} where N and M are decimal or hexadecimal positive integers, M must be greater than N, with a maximum expanded range size of {MAX_ELLIPSES_RANGE_SIZE}"
+                "Invalid ellipsis format. Ellipsis range must be provided in format {{N...M}} where N and M are decimal or hexadecimal positive integers, M must be greater than N, with a maximum expanded range size of {MAX_ELLIPSES_RANGE_SIZE}"
             )));
         }
     }
@@ -938,6 +938,15 @@ mod tests {
         assert!(err.to_string().contains("decimal or hexadecimal"), "unexpected error message: {err}");
     }
 
+    #[test]
+    fn test_find_ellipses_patterns_leftover_brace_error_does_not_echo_input() {
+        let err = find_ellipses_patterns("http://:brace-secret@server/{1...2}}").unwrap_err();
+        assert!(
+            !err.to_string().contains("brace-secret"),
+            "ellipsis error leaked endpoint credentials: {err}"
+        );
+    }
+
     #[test]
     fn test_parse_ellipses_range_rejects_oversized_ranges() {
         let err = parse_ellipses_range("{0...10000}").unwrap_err();

deploy/observability/README.md

@@ -0,0 +1,11 @@
+# RustFS Observability Dashboards
+
+This directory contains optional dashboards and observability assets for operating RustFS deployments.
+
+## Grafana
+
+Import `grafana/rustfs-node-observability.json` into Grafana and select a Prometheus data source that scrapes RustFS metrics.
+
+The dashboard uses the RustFS `server` metric label introduced with the node-local observability updates. `server` represents the RustFS node identity and is preferred for RustFS node comparisons. Prometheus `instance` still identifies the scrape target and remains useful for scrape/debugging views, but dashboards that compare RustFS nodes should group and filter by `server`.
+
+During a rolling upgrade, older nodes may still emit metrics without the `server` label. Complete the rollout before using this dashboard for node-by-node comparisons.

deploy/observability/grafana/rustfs-node-observability.json

@@ -0,0 +1,885 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "target": {
+          "limit": 100,
+          "matchAny": false,
+          "tags": [],
+          "type": "dashboard"
+        },
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 12,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 4,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (server) (rate(rustfs_system_network_internode_sent_bytes_total{server=~\"$server\"}[$__rate_interval]))",
+          "legendFormat": "{{server}} sent",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (server) (rate(rustfs_system_network_internode_recv_bytes_total{server=~\"$server\"}[$__rate_interval]))",
+          "legendFormat": "{{server}} received",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Internode Traffic by Server",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 4,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (server, operation, backend) (rate(rustfs_system_network_internode_operation_sent_bytes_total{server=~\"$server\"}[$__rate_interval]))",
+          "legendFormat": "{{server}} {{backend}} {{operation}} sent",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (server, operation, backend) (rate(rustfs_system_network_internode_operation_recv_bytes_total{server=~\"$server\"}[$__rate_interval]))",
+          "legendFormat": "{{server}} {{backend}} {{operation}} received",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Internode Operation Traffic",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 4,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 3,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (server, direction) (rate(rustfs_system_network_host_network_io{server=~\"$server\"}[$__rate_interval]))",
+          "legendFormat": "{{server}} {{direction}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Host Network I/O by Server",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 8,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 4,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "Bps"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (server, interface, direction) (rate(rustfs_system_network_host_network_io_per_interface{server=~\"$server\"}[$__rate_interval]))",
+          "legendFormat": "{{server}} {{interface}} {{direction}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Host Network I/O by Interface",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 4,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 70
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          },
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 16
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "avg by (server) (rustfs_system_cpu_usage_perc{server=~\"$server\"})",
+          "legendFormat": "{{server}} CPU",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "CPU Usage by Server",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 4,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "orange",
+                "value": 75
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          },
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 16
+      },
+      "id": 6,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "avg by (server) (rustfs_system_memory_used_perc{server=~\"$server\"})",
+          "legendFormat": "{{server}} memory",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Memory Usage by Server",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 4,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "bytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 24
+      },
+      "id": 7,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "avg by (server) (rustfs_system_process_resident_memory_bytes{server=~\"$server\"})",
+          "legendFormat": "{{server}} resident",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "avg by (server) (rustfs_system_process_virtual_memory_bytes{server=~\"$server\"})",
+          "legendFormat": "{{server}} virtual",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Process Memory by Server",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${datasource}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 4,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          },
+          "unit": "ops"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 24
+      },
+      "id": 8,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "table",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (server) (rate(rustfs_system_network_internode_requests_outgoing_total{server=~\"$server\"}[$__rate_interval]))",
+          "legendFormat": "{{server}} outgoing",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (server) (rate(rustfs_system_network_internode_requests_incoming_total{server=~\"$server\"}[$__rate_interval]))",
+          "legendFormat": "{{server}} incoming",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Internode Request Rate by Server",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "style": "dark",
+  "tags": [
+    "rustfs",
+    "observability",
+    "server-label"
+  ],
+  "templating": {
+    "list": [
+      {
+        "current": {},
+        "hide": 0,
+        "includeAll": false,
+        "label": "Prometheus",
+        "multi": false,
+        "name": "datasource",
+        "options": [],
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      },
+      {
+        "allValue": ".*",
+        "current": {
+          "selected": true,
+          "text": "All",
+          "value": "$__all"
+        },
+        "datasource": {
+          "type": "prometheus",
+          "uid": "${datasource}"
+        },
+        "definition": "label_values(rustfs_system_cpu_usage_perc, server)",
+        "hide": 0,
+        "includeAll": true,
+        "label": "RustFS server",
+        "multi": true,
+        "name": "server",
+        "options": [],
+        "query": "label_values(rustfs_system_cpu_usage_perc, server)",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 1,
+        "type": "query"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "RustFS Node Observability",
+  "uid": "rustfs-node-observability",
+  "version": 1,
+  "weekStart": ""
+}

docs/architecture/compat-cleanup-register.md

@@ -12,6 +12,10 @@ for later deletion.
 
 ## Open Items
 
+- `rustfs-5416` Helm distributed startup wait setting: charts that predate explicit local endpoint identity expose startupWaitTimeoutSeconds for their peer DNS/TCP init gate. The new chart keeps the value accepted but ignores it after moving startup convergence into RustFS. Remove the value and its documentation after the minimum supported direct-upgrade chart includes localEndpointHost.autoInject and no longer renders the peer gate.
+- `rustfs-5416-wait-mode` startup wait-mode validation: releases before explicit local endpoint identity treat an unknown RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE value as auto. New servers retain that fallback only when no explicit local endpoint host is configured; an anchor requires a recognized mode so a typo cannot bypass DNS locality. Remove the fallback and reject every unknown value after every supported direct-upgrade chart validates this setting before rollout.
+- `rustfs-5416-kubernetes-alias-dns` Kubernetes endpoint identity fallback: deployments created before explicit local endpoint identity may use resolvable aliases that do not match the Pod hostname. An implicit auto-mode zero match retains legacy DNS locality with a bounded deadline, while ambiguous matches and invalid explicit anchors still fail closed. Remove the fallback after every supported direct-upgrade chart and deployment manifest provides a canonical RUSTFS_LOCAL_ENDPOINT_HOST for domain-based distributed topologies.
+- `rustfs-5416-zero-retry-delay` startup retry-delay validation: releases before bounded topology convergence accept RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY values of 0 or 0ms. New servers replace those values with the safe nonzero default so a direct upgrade neither fails startup nor enters a busy loop. Reject zero after the minimum supported direct-upgrade release validates or rewrites this setting before rollout.
 - `scanner-usage-v2` persisted scanner usage migration: pre-v2 scanners write `.usage.json`, so upgraded clusters read that primary/backup pair only while `.usage.v2.json` is absent and continue removing deleted buckets from legacy copies that still exist. The additive usage_snapshot_complete field in `.usage.v2.json` must remain optional while mixed-version clusters are supported; a missing field means the snapshot is not authoritative. Remove the legacy object fallback and cleanup only after every supported direct-upgrade source writes `.usage.v2.json`.
 - `ns-scanner-rpc-v3` namespace scanner capability and activity handshake: old peers and legacy internode transports lack the authenticated startup-epoch handshake. The oldest peers send an empty activity request and receive a field-empty protocol-0 response. Protocol v4 binds the challenge and response topology but cannot authenticate distributed dirty-usage state. Current protocol v5 binds the request version, acknowledgement target and generation, and the response dirty-usage state. Servers retain protocol-0 and protocol-v4 codecs for rolling upgrades, while the distributed scanner publishes usage only after every peer returns authenticated protocol v5 state. Scanner selection treats HTTP 404/405/426 and the legacy MethodNotAllowed default as an explicit lack of remote scanner v3 support and assigns those disks to coordinator-driven workers; transient capability failures remain incomplete and do not activate the fallback. Remove the coordinator fallback after the minimum supported RustFS peer version implements namespace scanner protocol v3, remove protocol-0 activity requests and responses after every supported peer implements authenticated scanner activity protocol v4, and remove the protocol-v4 activity codec after every supported peer implements protocol v5; future protocol revisions must keep the same dual-version server/codec window before changing the advertised version.
 - `#4648` walk-dir stream completion capability: old clients can append fallback output to an already-used metacache writer after a terminal body error, so servers emit terminal walk errors only to clients that sign the `walk_dir_stream_completion=error-v1` query capability and its request-body digest. Remove the legacy clean-EOF path after the minimum supported RustFS peer version always advertises this capability.

docs/testing/ecstore-validation-suite-design.md

@@ -358,7 +358,7 @@ Fixture-backed tests should run when the fixture path is present:
 
 ```bash
 cargo test -p rustfs-ecstore --test legacy_bitrot_read_test -- --nocapture
-cargo test -p rustfs-ecstore --features rio-v2 --test minio_generated_read_test -- --ignored --nocapture
+cargo test -p rustfs --features rio-v2 storage::minio_generated_read_test --lib -- --ignored --nocapture
 ```
 
 ## Multi-Expert Adversarial Review Summary

flake.nix

@@ -59,7 +59,7 @@
         {
           default = rustPlatform.buildRustPackage {
             pname = "rustfs";
-            version = "1.0.0-beta.11";
+            version = "1.0.0-beta.12";
 
             src = ./.;
 

helm/README.md

@@ -47,6 +47,60 @@ without manual intervention:
 If you previously set `replicaCount=16` and now want a different topology,
 set both `replicaCount` and `drivesPerNode` explicitly.
 
+For distributed deployments that use the chart-generated
+`RUSTFS_VOLUMES`, `localEndpointHost.autoInject` defaults to automatic
+selection. Without `secret.existingSecret`, the chart injects a private
+Downward API variable, `RUSTFS_CHART_POD_NAME`, and uses it to build the pod's
+fully qualified hostname in
+`RUSTFS_LOCAL_ENDPOINT_HOST`. RustFS can then identify local drives without
+waiting for every peer's DNS record or TCP listener. A user-defined `POD_NAME`
+is preserved and does not interfere with the private chart variable. Setting
+`RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE` explicitly to `bounded`, `fail-fast`,
+`failfast`, or `strict` also keeps legacy DNS-based locality discovery. A
+dynamically sourced wait mode keeps the legacy path because its value cannot be
+validated while rendering. Otherwise, Kubernetes auto-detection selects
+orchestrated startup when RustFS consumes the generated anchor.
+
+An existing Secret is opaque to the chart and may historically contain more
+than credentials, so the chart does not inject an anchor whenever
+`secret.existingSecret` is set. In Kubernetes auto/orchestrated mode, RustFS
+then derives a DNS-free identity from the kernel hostname when exactly one
+domain endpoint at the server port has the same full hostname or first label.
+All-IP topologies retain direct IP locality detection. A domain topology with
+zero matches retains legacy DNS locality discovery; implicit auto mode bounds
+that compatibility path by `RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT` (180 seconds
+by default). Multiple matching candidates remain an error. Set
+`RUSTFS_LOCAL_ENDPOINT_HOST` explicitly to avoid DNS discovery. For a
+credentials-only Secret, set
+`localEndpointHost.autoInject=true` to add the chart anchor without changing
+the historical ConfigMap-then-Secret `envFrom` precedence. If injection is
+explicitly enabled with an incompatible hidden `RUSTFS_VOLUMES`,
+`RUSTFS_ADDRESS`, or `RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE`, RustFS also fails
+during endpoint construction; it does not silently fall back to a different
+topology.
+
+When `config.rustfs.volumes` is set explicitly, the chart does not infer a
+local endpoint identity. RustFS applies the same kernel-hostname inference to
+custom domain topologies in Kubernetes auto/orchestrated mode; aliases that do
+not match the Pod hostname retain legacy DNS locality, with the bounded auto
+fallback described above. They may provide `RUSTFS_LOCAL_ENDPOINT_HOST`
+through `extraEnv` for DNS-free startup. An explicit `RUSTFS_VOLUMES`, explicit
+`RUSTFS_LOCAL_ENDPOINT_HOST`, bounded/dynamic or unrecognized startup mode, or
+`localEndpointHost.autoInject=false`, also disables chart injection. A
+`RUSTFS_ADDRESS` override alone does not disable it; the effective address and
+generated topology must agree on the endpoint port. Custom anchor-based
+configurations must resolve to `orchestrated` startup mode and must not receive
+a conflicting mode from an `envFrom` source.
+`startupWaitTimeoutSeconds` is retained for values-file compatibility but is
+deprecated and ignored.
+Historical `RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY` values of `0` or `0ms`
+are replaced with the safe default retry cap instead of causing a busy loop or
+blocking a direct upgrade.
+
+Upgrade the chart and RustFS image together. An older image that does not
+recognize `RUSTFS_LOCAL_ENDPOINT_HOST` retains its previous DNS-based startup
+behavior.
+
 ---
 
 # Parameters Overview
@@ -57,6 +111,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
 | affinity.podAntiAffinity.enabled | bool | `true` |  |
 | affinity.podAntiAffinity.topologyKey | string | `"kubernetes.io/hostname"` |  |
 | clusterDomain | string | `"cluster.local"` | Kubernetes cluster DNS domain used to build in-cluster FQDNs for `RUSTFS_VOLUMES` (distributed mode) and mTLS server certificate SANs. Override for clusters not using the default `cluster.local`. Provide the DNS root only, without a `svc.` prefix or leading/trailing dots. |
+| localEndpointHost.autoInject | bool or null | `null` | Automatically inject `RUSTFS_LOCAL_ENDPOINT_HOST` for chart-generated distributed topologies unless `secret.existingSecret` is set. Use `true` for a credentials-only existing Secret or `false` to preserve legacy DNS locality explicitly. |
 | commonLabels | object | `{}` | Labels to add to all deployed objects. |
 | config.rustfs.address | string | `":9000"` |  |
 | config.rustfs.console_address | string | `":9001"` |  |
@@ -66,7 +121,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
 | config.rustfs.obs_environment | string | `"development"` |  |
 | config.rustfs.obs_log_directory | string | `"/logs"` | Log directory inside the RustFS container. Set to `""` to disable log PVCs and mounts. |
 | config.rustfs.region | string | `"us-east-1"` |  |
-| config.rustfs.volumes | string | `""` |  |
+| config.rustfs.volumes | string | `""` | Explicit distributed volume topology. When empty, the chart generates the topology and normally injects `RUSTFS_LOCAL_ENDPOINT_HOST`; custom topologies must configure local endpoint identity explicitly when needed. |
 | config.rustfs.log_rotation.size | int | `"100"` | Default log rotation size mb for rustfs. |
 | config.rustfs.log_rotation.time | string | `"hour"` | Default log rotation time for rustfs. |
 | config.rustfs.log_rotation.keep_files | int | `"30"` | Default log keep files for rustfs.  |
@@ -107,7 +162,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
 | config.rustfs.kms.vault.vault_token | string | `""`| The vault token. Rendered into a dedicated Secret (`<fullname>-kms-secret`), never into the ConfigMap. |
 | config.rustfs.kms.vault.vault_mount_path | string | `"transit"`| The vault mount path, only works if `vault_backend` equals `vault-transit` . |
 | config.rustfs.kms.vault.default_key | string | `"transit"`| The master key id for RustFS. |
-| extraEnv | map | `[]` |  Extra environment variables for RustFS container. |
+| extraEnv | list | `[]` | Extra environment variables for the RustFS container. An explicit `RUSTFS_LOCAL_ENDPOINT_HOST` or `RUSTFS_VOLUMES`, or a bounded, dynamic, or unrecognized startup mode, disables generated anchor injection. `POD_NAME` and `RUSTFS_ADDRESS` remain independent overrides. |
 | extraVolumes | list | `[]` | Extra volumes to add to the pod spec. Supported in both standalone (Deployment) and distributed (StatefulSet) modes. |
 | extraVolumeMounts | list | `[]` | Extra volume mounts to add to the RustFS container. Supported in both standalone (Deployment) and distributed (StatefulSet) modes. |
 | containerSecurityContext.capabilities.drop[0] | string | `"ALL"` |  |
@@ -190,7 +245,7 @@ uer. `ClusterIssuer` or `Issuer`. |
 | resources.limits.memory | string | `"512Mi"` |  |
 | resources.requests.cpu | string | `"100m"` |  |
 | resources.requests.memory | string | `"128Mi"` |  |
-| secret.existingSecret | string | `""` | Use existing secret with a credentials. |
+| secret.existingSecret | string | `""` | Use an existing Secret. Automatic endpoint-anchor injection is disabled because the Secret is opaque; set `localEndpointHost.autoInject=true` only after confirming it contains credentials rather than runtime topology, address, or startup-mode overrides. |
 | secret.rustfs.access_key | string | `"rustfsadmin"` | RustFS Access Key ID |
 | secret.rustfs.secret_key | string | `"rustfsadmin"` | RustFS Secret Key ID |
 | service.type | string | `"ClusterIP"` |  |
@@ -202,6 +257,7 @@ uer. `ClusterIssuer` or `Issuer`. |
 | serviceAccount.automount | bool | `true` |  |
 | serviceAccount.create | bool | `true` |  |
 | serviceAccount.name | string | `""` |  |
+| startupWaitTimeoutSeconds | int | `300` | Deprecated and ignored; retained for values-file compatibility. |
 | storageclass.dataStorageSize | string | `"256Mi"` | The storage size for data PVC. |
 | storageclass.logStorageSize | string | `"256Mi"` | The storage size for logs PVC. |
 | storageclass.name | string | `"local-path"` | The name for StorageClass. |
@@ -291,20 +347,17 @@ Notes:
 * **Pools are append-only.** The list index determines the StatefulSet name —
   never remove or reorder entries. Retire a pool with
   `rc admin decommission` before removing it from the list.
-* During the expansion rollout, pods restart until every pod of every pool is
-  resolvable — the server refuses to start with unresolvable peers, so expect
-  a few crash/restart cycles before the cluster converges. This is harmless.
+* With chart-generated volumes, each pod receives an explicit local endpoint
+  identity. An unavailable peer no longer blocks a pod from reaching RustFS's
+  own startup and quorum checks.
 * After the cluster converges, run `rc admin rebalance start <alias>` to
   spread existing objects across the new pool.
 * Pod anti-affinity in pool mode is scoped per pool and **preferred**
   (soft), not required: two pools can share nodes, and each pool's own pods
-  spread across distinct nodes when capacity allows. Soft affinity is
-  load-bearing for in-place expansion — with required rules, the
-  not-yet-rolled pods of the existing pool block the new pool's pods from
-  their nodes while the rolled pods crash on the unresolvable (Pending)
-  peers, deadlocking the rollout on any cluster with fewer nodes than the
-  total pod count. Single-pool deployments (`pools.enabled=false`) keep the
-  chart's existing required anti-affinity unchanged.
+  spread across distinct nodes when capacity allows. Preferred affinity keeps
+  additional pools schedulable when the cluster has fewer nodes than total
+  pods. Single-pool deployments (`pools.enabled=false`) keep the chart's
+  existing required anti-affinity unchanged.
 * The PodDisruptionBudget spans all pools: with the default
   `pdb.maxUnavailable: 1`, at most one pod of the whole cluster may be
   evicted at a time. This is deliberately conservative — quorum safety
@@ -315,7 +368,8 @@ Notes:
 ## Requirement
 
 * Helm V3
-* RustFS >= 1.0.0-alpha.69
+* The RustFS image from the same release as the chart. If `image.rustfs.tag`
+  is overridden, that image must support `RUSTFS_LOCAL_ENDPOINT_HOST`.
 
 Due to the traefik and ingress has different session sticky/affinity annotations, and rustfs support both those two controller, you should specify parameter `ingress.className` to select the right one which suits for you.
 

helm/rustfs/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: rustfs
 description: RustFS helm chart to deploy RustFS on kubernetes cluster.
 type: application
-version: "0.11.0"
-appVersion: "1.0.0-beta.11"
+version: "0.12.0"
+appVersion: "1.0.0-beta.12"
 home: https://rustfs.com
 icon: https://media.sys.truenas.net/apps/rustfs/icons/icon.svg
 maintainers:

helm/rustfs/templates/_helpers.tpl

@@ -322,21 +322,23 @@ Render RUSTFS_SERVER_DOMAINS
 {{- join "," $domains -}}
 {{- end -}}
 
-{{/* Render probe command for liveness and readiness
+{{/* Render an mTLS probe command
 */}}
 
 {{- define "rustfs.probeCommand" -}}
-{{- $endpoint_port := .Values.service.endpoint.port | default 9000 -}}
-{{- $console_port := .Values.service.console.port | default 9001 -}}
+{{- $root := .root -}}
+{{- $endpointPath := .endpointPath -}}
+{{- $endpoint_port := $root.Values.service.endpoint.port | default 9000 -}}
+{{- $console_port := $root.Values.service.console.port | default 9001 -}}
 {{- $args := "-skf" -}}
 
-{{- if and .Values.mtls.enabled -}}
-  {{- $args = printf "%s --cert %s --key %s" $args .Values.mtls.clientCertPath .Values.mtls.clientKeyPath -}}
+{{- if and $root.Values.mtls.enabled -}}
+  {{- $args = printf "%s --cert %s --key %s" $args $root.Values.mtls.clientCertPath $root.Values.mtls.clientKeyPath -}}
 {{- end -}}
 - /bin/sh
 - -c
 - |
-  curl {{ $args }} https://127.0.0.1:{{ $endpoint_port }}/health/ready && \
+  curl {{ $args }} https://127.0.0.1:{{ $endpoint_port }}{{ $endpointPath }} && \
   curl {{ $args }} https://127.0.0.1:{{ $console_port }}/rustfs/console/health
 {{- end -}}
 
@@ -350,7 +352,7 @@ livenessProbe:
   {{- if .Values.mtls.enabled }}
   exec:
     command:
-{{ include "rustfs.probeCommand" . | nindent 6 }}
+{{ include "rustfs.probeCommand" (dict "root" . "endpointPath" "/health") | nindent 6 }}
   {{- else }}
   httpGet:
     path: /health
@@ -369,7 +371,7 @@ readinessProbe:
   {{- if .Values.mtls.enabled }}
   exec:
     command:
-{{ include "rustfs.probeCommand" . | nindent 6 }}
+{{ include "rustfs.probeCommand" (dict "root" . "endpointPath" "/health/ready") | nindent 6 }}
   {{- else }}
   httpGet:
     path: /health/ready

helm/rustfs/templates/statefulset.yaml

@@ -1,12 +1,37 @@
 {{- $logDir := .Values.config.rustfs.obs_log_directory }}
 {{- $logDirEnabled := ne $logDir "" }}
 {{- $poolsEnabled := and .Values.pools .Values.pools.enabled }}
+{{- $generatedVolumes := empty .Values.config.rustfs.volumes }}
+{{- $localEndpointHostAutoInject := empty .Values.secret.existingSecret }}
+{{- with .Values.localEndpointHost }}
+{{- if and (hasKey . "autoInject") (ne .autoInject nil) }}
+{{- if not (kindIs "bool" .autoInject) }}
+{{- fail "localEndpointHost.autoInject must be a boolean or null" }}
+{{- end }}
+{{- $localEndpointHostAutoInject = .autoInject }}
+{{- end }}
+{{- end }}
+{{- $injectLocalEndpointHost := and $generatedVolumes $localEndpointHostAutoInject }}
+{{- range $env := .Values.extraEnv -}}
+{{- $name := default "" $env.name -}}
+{{- if eq $name "RUSTFS_VOLUMES" -}}
+{{- $injectLocalEndpointHost = false -}}
+{{- else if eq $name "RUSTFS_LOCAL_ENDPOINT_HOST" -}}
+{{- $injectLocalEndpointHost = false -}}
+{{- else if eq $name "RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE" -}}
+{{- if hasKey $env "value" -}}
+{{- $mode := lower (trim (toString $env.value)) -}}
+{{- if and (ne $mode "") (ne $mode "auto") (ne $mode "orchestrated") -}}
+{{- $injectLocalEndpointHost = false -}}
+{{- end -}}
+{{- else -}}
+{{- $injectLocalEndpointHost = false -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
 {{- if and .Values.mode.distributed.enabled (le (int .Values.replicaCount) 1) -}}
 {{- fail "Distributed mode requires replicaCount >= 2" -}}
 {{- end -}}
-{{- if and .Values.mode.distributed.enabled (le (int .Values.startupWaitTimeoutSeconds) 0) -}}
-{{- fail "startupWaitTimeoutSeconds must be greater than 0 in distributed mode" -}}
-{{- end -}}
 
 {{- if .Values.mode.distributed.enabled }}
 {{- $pools := include "rustfs.pools" . | fromJsonArray }}
@@ -74,13 +99,10 @@ spec:
         {{- if $.Values.affinity.podAntiAffinity.enabled }}
         podAntiAffinity:
           {{- if $poolsEnabled }}
-          {{- /* Pool-scoped and PREFERRED, not required: during in-place
-                 expansion the not-yet-rolled pods' required rules block new
-                 pods from their nodes, while the new pods' DNS must resolve
-                 before the roll can proceed - required rules deadlock
-                 expansion whenever the cluster has fewer nodes than total
-                 pods. Soft anti-affinity converges (the scheduler still
-                 spreads when capacity allows). */}}
+          {{- /* Pool-scoped and PREFERRED, not required: additional pools
+                 must remain schedulable when the cluster has fewer nodes
+                 than total pods. The scheduler still spreads them when
+                 capacity allows. */}}
           preferredDuringSchedulingIgnoredDuringExecution:
             - weight: 100
               podAffinityTerm:
@@ -127,10 +149,6 @@ spec:
           env:
             - name: DRIVES_PER_NODE
               value: {{ $drivesPerNode | quote }}
-            - name: ENDPOINT_PORT
-              value: {{ $.Values.service.endpoint.port | quote }}
-            - name: STARTUP_WAIT_TIMEOUT_SECONDS
-              value: {{ $.Values.startupWaitTimeoutSeconds | quote }}
           command:
             - sh
             - -c
@@ -146,44 +164,6 @@ spec:
               mkdir -p /mnt/rustfs/logs
               chmod 755 /mnt/rustfs/logs
               {{- end }}
-              deadline=$(($(date +%s) + STARTUP_WAIT_TIMEOUT_SECONDS))
-              wait_for_peer() {
-                description=$1
-                shift
-                until "$@" >/dev/null 2>&1; do
-                  if [ "$(date +%s)" -ge "$deadline" ]; then
-                    echo "Timed out after ${STARTUP_WAIT_TIMEOUT_SECONDS}s waiting for ${description}" >&2
-                    exit 1
-                  fi
-                  sleep 2
-                done
-              }
-
-              echo "Waiting for distributed peer DNS records"
-              {{- range $peerPool := $pools }}
-              {{- range $i := until (int $peerPool.replicaCount) }}
-              wait_for_peer "DNS for {{ $peerPool.fullname }}-{{ $i }}" \
-                nslookup "{{ $peerPool.fullname }}-{{ $i }}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}"
-              {{- end }}
-              {{- end }}
-
-              ordinal=${HOSTNAME##*-}
-              echo "Waiting for earlier distributed peers to listen on port ${ENDPOINT_PORT}"
-              {{- range $peerPool := $pools }}
-              {{- if lt (int $peerPool.index) (int $pool.index) }}
-              {{- range $i := until (int $peerPool.replicaCount) }}
-              wait_for_peer "{{ $peerPool.fullname }}-{{ $i }}:${ENDPOINT_PORT}" \
-                nc -z -w 2 "{{ $peerPool.fullname }}-{{ $i }}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}" "$ENDPOINT_PORT"
-              {{- end }}
-              {{- else if eq (int $peerPool.index) (int $pool.index) }}
-              i=0
-              while [ "$i" -lt "$ordinal" ]; do
-                wait_for_peer "{{ $peerPool.fullname }}-${i}:${ENDPOINT_PORT}" \
-                  nc -z -w 2 "{{ $peerPool.fullname }}-${i}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}" "$ENDPOINT_PORT"
-                i=$((i + 1))
-              done
-              {{- end }}
-              {{- end }}
           volumeMounts:
             {{- if gt $drivesPerNode 1 }}
             {{- range $i := until $drivesPerNode }}
@@ -210,9 +190,19 @@ spec:
               containerPort: {{ $.Values.service.endpoint.port }}
             - name: console
               containerPort: {{ $.Values.service.console.port }}
-          {{- with $.Values.extraEnv }}
+          {{- if or $injectLocalEndpointHost (not (empty $.Values.extraEnv)) }}
           env:
+            {{- if $injectLocalEndpointHost }}
+            - name: RUSTFS_CHART_POD_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+            - name: RUSTFS_LOCAL_ENDPOINT_HOST
+              value: {{ printf "$(RUSTFS_CHART_POD_NAME).%s-headless.%s.svc.%s" (include "rustfs.fullname" $) $.Release.Namespace (include "rustfs.clusterDomain" $) | quote }}
+            {{- end }}
+            {{- with $.Values.extraEnv }}
             {{- toYaml . | nindent 12 }}
+            {{- end }}
           {{- end }}
           envFrom:
             - configMapRef:

helm/rustfs/values.yaml

@@ -30,11 +30,22 @@ drivesPerNode: null
 # Provide the DNS root only, without a `svc.` prefix or leading/trailing dots.
 clusterDomain: cluster.local
 
-# Maximum time an init container waits for distributed peers to publish DNS
-# and for earlier peers to start listening. The wait is shared across all
-# peers, not applied once per peer.
+# Deprecated and ignored. Retained so existing values files continue to
+# render after peer startup coordination moved into RustFS.
+# RUSTFS_COMPAT_TODO(rustfs-5416): Keep the removed init-gate setting visible to existing values files. Remove after the minimum supported direct-upgrade chart includes localEndpointHost.autoInject and no longer renders peer gates.
 startupWaitTimeoutSeconds: 300
 
+# Inject a pod-specific endpoint identity for chart-generated distributed
+# topologies. The default enables injection unless secret.existingSecret is
+# set, because the chart cannot inspect runtime overrides hidden in that
+# Secret. Without injection, the server infers a unique matching domain
+# endpoint from its kernel hostname in Kubernetes auto/orchestrated mode; custom
+# aliases retain bounded legacy DNS locality when no hostname matches. Configure
+# RUSTFS_LOCAL_ENDPOINT_HOST explicitly for DNS-free startup. Set true for a
+# credentials-only existing Secret or false to suppress chart injection.
+localEndpointHost:
+  autoInject: null
+
 # This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
 image:
   rustfs: # This sets the rustfs image repository and tag.
@@ -223,7 +234,18 @@ config:
         default_key: ""
 
 
-extraEnv: [] # This is for setting extra environment variables in the rustfs container. It should be a list of key value pairs. For example:
+# Extra environment variables for the RustFS container.
+# In distributed mode with chart-generated volumes (config.rustfs.volumes is
+# empty), the chart injects a private Downward API pod name and
+# RUSTFS_LOCAL_ENDPOINT_HOST to identify the pod's local endpoints. An explicit
+# RUSTFS_LOCAL_ENDPOINT_HOST or RUSTFS_VOLUMES entry disables automatic anchor
+# injection. An explicit bounded or fail-fast
+# RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE, or an unrecognized static value, also keeps
+# legacy DNS-based locality discovery. Otherwise Kubernetes auto-detection
+# selects orchestrated startup when RustFS consumes the generated anchor.
+# With custom volumes, the chart does not inject them and extraEnv may define
+# the local endpoint identity explicitly.
+extraEnv: []
 # extraEnv:
 #   - name: RUSTFS_ERASURE_SET_DRIVE_COUNT
 #     value: "16"

rustfs.spec

@@ -1,9 +1,9 @@
 %global _enable_debug_packages 0
 %global _empty_manifest_terminate_build 0
-%global prerelease beta.11
+%global prerelease beta.12
 Name:           rustfs
 Version:        1.0.0
-Release:        beta.11
+Release:        beta.12
 Summary:       High-performance distributed object storage for MinIO alternative
 
 License:        Apache-2.0
@@ -58,6 +58,9 @@ install %_builddir/%{name}-%{version}-%{prerelease}/target/%_arch/%_arch-unknown
 %_bindir/rustfs
 
 %changelog
+* Thu Jul 30 2026 overtrue <anzhengchao@gmail.com>
+- Update RPM package to RustFS 1.0.0-beta.12
+
 * Thu Jul 23 2026 overtrue <anzhengchao@gmail.com>
 - Update RPM package to RustFS 1.0.0-beta.11
 

rustfs/src/admin/handlers/kms_dynamic.rs

@@ -393,36 +393,27 @@ impl Operation for ConfigureKmsHandler {
         // Convert request to KmsConfig
         let kms_config = configure_request.to_kms_config();
 
-        // Configure the service
-        let (success, message, status) = match service_manager.configure(kms_config.clone()).await {
+        let persisted_config = kms_config.clone();
+        let (success, message, status) = match service_manager
+            .configure_with_persistence(kms_config, || async move {
+                save_kms_config(&persisted_config)
+                    .await
+                    .map_err(|error| rustfs_kms::KmsError::backend_error(format!("Failed to persist KMS configuration: {error}")))
+            })
+            .await
+        {
             Ok(()) => {
-                // Persist the configuration to cluster storage
-                if let Err(e) = save_kms_config(&kms_config).await {
-                    let error_msg = format!("KMS configured in memory but failed to persist: {e}");
-                    error!(
-                        component = LOG_COMPONENT_ADMIN,
-                        subsystem = LOG_SUBSYSTEM_KMS,
-                        event = "kms_service_state",
-                        operation = "configure",
-                        state = "persist_failed",
-                        error = %e,
-                        "admin kms dynamic state"
-                    );
-                    let status = service_manager.get_status().await;
-                    (false, error_msg, status)
-                } else {
-                    let status = service_manager.get_status().await;
-                    info!(
-                        component = LOG_COMPONENT_ADMIN,
-                        subsystem = LOG_SUBSYSTEM_KMS,
-                        event = "kms_service_state",
-                        operation = "configure",
-                        state = "configured",
-                        status = ?status,
-                        "admin kms dynamic state"
-                    );
-                    (true, "KMS configured successfully".to_string(), status)
-                }
+                let status = service_manager.get_status().await;
+                info!(
+                    component = LOG_COMPONENT_ADMIN,
+                    subsystem = LOG_SUBSYSTEM_KMS,
+                    event = "kms_service_state",
+                    operation = "configure",
+                    state = "configured",
+                    status = ?status,
+                    "admin kms dynamic state"
+                );
+                (true, "KMS configured successfully".to_string(), status)
             }
             Err(e) => {
                 let error_msg = format!("Failed to configure KMS: {e}");
@@ -529,125 +520,61 @@ impl Operation for StartKmsHandler {
         );
 
         let service_manager = kms_service_manager_from_context();
-
-        // Check if already running and force flag
-        let current_status = service_manager.get_status().await;
-        if matches!(current_status, KmsServiceStatus::Running) && !start_request.force.unwrap_or(false) {
-            warn!(
-                component = LOG_COMPONENT_ADMIN,
-                subsystem = LOG_SUBSYSTEM_KMS,
-                event = "kms_service_state",
-                operation = "start",
-                state = "already_running",
-                "admin kms dynamic state"
-            );
-            let response = StartKmsResponse {
-                success: false,
-                message: "KMS service is already running. Use force=true to restart.".to_string(),
-                status: current_status,
-            };
-            let json_response = match serde_json::to_string(&response) {
-                Ok(json) => json,
-                Err(e) => {
-                    error!(
-                        component = LOG_COMPONENT_ADMIN,
-                        subsystem = LOG_SUBSYSTEM_KMS,
-                        event = EVENT_ADMIN_KMS_DYNAMIC_STATE,
-                        operation = "start",
-                        result = "response_serialize_failed",
-                        error = %e,
-                        "admin kms dynamic state"
-                    );
-                    return Ok(S3Response::new((
-                        StatusCode::INTERNAL_SERVER_ERROR,
-                        Body::from("Serialization error".to_string()),
-                    )));
-                }
-            };
-            return Ok(S3Response::new((StatusCode::OK, Body::from(json_response))));
-        }
-
-        // Start the service (or restart if force=true)
-        let (success, message, status) =
-            if start_request.force.unwrap_or(false) && matches!(current_status, KmsServiceStatus::Running) {
-                // Force restart
-                match service_manager.stop().await {
-                    Ok(()) => match service_manager.start().await {
-                        Ok(()) => {
-                            let status = service_manager.get_status().await;
-                            info!(
-                                component = LOG_COMPONENT_ADMIN,
-                                subsystem = LOG_SUBSYSTEM_KMS,
-                                event = "kms_service_state",
-                                operation = "restart",
-                                state = "running",
-                                status = ?status,
-                                "admin kms dynamic state"
-                            );
-                            (true, "KMS service restarted successfully".to_string(), status)
-                        }
-                        Err(e) => {
-                            let error_msg = format!("Failed to restart KMS service: {e}");
-                            error!(
-                                component = LOG_COMPONENT_ADMIN,
-                                subsystem = LOG_SUBSYSTEM_KMS,
-                                event = "kms_service_state",
-                                operation = "restart",
-                                state = "start_failed",
-                                error = %e,
-                                "admin kms dynamic state"
-                            );
-                            let status = service_manager.get_status().await;
-                            (false, error_msg, status)
-                        }
-                    },
-                    Err(e) => {
-                        let error_msg = format!("Failed to stop KMS service for restart: {e}");
-                        error!(
-                            component = LOG_COMPONENT_ADMIN,
-                            subsystem = LOG_SUBSYSTEM_KMS,
-                            event = "kms_service_state",
-                            operation = "restart",
-                            state = "stop_failed",
-                            error = %e,
-                            "admin kms dynamic state"
-                        );
-                        let status = service_manager.get_status().await;
-                        (false, error_msg, status)
-                    }
-                }
-            } else {
-                // Normal start
-                match service_manager.start().await {
-                    Ok(()) => {
-                        let status = service_manager.get_status().await;
-                        info!(
-                            component = LOG_COMPONENT_ADMIN,
-                            subsystem = LOG_SUBSYSTEM_KMS,
-                            event = "kms_service_state",
-                            operation = "start",
-                            state = "running",
-                            status = ?status,
-                            "admin kms dynamic state"
-                        );
-                        (true, "KMS service started successfully".to_string(), status)
-                    }
-                    Err(e) => {
-                        let error_msg = format!("Failed to start KMS service: {e}");
-                        error!(
-                            component = LOG_COMPONENT_ADMIN,
-                            subsystem = LOG_SUBSYSTEM_KMS,
-                            event = "kms_service_state",
-                            operation = "start",
-                            state = "start_failed",
-                            error = %e,
-                            "admin kms dynamic state"
-                        );
-                        let status = service_manager.get_status().await;
-                        (false, error_msg, status)
-                    }
-                }
-            };
+        let force = start_request.force.unwrap_or(false);
+        let (success, message, status) = match service_manager.start_or_restart(force).await {
+            Ok(rustfs_kms::KmsStartOutcome::Started) => {
+                let status = service_manager.get_status().await;
+                info!(
+                    component = LOG_COMPONENT_ADMIN,
+                    subsystem = LOG_SUBSYSTEM_KMS,
+                    event = "kms_service_state",
+                    operation = "start",
+                    state = "running",
+                    status = ?status,
+                    "admin kms dynamic state"
+                );
+                (true, "KMS service started successfully".to_string(), status)
+            }
+            Ok(rustfs_kms::KmsStartOutcome::Restarted) => {
+                let status = service_manager.get_status().await;
+                info!(
+                    component = LOG_COMPONENT_ADMIN,
+                    subsystem = LOG_SUBSYSTEM_KMS,
+                    event = "kms_service_state",
+                    operation = "restart",
+                    state = "running",
+                    status = ?status,
+                    "admin kms dynamic state"
+                );
+                (true, "KMS service restarted successfully".to_string(), status)
+            }
+            Ok(rustfs_kms::KmsStartOutcome::AlreadyRunning) => {
+                let status = service_manager.get_status().await;
+                warn!(
+                    component = LOG_COMPONENT_ADMIN,
+                    subsystem = LOG_SUBSYSTEM_KMS,
+                    event = "kms_service_state",
+                    operation = "start",
+                    state = "already_running",
+                    "admin kms dynamic state"
+                );
+                (false, "KMS service is already running. Use force=true to restart.".to_string(), status)
+            }
+            Err(e) => {
+                let error_msg = format!("Failed to start or restart KMS service: {e}");
+                error!(
+                    component = LOG_COMPONENT_ADMIN,
+                    subsystem = LOG_SUBSYSTEM_KMS,
+                    event = "kms_service_state",
+                    operation = "start",
+                    state = "start_failed",
+                    error = %e,
+                    "admin kms dynamic state"
+                );
+                let status = service_manager.get_status().await;
+                (false, error_msg, status)
+            }
+        };
 
         let response = StartKmsResponse {
             success,
@@ -804,8 +731,7 @@ impl Operation for GetKmsStatusHandler {
 
         let service_manager = kms_service_manager_from_context();
 
-        let status = service_manager.get_status().await;
-        let config = service_manager.get_redacted_config().await;
+        let (status, config) = service_manager.get_redacted_state().await;
 
         // Get backend type and health status
         let backend_type = config.as_ref().map(|c| c.backend.clone());
@@ -938,36 +864,27 @@ impl Operation for ReconfigureKmsHandler {
         // Convert request to KmsConfig
         let kms_config = configure_request.to_kms_config();
 
-        // Reconfigure the service (stops, reconfigures, and starts)
-        let (success, message, status) = match service_manager.reconfigure(kms_config.clone()).await {
+        let persisted_config = kms_config.clone();
+        let (success, message, status) = match service_manager
+            .reconfigure_with_persistence(kms_config, || async move {
+                save_kms_config(&persisted_config)
+                    .await
+                    .map_err(|error| rustfs_kms::KmsError::backend_error(format!("Failed to persist KMS configuration: {error}")))
+            })
+            .await
+        {
             Ok(()) => {
-                // Persist the configuration to cluster storage
-                if let Err(e) = save_kms_config(&kms_config).await {
-                    let error_msg = format!("KMS reconfigured in memory but failed to persist: {e}");
-                    error!(
-                        component = LOG_COMPONENT_ADMIN,
-                        subsystem = LOG_SUBSYSTEM_KMS,
-                        event = "kms_service_state",
-                        operation = "reconfigure",
-                        state = "persist_failed",
-                        error = %e,
-                        "admin kms dynamic state"
-                    );
-                    let status = service_manager.get_status().await;
-                    (false, error_msg, status)
-                } else {
-                    let status = service_manager.get_status().await;
-                    info!(
-                        component = LOG_COMPONENT_ADMIN,
-                        subsystem = LOG_SUBSYSTEM_KMS,
-                        event = "kms_service_state",
-                        operation = "reconfigure",
-                        state = "reconfigured",
-                        status = ?status,
-                        "admin kms dynamic state"
-                    );
-                    (true, "KMS reconfigured and restarted successfully".to_string(), status)
-                }
+                let status = service_manager.get_status().await;
+                info!(
+                    component = LOG_COMPONENT_ADMIN,
+                    subsystem = LOG_SUBSYSTEM_KMS,
+                    event = "kms_service_state",
+                    operation = "reconfigure",
+                    state = "reconfigured",
+                    status = ?status,
+                    "admin kms dynamic state"
+                );
+                (true, "KMS reconfigured and restarted successfully".to_string(), status)
             }
             Err(e) => {
                 let error_msg = format!("Failed to reconfigure KMS: {e}");

rustfs/src/admin/handlers/policies.rs

@@ -148,7 +148,7 @@ impl Operation for ListCannedPolicies {
             return Err(s3_error!(InternalError, "iam is not initialized"));
         };
 
-        let policies = iam_store.list_polices(&query.bucket).await.map_err(|e| {
+        let policies = iam_store.list_policies(&query.bucket).await.map_err(|e| {
             warn!(
                 component = LOG_COMPONENT_ADMIN,
                 subsystem = LOG_SUBSYSTEM_POLICY,

rustfs/src/admin/handlers/replication.rs

@@ -179,20 +179,6 @@ impl RemoteTargetRequest {
             return Err(s3_error!(InvalidRequest, "credentials.secretKey is required"));
         }
 
-        for (unsupported, configured) in [
-            ("disableProxy", self.disable_proxy),
-            ("healthCheckDuration", self.health_check_duration != 0),
-            ("edge", self.edge),
-            ("edgeSyncBeforeExpiry", self.edge_sync_before_expiry),
-        ] {
-            if configured {
-                return Err(s3_error!(
-                    InvalidRequest,
-                    "remote target field {unsupported} is not supported by this RustFS version"
-                ));
-            }
-        }
-
         Ok(BucketTarget {
             source_bucket: self.source_bucket,
             endpoint: self.endpoint,
@@ -1218,27 +1204,6 @@ mod tests {
         assert!(err.to_string().contains("credentials.secretKey is required"));
     }
 
-    #[test]
-    fn remote_target_request_rejects_unimplemented_fields() {
-        for (field, value) in [
-            ("disableProxy", serde_json::json!(true)),
-            ("healthCheckDuration", serde_json::json!(5)),
-            ("edge", serde_json::json!(true)),
-            ("edgeSyncBeforeExpiry", serde_json::json!(true)),
-        ] {
-            let mut request = valid_remote_target_request();
-            request[field] = value;
-            let request: RemoteTargetRequest =
-                serde_json::from_value(request).expect("unsupported field should still deserialize");
-            let err = request
-                .into_bucket_target()
-                .expect_err("unimplemented remote target fields must not be persisted");
-
-            assert!(err.to_string().contains(field));
-            assert!(err.to_string().contains("not supported by this RustFS version"));
-        }
-    }
-
     #[test]
     fn remote_target_request_converts_to_bucket_target() {
         let target = serde_json::from_value::<RemoteTargetRequest>(valid_remote_target_request())