test(ci): accept top.disk through authenticated proxy

Merge the reviewed fix from pull request #8104.

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rustfs-release-publish
-description: "Run the end-to-end RustFS console gate, version bump, preview validation, human confirmation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
+description: "Run the RustFS console gate, source version bump, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
 ---
 # RustFS Release Publish (preview-validated pipeline)
 
@@ -18,16 +18,18 @@ Pipeline shape:
 ```
 check console main against its latest Release
   -> if ahead: publish console -> wait for Release asset + latest API
-  -> bump RustFS version files to <target> (final version, ONE commit) -> merge
+  -> bump Cargo.toml and Cargo.lock to <target> -> merge
   -> tag <preview-tag> at that commit -> CI green
   -> verify preview Release assets -> run binary locally + console checks
   -> validate with latest rc client
   -> report preview acceptance results -> STOP for explicit human confirmation
   -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
   -> CI deletes the <target>-preview.N Releases (tags kept)
+  -> verify published images -> publish Helm chart from the validated source
+  -> update installation references on main -> update rustfs.com announcement
 ```
 
-On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
+On preview validation failure: fix lands on main via normal PR (Cargo versions are already at `<target>`, no new source bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2. Installation references remain on the previous published deliverable throughout preview validation.
 
 ## Required inputs
 
@@ -56,19 +58,21 @@ Rules:
 
 - Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
 - The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
-- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
+- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
 - Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
 
 ## Hard rules
 
-- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
+- Before preview, bump only Cargo.toml (workspace package and internal dependency versions) and the corresponding workspace members in Cargo.lock, directly to `<target>`. Keep README installation examples, flake.nix, Chart.yaml, and rustfs.spec on the previous published deliverable until Phase 7. Never write a `-preview.N` suffix into these files; preview identifiers belong to validation tags and artifacts.
+- Publishing a tag alone does not make an installation target available. Verify the exact image manifest, Release assets, and source archive before advancing references that consume them. Helm packaging derives its versions from the final tag and waits for that image; it does not require an early Chart.yaml bump on main.
 - Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
 - Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
 - Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
-- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
+- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
 - When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
 - Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
-- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
+- Phases run in order; a failure blocks dependent steps. For a preview/source acceptance failure before the final tag exists, land the fix, verify Cargo still matches the confirmed target, and restart from Phase 2 with the next preview at the fixed commit. If main has advanced to another target, resolve that source/version mismatch before another preview.
+- Once the final tag exists, preserve its commit. Retry failed publication jobs for that exact tag; do not recreate the tag or restart preview on newer main. If a source change is required after final-tag publication, report the failure and obtain a new target version. Phase 7 failures resume only the failed follow-up after rechecking artifact availability and whether a newer release has superseded it.
 - Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
 - Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
 - If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
@@ -84,10 +88,10 @@ Rules:
 
 Read and complete [the Console gate](references/console-gate.md) before Phase 1. Verify the latest published Console asset and exact commit; if Console main is ahead, complete its release and asset verification first. A successful build alone does not satisfy this gate.
 
-## Phase 1 — Version bump to the final target (once)
+## Phase 1 — Source version bump to the final target (once)
 
-- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
-- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
+- If Cargo.toml and all workspace members in Cargo.lock already read `<target>` (e.g. this is a restart after a failed preview), verify both and skip the source bump. Installation versions intentionally differ during preview; do not treat them as leftovers. If a previous preparation advanced installation references to an unavailable target, restore those references to the verified published baseline before selecting the next preview commit.
+- Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
 - Get the PR merged into main. Record the resulting main commit:
 
 ```bash
@@ -106,7 +110,7 @@ git push origin "<preview-tag>"
 
 Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
 
-The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
+The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
 
 On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
 
@@ -116,7 +120,7 @@ On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first
 - Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
 - Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
 - Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
-- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
+- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
 
 ## Phases 4–5 — Local artifact, Console, and rc acceptance
 
@@ -130,7 +134,7 @@ Continue to Phase 6 only after a new user reply explicitly confirms the reported
 
 ## Phase 6 — Publish the final tag on the validated commit
 
-No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
+No second source version bump. The final tag goes on the exact commit the preview validated:
 
 ```bash
 git fetch origin --tags
@@ -145,6 +149,10 @@ git push origin "<target>"
 - Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
 - Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
 
+## Phase 7 — Installation references and website announcement
+
+Only after Phase 6 succeeds, complete [post-release updates](references/post-release-updates.md): align installation references on main, then update the existing top banner in `rustfs/rustfs.com`. A failed or incomplete publication leaves both on the previous available version. These follow-up commits never change `PREVIEW_HASH` or either release tag. Track their PR, merge, and deployment states separately from artifact publication; an open PR is not a live website update.
+
 ## Output contract
 
 Always report:
@@ -153,4 +161,5 @@ Always report:
 - Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
 - Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
 - Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
+- Post-release installation PR and verification results; website banner target, text, link, PR, and observed deployment state. Report any remaining merge authorization or failed deployment explicitly rather than claiming the banner is live.
 - Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-publish/references/post-release-updates.md

@@ -0,0 +1,32 @@
+# Post-release Installation and Website Updates
+
+Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase.
+
+## Installation references
+
+1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
+2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
+3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
+4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
+
+## rustfs.com top banner
+
+Use the existing announcement configuration from [website PR #112](https://github.com/rustfs/rustfs.com/pull/112). Do not add a second banner or redesign the header.
+
+1. In an isolated checkout of `rustfs/rustfs.com`, read its current `AGENTS.md`, PR template if present, `data/announcement.ts`, and the consuming header component. Search for an existing announcement PR for this target before creating another.
+2. Update `homeAnnouncement` in `data/announcement.ts` after the release and installation availability checks pass. Keep `enabled: true`, `badge: 'New'`, and the existing component. For a stable release, use:
+
+   ```ts
+   export const homeAnnouncement = {
+     enabled: true,
+     badge: 'New',
+     message: 'RustFS <target> is now available! Please',
+     linkText: 'upgrade and try',
+     href: '/download',
+   } as const
+   ```
+
+   Replace `<target>` with the exact published version. Reserve GA wording for the actual first stable release. For a final alpha/beta/rc tag, explicitly call it a prerelease; use its GitHub Release URL if `/download` does not expose that target. Never announce a `-preview.N` build here.
+3. Use the site's current package manager and required checks. For this config-only change, type-check and lint the changed file, build the site, and inspect the generated homepage for the version, visible link text, and destination. Confirm `/download` actually offers the announced version before using that link.
+4. Commit, push, and create or update the website PR when included in the authorized release scope. Include the published RustFS Release URL and checks run. Keep merge and production-deployment authority separate; reuse permission already given, but this reference grants none by itself.
+5. After an authorized merge/deployment, verify the live `https://rustfs.com/` banner and its destination. If only the PR is ready, report that state and URL; do not report the banner as deployed. A website failure does not invalidate or move the published release tag: retry this phase after resolving the failure.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,16 +1,15 @@
 ---
 name: rustfs-release-version-bump
-description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
+description: "Prepare Cargo versions for an exact RustFS target, or align installation references after its artifacts are published, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
 ---
 # RustFS Release Version Bump
 
 Use this skill to prepare and verify release version files. Commit, push, and PR steps apply only when included in the user's delivery scope; publishing release tags belongs to `rustfs-release-publish`.
 
-Validated baseline: release pattern used in PR `#2957`.
-
 ## Required inputs
 
 - Exact target version, for example `1.0.0-beta.4`.
+- Stage: `prepare` (default) or `post-release`. Infer `post-release` only when the user or parent release workflow explicitly requests installation updates after publication; a normal version bump means `prepare`.
 - Delivery scope: local (`edit/verify`), git (`commit/push`), or GitHub
   (`commit/push/PR`). Derive it from the conversation; when unspecified, prepare
   and verify locally without blocking on a delivery question.
@@ -25,25 +24,32 @@ Reject any target version containing `-preview`: preview identifiers are tag-onl
 - `.github/pull_request_template.md` only when preparing a PR.
 - Current branch status and diff against `origin/main`.
 
-## Default release file scope
+## Stage boundaries
 
-Treat the following file list as the default checklist for each release bump:
+`prepare` updates only:
 
 - `Cargo.toml`
 - `Cargo.lock`
+
+`post-release` updates installation and packaging references only:
+
 - `README.md`
 - `README_ZH.md`
 - `flake.nix`
 - `helm/rustfs/Chart.yaml`
 - `rustfs.spec`
 
-Only drop a file when the current repository release process clearly no longer requires it.
+During preview, installation references intentionally retain the previous published deliverable. Do not sweep them into a source bump to make every version string match. `flake.nix` builds local source, but its package label is aligned with the other packaging references after publication.
+
+Before `post-release` edits, verify that the exact non-preview target has a published GitHub Release, downloadable assets and source archive, and a pullable `rustfs/rustfs:<target>` image manifest. If any prerequisite is missing, report `BLOCKED` and leave installation references unchanged. If a newer deliverable has already superseded this target, do not downgrade installation defaults during a retry.
+
+Helm CI derives chart versions from the triggering tag, so the final tag can retain the previous Chart.yaml values. Update the repository copy only after the target image and published chart are available. Neither post-release changes nor their merge commit may replace the preview-validated final tag.
 
 ## Hard release policy
 
 - Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
-- Helm chart version mapping follows `beta.N -> 0.N.0`.
-- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
+- Derive Helm chart and app versions with `scripts/helm_chart_version.sh <target>`; the existing mapping is `beta.N -> 0.N.0`, with other target versions retained.
+- `rustfs.spec` `Release` uses the prerelease suffix (for example `beta.4`), or `1` for a stable release.
 - Do not change these rules without explicit confirmation.
 
 ## Step-by-step workflow
@@ -52,22 +58,18 @@ Only drop a file when the current repository release process clearly no longer r
 - Use the exact target and delivery scope already supplied; ask only for a missing or ambiguous target or a material release-policy choice.
 - Inspect current branch and ensure only release-related files are touched for this task.
 
-2. Update workspace versions
+2. Update workspace versions (`prepare` only)
 - Bump `[workspace.package].version` in `Cargo.toml`.
 - Bump internal workspace crate dependency versions in `Cargo.toml`.
 - Update `Cargo.lock` so workspace package versions match target version.
-- Re-scan for partial leftovers.
+- Re-scan Cargo.toml and workspace members in Cargo.lock for partial leftovers; leave external dependency versions unchanged.
 
-3. Update release assets
+3. Update installation references (`post-release` only)
 - `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
 - `flake.nix`: update package version to target version.
-- `helm/rustfs/Chart.yaml`:
-- `appVersion` = target version.
-- `version` follows chart mapping rule, for example:
-- `1.0.0-beta.3` -> `0.3.0`
-- `1.0.0-beta.4` -> `0.4.0`
+- `helm/rustfs/Chart.yaml`: use the app and chart versions returned by `scripts/helm_chart_version.sh <target>`.
 - `rustfs.spec`:
-- Set `Release` to prerelease suffix (example `beta.4`).
+- Set `Version` to the numeric version and `Release` to the prerelease suffix (example `beta.4`), or `1` for a stable release. Verify that `Source0` and the unpacked source directory resolve to the exact published target, including its prerelease suffix when present.
 - Add/update top changelog entry with exact format:
 - `* Thu May 20 2026 houseme <housemecn@gmail.com>`
 - `- Update RPM package to RustFS 1.0.0-beta.4`
@@ -78,15 +80,14 @@ Only drop a file when the current repository release process clearly no longer r
 - Changelog version text must match target release version exactly.
 
 4. Verify before shipping
-- Run:
-- `make pre-commit`
-- If `make pre-commit` fails, fix task-attributable failures and rerun affected checks. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
+- Follow the root verification tiers for the final diff instead of running a full-workspace gate for version strings.
+- For `prepare`, validate Cargo metadata with the updated lockfile and confirm workspace package/internal dependency versions agree. Do not accept a lockfile containing unrelated dependency updates.
+- For `post-release`, render the Helm chart and confirm its default image is the verified target; run `scripts/test_helm_chart_version.sh` when chart versions change. Check the README image tags, Nix package version, and expanded RPM source URL against that same target.
+- Run `git diff --check` for either stage. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
 
 5. Commit strategy (only when committing is authorized)
-- Preferred split when both parts changed:
-- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
-- `chore(release): align release assets for <version>` for docs and packaging files.
-- If user asks for one commit, use one commit.
+- Use `chore(release): prepare <version>` for `prepare` and `chore(release): align installation references for <version>` for `post-release`.
+- These stages happen on opposite sides of publication; do not combine them in a preview-preparation commit or PR.
 - Stage only intended release files; do not include unrelated working tree changes.
 
 6. Push and PR (only for the authorized delivery scope)
@@ -104,13 +105,12 @@ Only drop a file when the current repository release process clearly no longer r
 - `git diff --name-only origin/main...HEAD`
 - `git diff --stat origin/main...HEAD`
 - `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
-- `make pre-commit`
 
 ## Output contract
 
 When using this skill, always report:
 
-- Target version.
+- Target version and stage.
 - Files changed.
 - Any assumptions or uncertainties requiring confirmation.
 - Verification result (`PASSED` or `BLOCKED`) with key evidence.

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=548a58d74f3cb3b5a1ab9dcd4d3f8625b9ddebe847a2e14de1d08c3b2e6292eb
+sha256=5de6852b2e606eca4ee0d86dbb3b725a3eb8923143fd00df96c549acc36040bd

.config/make/tests.mak

@@ -24,6 +24,7 @@ TEST_THREADS ?= 1
 .PHONY: script-tests
 script-tests: ## Run shell script tests
 	@echo "Running script tests..."
+	trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
 	./scripts/test_build_rustfs_options.sh
 	./scripts/test_docker_runtime_timezone.sh
 	./scripts/test_entrypoint_credentials.sh

.config/nextest.toml

@@ -132,10 +132,11 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
 test-group = 'embedded-test-ports'
 
-# The real object probe includes bucket creation and cleanup in its short
-# measurement budget. Keep competing storage fixtures outside that budget.
+# Inventory delivery and real drive/object probes include durable filesystem IO
+# in short deadlines. Reserve capacity so unrelated storage fixtures cannot
+# exhaust those budgets. Keep the deadlines and assertions unchanged.
 [[profile.default.overrides]]
-filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
+filter = 'package(rustfs) & (binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
 threads-required = "num-test-threads"
 
 # Serialize the durable manual-transition checkpoint test across nextest's
@@ -334,7 +335,7 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
 test-group = 'embedded-test-ports'
 
 [[profile.ci.overrides]]
-filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
+filter = 'package(rustfs) & (binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
 threads-required = "num-test-threads"
 
 # Serialize the durable manual-transition checkpoint test under the ci profile

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,38 +0,0 @@
----
-name: Bug report
-about: Create a report to help us improve
-title: ''
-labels: ''
-assignees: ''
-
----
-
-**Describe the bug**
-A clear and concise description of what the bug is.
-
-**To Reproduce**
-Steps to reproduce the behavior:
-1. Go to '...'
-2. Click on '....'
-3. Scroll down to '....'
-4. See error
-
-**Expected behavior**
-A clear and concise description of what you expected to happen.
-
-**Screenshots**
-If applicable, add screenshots to help explain your problem.
-
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
-
-**Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
-
-**Additional context**
-Add any other context about the problem here.

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,110 @@
+name: Bug report
+description: Create a report to help us improve
+body:
+  - type: markdown
+    attributes:
+      value: |
+        ### Storage support requirements
+        **Supported:** local disks, SAN (Fibre Channel) attached volumes, or JBOD-mounted disks. We recommend formatting them with the **XFS** file system.
+
+        **Not supported:** NFS, SMB/Samba/CIFS, and other remote or shared filesystems as RustFS data directories. This also applies to the underlying host storage for Docker volumes and Kubernetes PersistentVolumes.
+
+        Reports that only reproduce on unsupported storage will be closed without investigation. Please reproduce the issue on supported storage before submitting a bug report.
+
+        See [Disk Planning and File System Selection](https://docs.rustfs.com/en/installation/linux/prerequisites-and-service) for details.
+  - type: checkboxes
+    id: storage-support
+    attributes:
+      label: Supported storage confirmation
+      options:
+        - label: I have reproduced this issue with all RustFS data directories on supported storage (local disks, SAN/Fibre Channel, or JBOD; XFS recommended), without NFS, SMB/Samba/CIFS, or other remote or shared filesystem mounts, including storage backing Docker volumes and Kubernetes PersistentVolumes.
+          required: true
+  - type: textarea
+    id: bug-description
+    attributes:
+      label: Describe the bug
+      description: A clear and concise description of what happened.
+    validations:
+      required: true
+  - type: textarea
+    id: reproduction-steps
+    attributes:
+      label: Steps to reproduce
+      description: Include the commands, configuration, or actions needed to reproduce the problem.
+      placeholder: |
+        1. Start RustFS with ...
+        2. Run ...
+        3. Observe ...
+    validations:
+      required: true
+  - type: textarea
+    id: expected-behavior
+    attributes:
+      label: Expected behavior
+      description: A clear and concise description of what you expected to happen.
+    validations:
+      required: true
+  - type: input
+    id: rustfs-version
+    attributes:
+      label: RustFS version
+      description: Provide the release version, container image tag, or commit hash.
+    validations:
+      required: true
+  - type: dropdown
+    id: deployment-method
+    attributes:
+      label: Deployment method
+      description: How are you running RustFS? Add any relevant details in Additional context.
+      options:
+        - Binary
+        - Docker / Docker Compose
+        - Kubernetes / Helm
+        - Built from source
+        - Other
+  - type: textarea
+    id: storage-mounts
+    attributes:
+      label: Data directory filesystems and mounts
+      description: |
+        Identify the filesystem type and mount source for every RustFS data directory.
+        On Linux, include output from `findmnt -T /path/to/data -o TARGET,SOURCE,FSTYPE,OPTIONS` for each data path; on other systems, provide equivalent mount information.
+        For Docker, identify the backing host path and filesystem, not just the container path or volume name.
+        For Kubernetes, identify the PV, StorageClass/CSI driver, and actual backing filesystem; a PVC name alone is not sufficient.
+        Remove credentials and other sensitive information before submitting.
+      placeholder: |
+        RustFS data paths:
+        Filesystem types and mount sources:
+        Mount command output:
+        Docker host paths or Kubernetes volume backing storage (if applicable):
+    validations:
+      required: true
+  - type: input
+    id: operating-system
+    attributes:
+      label: Server operating system and architecture
+      description: Include the operating system version and CPU architecture where RustFS is running.
+      placeholder: "For example: Ubuntu 24.04, x86_64"
+    validations:
+      required: true
+  - type: textarea
+    id: client-environment
+    attributes:
+      label: Client environment
+      description: Include the S3 client or SDK and version. For console issues, include the browser, operating system, and desktop or mobile device details.
+  - type: textarea
+    id: logs
+    attributes:
+      label: Relevant logs
+      description: Paste relevant log output. Remove credentials and other sensitive information before submitting.
+      render: shell
+  - type: textarea
+    id: screenshots
+    attributes:
+      label: Screenshots
+      description: If applicable, drag and drop screenshots here to help explain the problem.
+  - type: textarea
+    id: additional-context
+    attributes:
+      label: Additional context
+      description: Add any other context, such as deployment topology or relevant configuration. Remove credentials and other sensitive information before submitting.

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: false

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,20 +0,0 @@
----
-name: Feature request
-about: Suggest an idea for this project
-title: ''
-labels: ''
-assignees: ''
-
----
-
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-
-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
-
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
-
-**Additional context**
-Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,38 @@
+name: Feature request
+description: Suggest an idea for this project
+body:
+  - type: checkboxes
+    id: request-scope
+    attributes:
+      label: Request scope
+      description: |
+        RustFS data directories backed by NFS, SMB/Samba/CIFS, or other remote or shared filesystems are unsupported, including those behind Docker volumes or Kubernetes PersistentVolumes.
+        Troubleshooting requests for these deployments will be closed without investigation. Use this form for new or improved functionality.
+      options:
+        - label: This is a feature request and does not ask for troubleshooting an unsupported remote or shared filesystem deployment.
+          required: true
+  - type: textarea
+    id: problem-description
+    attributes:
+      label: Problem or use case
+      description: Describe the problem or use case that motivates this feature request.
+      placeholder: "For example: I am trying to ... but ..."
+    validations:
+      required: true
+  - type: textarea
+    id: proposed-solution
+    attributes:
+      label: Proposed solution
+      description: A clear and concise description of what you would like to happen.
+    validations:
+      required: true
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: Alternatives considered
+      description: Describe any alternative solutions or features you have considered.
+  - type: textarea
+    id: additional-context
+    attributes:
+      label: Additional context
+      description: Add any other context or drag and drop screenshots about the feature request here.

.github/workflows/architecture-migration-rules.yml

@@ -36,7 +36,7 @@ jobs:
   cancel-closed-pr-runs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Explain cancellation run
@@ -45,7 +45,7 @@ jobs:
   architecture-migration-rules:
     name: Architecture Migration Rules
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7

.github/workflows/audit.yml

@@ -77,7 +77,7 @@ jobs:
   cancel-closed-pr-runs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Explain cancellation run
@@ -89,7 +89,7 @@ jobs:
   cargo-deny:
     name: Cargo Deny
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 20
     steps:
       - name: Checkout repository
@@ -132,7 +132,7 @@ jobs:
   workflow-pin-report:
     name: Workflow Pin Report
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 5
     steps:
       - name: Checkout repository
@@ -169,7 +169,7 @@ jobs:
 
   dependency-review:
     name: Dependency Review
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     if: github.event_name == 'pull_request' && github.event.action != 'closed'
     permissions:
@@ -204,7 +204,7 @@ jobs:
     # open the Actions tab. Same ci-8 mechanism coverage.yml and
     # e2e-replication-nightly.yml already use.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/build.yml

@@ -27,7 +27,8 @@
 #   schedule and manual dispatch build the full platform matrix.
 #
 # Manual Parameters:
-# - build_docker: Build and push Docker images (default: true)
+# - publish: Publish binaries/releases for manual runs (default: false)
+# - build_docker: Advisory only; manual runs do not trigger Docker publishing
 # - platforms: Comma-separated platform IDs or 'all' (default: all)
 
 name: Build and Release
@@ -55,12 +56,16 @@ on:
     - cron: "13 1 * * 0" # Weekly on Sunday 01:13 UTC
   workflow_dispatch:
     inputs:
+      publish:
+        description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
+        required: false
+        default: false
+        type: boolean
       build_docker:
         # Advisory only. docker.yml triggers on workflow_run and its job-level
         # condition requires the triggering event to be a tag push, so a manual
         # dispatch of this workflow never produces images regardless of this
-        # value. Kept because the summary step reports it; wiring it up would
-        # mean teaching docker.yml's version parser a second event shape.
+        # value. Publishing remains controlled by the explicit publish input.
         description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
         required: false
         default: true
@@ -88,10 +93,11 @@ jobs:
   # Build strategy check - determine build type based on trigger
   build-check:
     name: Build Strategy Check
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
+      should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
       build_type: ${{ steps.check.outputs.build_type }}
       version: ${{ steps.check.outputs.version }}
       short_sha: ${{ steps.check.outputs.short_sha }}
@@ -172,7 +178,7 @@ jobs:
   prepare-platform-matrix:
     name: Prepare Platform Matrix
     needs: build-check
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     outputs:
       matrix: ${{ steps.select.outputs.matrix }}
@@ -208,9 +214,9 @@ jobs:
 
           all='{"include":[
             {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":"","features":""},
+            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
             {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope"},
+            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
           ]}'
@@ -402,6 +408,8 @@ jobs:
 
       - name: Build RustFS
         shell: bash
+        env:
+          BUILD_TARGET: ${{ matrix.target }}
         run: |
           # Force rebuild by touching build.rs
           touch rustfs/build.rs
@@ -418,14 +426,37 @@ jobs:
           # Build one binary at a time so release LTO links cannot exhaust a
           # self-hosted runner by running concurrently.
           for binary in "${BINARY_NAMES[@]}"; do
+            PROFILE_ARGS=()
+            if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+              # The in-process profiler resolves names from the final executable.
+              # Retain function symbols without enabling DWARF source information.
+              PROFILE_ARGS=(--config 'profile.release.strip="none"')
+            fi
             if [[ "${{ matrix.cross }}" == "true" ]]; then
               # All cross targets in the matrix are Linux; zigbuild handles them.
-              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
+              cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
             else
-              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
+              cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
             fi
           done
 
+      - name: Generate the CPU symbol catalog
+        if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
+        shell: bash
+        run: |
+          cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
+            --features rustfs/pyroscope > "$RUNNER_TEMP/cpu-catalog-metadata.json"
+          features=$(jq -ce '
+            (.packages[] | select(.name == "rustfs") | .id) as $id |
+            .resolve.nodes[] | select(.id == $id) | .features | map(select(. != "default")) | sort
+          ' "$RUNNER_TEMP/cpu-catalog-metadata.json")
+          release_dir="target/${{ matrix.target }}/release"
+          # These task-owned outputs may remain in a restored target cache.
+          rm -f "$release_dir/rustfs.cpu-symbol-catalog.json" "$release_dir/rustfs.cpu-symbol-catalog.sha256"
+          cargo run --locked -p rustfs-connect-symbol-catalog -- \
+            "$release_dir/rustfs" "$(git rev-parse HEAD)" '${{ matrix.target }}' "$features" \
+            "$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
+
       - name: Create release package
         id: package
         shell: bash
@@ -516,6 +547,10 @@ jobs:
             BINARY_FILES+=("rustfs-cli")
           fi
 
+          if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
+            BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
+          fi
+
           # Verify the binaries exist before packaging
           for binary in "${BINARY_FILES[@]}"; do
             if [[ ! -f "$binary" ]]; then
@@ -625,7 +660,7 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
-      - name: Verify packaged Linux CLI
+      - name: Verify packaged Linux binaries
         if: matrix.platform == 'linux'
         shell: bash
         run: |
@@ -634,21 +669,27 @@ jobs:
           package_file="${{ steps.package.outputs.package_file }}"
           archive_members="$(unzip -Z1 "$package_file" | sort)"
           expected_members=$'rustfs\nrustfs-cli'
+          target="${{ matrix.target }}"
+          features="${{ matrix.features }}"
+          if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
+            expected_members+=$'\nrustfs.cpu-symbol-catalog.json\nrustfs.cpu-symbol-catalog.sha256'
+          fi
           if [[ "$archive_members" != "$expected_members" ]]; then
-            echo "Linux package must contain exactly rustfs and rustfs-cli" >&2
+            echo "Linux package does not contain the expected files" >&2
             printf '%s\n' "$archive_members" >&2
             exit 1
           fi
 
-          if [[ "${{ matrix.target }}" == "x86_64-unknown-linux-gnu" ]]; then
-            cli_dir="$(mktemp -d)"
-            trap 'rm -rf "$cli_dir"' EXIT
-            unzip -q "$package_file" rustfs-cli -d "$cli_dir"
-            "$cli_dir/rustfs-cli" --help >/dev/null
-          fi
+          # Every Linux target builds natively, so executing the binaries here
+          # proves each artifact matches its runner architecture.
+          cli_dir="$(mktemp -d)"
+          trap 'rm -rf "$cli_dir"' EXIT
+          unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
+          "$cli_dir/rustfs" --version
+          "$cli_dir/rustfs-cli" --help >/dev/null
 
       - name: Verify packaged console
-        if: matrix.target == 'x86_64-unknown-linux-gnu'
+        if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
         shell: bash
         run: |
           set -euo pipefail
@@ -709,7 +750,7 @@ jobs:
           retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
 
       - name: Upload to Cloudflare R2
-        if: env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
+        if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
         env:
           R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
           R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -726,12 +767,33 @@ jobs:
           fi
 
           # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
-          # images did). Install it on demand so R2 uploads survive a fresh runner
-          # instead of hard-failing here.
+          # images did). Install the official CLI bundle on demand so R2 uploads
+          # survive a fresh runner without relying on a distro package.
           if ! command -v aws >/dev/null 2>&1; then
             echo "aws CLI not found on runner; installing..."
             if command -v apt-get >/dev/null 2>&1; then
-              sudo apt-get update && sudo apt-get install -y awscli
+              aws_arch="$(uname -m)"
+              case "$aws_arch" in
+                x86_64) aws_arch="x86_64" ;;
+                aarch64|arm64) aws_arch="aarch64" ;;
+                *)
+                  echo "❌ Unsupported Linux architecture for AWS CLI: $aws_arch"
+                  exit 1
+                  ;;
+              esac
+
+              aws_install_root="${RUNNER_TEMP:-$PWD}/aws-cli"
+              rm -rf "$aws_install_root"
+              mkdir -p "$aws_install_root"
+              curl --fail --location --retry 3 \
+                "https://awscli.amazonaws.com/awscli-exe-linux-${aws_arch}.zip" \
+                --output "$aws_install_root/awscliv2.zip"
+              unzip -q "$aws_install_root/awscliv2.zip" -d "$aws_install_root"
+              "$aws_install_root/aws/install" \
+                --install-dir "$aws_install_root/aws-cli" \
+                --bin-dir "$aws_install_root/bin"
+              export PATH="$aws_install_root/bin:$PATH"
+              aws --version
             elif command -v brew >/dev/null 2>&1; then
               brew install awscli
             else
@@ -770,68 +832,41 @@ jobs:
     name: Build Summary
     needs: [ build-check, build-rustfs ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Build completion summary
         shell: bash
         env:
-          # dispatch input via env: free-form text must not be pasted into the
-          # script for bash to evaluate.
-          INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
+          SHOULD_PUBLISH: ${{ needs.build-check.outputs.should_publish }}
+          BUILD_TYPE: ${{ needs.build-check.outputs.build_type }}
+          VERSION: ${{ needs.build-check.outputs.version }}
+          BUILD_STATUS: ${{ needs.build-rustfs.result }}
+          EVENT_NAME: ${{ github.event_name }}
         run: |
-          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
-          VERSION="${{ needs.build-check.outputs.version }}"
-
-          echo "🎉 Build completed successfully!"
-          echo "📦 Build type: $BUILD_TYPE"
-          echo "🔢 Version: $VERSION"
-          echo ""
-
-          # Check build status
-          BUILD_STATUS="${{ needs.build-rustfs.result }}"
-
-          echo "📊 Build Results:"
-          echo "  📦 All platforms: $BUILD_STATUS"
-          echo ""
-
-          case "$BUILD_TYPE" in
-            "preview")
-              echo "🔍 Preview artifacts are published in a GitHub prerelease"
-              echo "⏭️  Preview releases do not update latest channels"
-              ;;
-            "development")
-              echo "🛠️  Development build artifacts have been uploaded to OSS dev directory"
-              echo "⚠️  This is a development build - not suitable for production use"
-              ;;
-            "release")
-              echo "🚀 Release build artifacts have been uploaded to OSS release directory"
-              echo "✅ This build is ready for production use"
-              echo "🏷️  GitHub Release will be created in this workflow"
-              ;;
-            "prerelease")
-              echo "🧪 Prerelease build artifacts have been uploaded to OSS release directory"
-              echo "⚠️  This is a prerelease build - use with caution"
-              echo "🏷️  GitHub Release will be created in this workflow"
-              ;;
-          esac
-
-          echo ""
-          echo "🐳 Docker Images:"
-          if [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
-            echo "⏭️  Docker image build was skipped (binary only build)"
+          echo "Build type: $BUILD_TYPE"
+          echo "Version: $VERSION"
+          echo "Build result: $BUILD_STATUS"
+          if [[ "$SHOULD_PUBLISH" != "true" ]]; then
+            echo "Artifact-only run: no download channels or GitHub Releases are updated."
+            echo "Successful platform packages are available as workflow artifacts."
           elif [[ "$BUILD_STATUS" == "success" ]]; then
-            echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
+            echo "Publishing enabled; check the upload and release jobs for their results."
           else
-            echo "❌ Docker image build will be skipped due to build failure"
+            echo "Build incomplete; check individual platform jobs for any completed uploads."
+          fi
+          if [[ "$EVENT_NAME" == "push" && "$GITHUB_REF" == refs/tags/* && "$BUILD_STATUS" == "success" ]]; then
+            echo "Docker publishing is eligible through the downstream workflow."
+          else
+            echo "This run does not trigger Docker publishing."
           fi
 
   # Create GitHub Release for every valid release tag, including previews
   create-release:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: ubuntu-latest
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
     timeout-minutes: 30
     permissions:
       contents: write
@@ -891,8 +926,8 @@ jobs:
   upload-release-assets:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: ubuntu-latest
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
     timeout-minutes: 30
     permissions:
       contents: write
@@ -985,8 +1020,8 @@ jobs:
   update-latest-version:
     name: Update Latest Version
     needs: [ build-check, publish-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: ubuntu-latest
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: sm-standard-2
     timeout-minutes: 30
     steps:
       - name: Update latest.json
@@ -1045,8 +1080,8 @@ jobs:
   publish-release:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: ubuntu-latest
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
     timeout-minutes: 30
     permissions:
       contents: write
@@ -1083,8 +1118,8 @@ jobs:
   cleanup-preview-releases:
     name: Cleanup Preview Releases
     needs: [ build-check, publish-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
-    runs-on: ubuntu-latest
+    if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
     timeout-minutes: 30
     permissions:
       contents: write
@@ -1132,7 +1167,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/cache-warm.yml

@@ -256,7 +256,7 @@ jobs:
   # belonged to it.
   warm-ci-uring:
     name: Warm ci-uring
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 60
     steps:
       - name: Checkout repository

.github/workflows/ci.yml

@@ -63,7 +63,7 @@ jobs:
   cancel-closed-pr-runs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Explain cancellation run
@@ -72,7 +72,7 @@ jobs:
   classify-changes:
     name: Select CI scope
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     outputs:
       mode: ${{ steps.scope.outputs.mode }}
@@ -98,12 +98,16 @@ jobs:
   typos:
     name: Typos
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
         with:
           persist-credentials: false
+      - name: Install typos download dependency
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends wget
       - name: Typos check with custom config file
         uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2
 
@@ -111,7 +115,7 @@ jobs:
   quick-checks:
     name: Quick Checks
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Checkout repository
@@ -1230,7 +1234,7 @@ jobs:
       - build-rustfs-debug-binary-rio-v2
       - e2e-tests-rio-v2
       - e2e-full
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -1266,7 +1270,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/cla.yml

@@ -46,7 +46,7 @@ jobs:
     # Echoes one line; the run exists only so the concurrency group cancels the
     # in-flight run of a closed PR.
     permissions: {}
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Explain cancellation run
@@ -61,7 +61,7 @@ jobs:
       checks: write
       issues: write
       pull-requests: write
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     steps:
       - name: Report CLA result for merge queue

.github/workflows/connect-diagnostic-schedule-acceptance.yml

@@ -38,7 +38,7 @@ permissions:
 jobs:
   diagnostic-schedule:
     name: Verify production scheduler receipts over mTLS
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 45
     steps:
       - name: Checkout exact RustFS source
@@ -60,7 +60,7 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 

.github/workflows/connect-health-service-acceptance.yml

@@ -0,0 +1,236 @@
+# Copyright 2026 RustFS Team
+# SPDX-License-Identifier: Apache-2.0
+
+name: Connect bounded health service acceptance
+
+on:
+  workflow_dispatch:
+    inputs:
+      build_run_id:
+        description: Successful main-branch Build and Release run ID
+        required: true
+        type: string
+      artifact_id:
+        description: Linux x86_64 GNU artifact from that run
+        required: true
+        type: string
+      source_sha:
+        description: Exact RustFS source commit
+        required: true
+        type: string
+      artifact_digest:
+        description: GitHub artifact digest including sha256 prefix
+        required: true
+        type: string
+      binary_sha256:
+        description: Independently verified rustfs binary SHA-256
+        required: true
+        type: string
+      connect_sha:
+        description: Exact Connect health service acceptance harness commit
+        required: true
+        type: string
+
+permissions:
+  actions: read
+  contents: read
+
+jobs:
+  health-service:
+    name: Verify bounded health service over mTLS
+    runs-on: ubuntu-24.04
+    timeout-minutes: 45
+    steps:
+      - name: Checkout exact RustFS source
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          path: rustfs-source
+          persist-credentials: false
+          ref: ${{ inputs.source_sha }}
+
+      - name: Checkout exact Connect harness
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          repository: rustfs/connect
+          path: connect-harness
+          persist-credentials: false
+          ref: ${{ inputs.connect_sha }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+
+      - name: Set up Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 25.8.1
+          cache: npm
+          cache-dependency-path: connect-harness/web/package-lock.json
+
+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
+
+      - name: Verify official source and artifact identity
+        shell: bash
+        env:
+          GH_TOKEN: ${{ github.token }}
+          BUILD_RUN_ID: ${{ inputs.build_run_id }}
+          ARTIFACT_ID: ${{ inputs.artifact_id }}
+          SOURCE_SHA: ${{ inputs.source_sha }}
+          ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_SHA: ${{ inputs.connect_sha }}
+        run: |
+          set -euo pipefail
+          [[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
+          [[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
+          [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
+          [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
+          [[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
+          [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
+          [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
+          [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
+          [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
+          run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
+          jq -e --arg source "$SOURCE_SHA" '
+            .head_sha == $source and .head_branch == "main"
+            and .head_repository.full_name == "rustfs/rustfs"
+            and .name == "Build and Release" and .path == ".github/workflows/build.yml"
+            and .status == "completed" and .conclusion == "success"
+          ' <<<"$run" >/dev/null
+          jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
+          jq -e '
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            | ($matches | length) == 1 and $matches[0].conclusion == "success"
+          ' <<<"$jobs" >/dev/null
+          artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
+          jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
+            .workflow_run.id == $run and .workflow_run.head_sha == $source
+            and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
+            and .digest == $digest and .expired == false
+            and (.expires_at | fromdateiso8601) > now
+            and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
+          ' <<<"$artifact" >/dev/null
+          jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
+          gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
+          [[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
+          printf '%s  official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
+
+      - name: Extract only the verified official service binary
+        shell: bash
+        env:
+          SOURCE_SHA: ${{ inputs.source_sha }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+        run: |
+          set -euo pipefail
+          python3 - <<'PY'
+          import io
+          import os
+          import pathlib
+          import shutil
+          import stat
+          import zipfile
+
+          def regular(entry):
+              mode = entry.external_attr >> 16
+              return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
+
+          package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
+          with zipfile.ZipFile('official-artifact.zip') as outer:
+              entries = outer.infolist()
+              assert 1 <= len(entries) <= 16
+              assert len({entry.filename for entry in entries}) == len(entries)
+              assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
+              assert sum(entry.file_size for entry in entries) <= 2147483648
+              assert outer.testzip() is None
+              with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
+                  entries = inner.infolist()
+                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
+                  assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
+                  assert inner.testzip() is None
+                  pathlib.Path('binary').mkdir()
+                  with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
+                      header = source.read(20)
+                      assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
+                      destination.write(header)
+                      shutil.copyfileobj(source, destination)
+          PY
+          chmod 0755 binary/rustfs
+          printf '%s  binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
+
+      - name: Run the real health service job
+        shell: bash
+        env:
+          SOURCE_SHA: ${{ inputs.source_sha }}
+        run: |
+          set -euo pipefail
+          git -C connect-harness diff --exit-code
+          printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
+          changed=$(git -C connect-harness diff --name-only)
+          [[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
+          npm --prefix connect-harness/web ci
+          connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
+          make -C connect-harness e2e-connected-dispatch-check
+          (cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+            RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
+            CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
+            make -C connect-harness e2e-connected E2E_SCENARIO=health-service
+
+      - name: Bind and validate sanitized evidence
+        shell: bash
+        env:
+          BUILD_RUN_ID: ${{ inputs.build_run_id }}
+          ARTIFACT_ID: ${{ inputs.artifact_id }}
+          ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_SHA: ${{ inputs.connect_sha }}
+          SOURCE_SHA: ${{ inputs.source_sha }}
+        run: |
+          set -euo pipefail
+          jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
+            .provenance.connectSha == $connect and .provenance.sourceSha == $source
+            and .provenance.binarySha256 == $binary
+          ' health-service-evidence.json >/dev/null
+          jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
+            --arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
+            --arg expiry "$(cat artifact-expires-at)" '
+            .provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry}
+          ' health-service-evidence.json > health-service-evidence.bound.json
+          mv health-service-evidence.bound.json health-service-evidence.json
+          node connect-harness/tests/connectivity/health-service-evidence.mjs health-service-evidence.json --bound
+
+      - name: Upload only verified sanitized evidence
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: connect-health-service-evidence-${{ github.run_id }}
+          path: health-service-evidence.json
+          retention-days: 14
+          if-no-files-found: error

.github/workflows/connect-performance-drive-acceptance.yml

@@ -49,7 +49,7 @@ permissions:
 jobs:
   performance-drive:
     name: Verify native Linux x86_64 performance.drive service job
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 45
     steps:
       - name: Checkout acceptance harness
@@ -76,7 +76,7 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 
@@ -107,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-performance-network-acceptance.yml

@@ -49,7 +49,7 @@ permissions:
 jobs:
   performance-network:
     name: Verify native Linux x86_64 performance.network service job
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 45
     steps:
       - name: Checkout acceptance harness
@@ -76,7 +76,7 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 
@@ -107,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-profile-cpu-acceptance.yml

@@ -37,6 +37,10 @@ on:
         description: Expected rustfs binary SHA-256
         required: true
         type: string
+      catalog_sha256:
+        description: Independently reviewed release CPU catalog SHA-256
+        required: true
+        type: string
       connect_sha:
         description: Exact 40-character RustFS Connect harness commit
         required: true
@@ -49,7 +53,7 @@ permissions:
 jobs:
   profile-cpu:
     name: Verify native Linux x86_64 profile.cpu service job
-    runs-on: ubuntu-latest
+    runs-on: dind-sm-standard-2
     timeout-minutes: 45
     steps:
       - name: Checkout acceptance harness
@@ -76,7 +80,7 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 
@@ -101,20 +105,27 @@ jobs:
           [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
           [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
 
-          run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
+          github_api() {
+            curl --fail --silent --show-error \
+              --header "Authorization: Bearer $GH_TOKEN" \
+              --header 'Accept: application/vnd.github+json' \
+              "https://api.github.com/$1"
+          }
+
+          run=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
           [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
           [[ $(jq -r '.head_branch' <<<"$run") == main ]]
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
-          jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
-            [.[].jobs[] | select(.name == $name)] as $matches
+            [.jobs[] | select(.name == $name)] as $matches
             | (($matches | length) == 1 and $matches[0].conclusion == "success")
           ' <<<"$jobs" >/dev/null
 
-          artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
+          artifact=$(github_api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
           [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
           [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
           [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
@@ -129,24 +140,32 @@ jobs:
           run-id: ${{ inputs.build_run_id }}
           github-token: ${{ github.token }}
 
-      - name: Extract exact RustFS binary
+      - name: Extract exact RustFS binary and reviewed catalog
         shell: bash
         env:
           BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
           SOURCE_SHA: ${{ inputs.source_sha }}
         run: |
           set -euo pipefail
+          [[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
+          [[ "$CATALOG_SHA256" =~ ^[0-9a-f]{64}$ ]]
           short_sha=${SOURCE_SHA:0:7}
           package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
           [[ -n "$package" ]]
           mkdir -p binary
-          unzip -qq "$package" rustfs -d binary
+          unzip -qq "$package" rustfs rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256 -d binary
           chmod +x binary/rustfs
           printf '%s  %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict
+          # The packaged digest is corroboration, never the independent trust input.
+          [[ $(cat binary/rustfs.cpu-symbol-catalog.sha256) == "$CATALOG_SHA256" ]]
+          printf '%s  %s\n' "$CATALOG_SHA256" binary/rustfs.cpu-symbol-catalog.json | sha256sum --check --strict
+          chmod 0444 binary/rustfs.cpu-symbol-catalog.json
 
       - name: Run signed service job acceptance
         shell: bash
         env:
+          CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
           SOURCE_SHA: ${{ inputs.source_sha }}
         run: |
           set -euo pipefail
@@ -156,6 +175,8 @@ jobs:
           make -C connect-harness e2e-connected-dispatch-check
           RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
             RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
+            CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
+            CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \
             CONNECT_E2E_PROFILE_EVIDENCE="$GITHUB_WORKSPACE/profile-service-job-evidence.json" \
             make -C connect-harness e2e-connected E2E_SCENARIO=profile
 

.github/workflows/connect-profile-memory-acceptance.yml

@@ -38,7 +38,7 @@ permissions:
 jobs:
   profile-memory:
     name: Verify signed service memory profile over mTLS
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     timeout-minutes: 45
     steps:
       - name: Checkout exact RustFS source
@@ -60,10 +60,43 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 
+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
+
       - name: Verify official source and artifact identity
         shell: bash
         env:
@@ -139,7 +172,8 @@ jobs:
               assert outer.testzip() is None
               with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
                   entries = inner.infolist()
-                  assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
                   assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
                   assert inner.testzip() is None
                   pathlib.Path('binary').mkdir()

.github/workflows/connect-profile-threads-acceptance.yml

@@ -17,6 +17,14 @@ name: Connect profile.threads service-job acceptance
 on:
   workflow_dispatch:
     inputs:
+      thread_scope:
+        description: Thread profile scope
+        required: true
+        type: choice
+        default: native
+        options:
+          - native
+          - runtime
       build_run_id:
         description: Main-branch Build and Release workflow run ID
         required: true
@@ -48,8 +56,8 @@ permissions:
 
 jobs:
   profile-threads:
-    name: Verify native Linux x86_64 profile.threads service job
-    runs-on: ubuntu-latest
+    name: Verify Linux x86_64 profile.threads service job
+    runs-on: ubuntu-24.04
     timeout-minutes: 45
     steps:
       - name: Checkout acceptance harness
@@ -76,10 +84,43 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 
+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
+
       - name: Verify source run and artifact identity
         shell: bash
         env:
@@ -107,7 +148,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches
@@ -148,8 +189,14 @@ jobs:
         shell: bash
         env:
           SOURCE_SHA: ${{ inputs.source_sha }}
+          THREAD_SCOPE: ${{ inputs.thread_scope }}
         run: |
           set -euo pipefail
+          case "$THREAD_SCOPE" in
+            native) export CONNECT_E2E_THREAD_SCOPE=NATIVE_THREADS ;;
+            runtime) export CONNECT_E2E_THREAD_SCOPE=TOKIO_RUNTIME ;;
+            *) exit 1 ;;
+          esac
           printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
           npm --prefix connect-harness/web ci
           connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
@@ -168,6 +215,7 @@ jobs:
           ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
           CONNECT_SHA: ${{ inputs.connect_sha }}
           SOURCE_SHA: ${{ inputs.source_sha }}
+          THREAD_SCOPE: ${{ inputs.thread_scope }}
         run: |
           set -euo pipefail
           jq \
@@ -177,7 +225,8 @@ jobs:
             --arg artifactDigest "$ARTIFACT_DIGEST" \
             --arg connectSha "$CONNECT_SHA" \
             --arg sourceSha "$SOURCE_SHA" \
-            '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \
+            --arg requestedThreadScope "$THREAD_SCOPE" \
+            '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha, requestedThreadScope: $requestedThreadScope}' \
             profile-threads-service-job-evidence.json >profile-threads-service-job-evidence.bound.json
           mv profile-threads-service-job-evidence.bound.json profile-threads-service-job-evidence.json
 

.github/workflows/connect-top-api-acceptance.yml

@@ -49,7 +49,7 @@ permissions:
 jobs:
   top-api:
     name: Verify native Linux x86_64 top.api service job
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 45
     steps:
       - name: Checkout acceptance harness
@@ -76,7 +76,7 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 
@@ -107,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-top-disk-acceptance.yml

@@ -12,21 +12,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-name: Connect top.disk artifact acceptance
+name: Connect top.disk proxy service-job acceptance
 
 on:
   workflow_dispatch:
     inputs:
       build_run_id:
-        description: Successful main-branch Build and Release workflow run ID
+        description: Successful main-branch Build and Release run ID
         required: true
         type: string
       artifact_id:
-        description: Linux x86_64 GNU artifact ID from that run
+        description: Linux x86_64 GNU artifact from that run
         required: true
         type: string
       source_sha:
-        description: Exact 40-character source commit
+        description: Exact RustFS source commit
         required: true
         type: string
       artifact_digest:
@@ -34,7 +34,11 @@ on:
         required: true
         type: string
       binary_sha256:
-        description: Expected rustfs binary SHA-256
+        description: Independently verified rustfs binary SHA-256
+        required: true
+        type: string
+      connect_sha:
+        description: Exact Connect top.disk proxy service-job acceptance harness commit
         required: true
         type: string
 
@@ -44,16 +48,67 @@ permissions:
 
 jobs:
   top-disk:
-    name: Verify native Linux x86_64 top.disk artifact
-    runs-on: ubuntu-latest
-    timeout-minutes: 15
+    name: Verify native top.disk service job through authenticated proxy
+    runs-on: ubuntu-24.04
+    timeout-minutes: 45
     steps:
-      - name: Checkout acceptance harness
+      - name: Checkout exact RustFS source
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
         with:
+          path: rustfs-source
           persist-credentials: false
+          ref: ${{ inputs.source_sha }}
 
-      - name: Verify source run and artifact identity
+      - name: Checkout exact Connect harness
+        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          repository: rustfs/connect
+          path: connect-harness
+          persist-credentials: false
+          ref: ${{ inputs.connect_sha }}
+          token: ${{ secrets.PF_TESTING_GH_TOKEN }}
+
+      - name: Set up Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+        with:
+          node-version: 25.8.1
+          cache: npm
+          cache-dependency-path: connect-harness/web/package-lock.json
+
+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
+
+      - name: Verify official source and artifact identity
         shell: bash
         env:
           GH_TOKEN: ${{ github.token }}
@@ -61,82 +116,150 @@ jobs:
           ARTIFACT_ID: ${{ inputs.artifact_id }}
           SOURCE_SHA: ${{ inputs.source_sha }}
           ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_SHA: ${{ inputs.connect_sha }}
         run: |
           set -euo pipefail
-          [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]]
-          [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]]
-          [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
+          [[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
+          [[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
+          [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
           [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
+          [[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
+          [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
+          [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
+          [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
+          [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
+          run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
+          jq -e --arg source "$SOURCE_SHA" '
+            .head_sha == $source and .head_branch == "main"
+            and .head_repository.full_name == "rustfs/rustfs"
+            and .name == "Build and Release" and .path == ".github/workflows/build.yml"
+            and .status == "completed" and .conclusion == "success"
+          ' <<<"$run" >/dev/null
+          jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
+          jq -e '
+            [.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
+            | ($matches | length) == 1 and $matches[0].conclusion == "success"
+          ' <<<"$jobs" >/dev/null
+          artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
+          jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
+            .workflow_run.id == $run and .workflow_run.head_sha == $source
+            and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
+            and .digest == $digest and .expired == false
+            and (.expires_at | fromdateiso8601) > now
+            and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
+          ' <<<"$artifact" >/dev/null
+          jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
+          gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
+          [[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
+          printf '%s  official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
 
-          run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
-          [[ $(jq -r '.conclusion' <<<"$run") == success ]]
-          [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
-          [[ $(jq -r '.head_branch' <<<"$run") == main ]]
-          [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
-          [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
-
-          artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
-          [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
-          [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
-          [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
-          [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]]
-          [[ $(jq -r '.expired' <<<"$artifact") == false ]]
-
-      - name: Download exact build artifact
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-        with:
-          artifact-ids: ${{ inputs.artifact_id }}
-          path: artifact
-          run-id: ${{ inputs.build_run_id }}
-          github-token: ${{ github.token }}
-
-      - name: Extract exact RustFS binary
-        shell: bash
-        env:
-          SOURCE_SHA: ${{ inputs.source_sha }}
-        run: |
-          set -euo pipefail
-          short_sha=${SOURCE_SHA:0:7}
-          package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
-          [[ -n "$package" ]]
-          mkdir -p binary
-          unzip -qq "$package" rustfs -d binary
-          chmod +x binary/rustfs
-
-      - name: Run top.disk acceptance
+      - name: Extract only the verified official service binary
         shell: bash
         env:
           SOURCE_SHA: ${{ inputs.source_sha }}
           BINARY_SHA256: ${{ inputs.binary_sha256 }}
         run: |
           set -euo pipefail
-          scripts/ci/check_connect_top_disk_artifact.sh \
-            binary/rustfs "$SOURCE_SHA" "$BINARY_SHA256" \
-            top-disk-runtime-evidence.json top-disk.zip
+          python3 - <<'PY'
+          import io
+          import os
+          import pathlib
+          import shutil
+          import stat
+          import zipfile
 
-      - name: Bind workflow and artifact provenance
+          def regular(entry):
+              mode = entry.external_attr >> 16
+              return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
+
+          package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
+          with zipfile.ZipFile('official-artifact.zip') as outer:
+              entries = outer.infolist()
+              assert 1 <= len(entries) <= 16
+              assert len({entry.filename for entry in entries}) == len(entries)
+              assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
+              assert sum(entry.file_size for entry in entries) <= 2147483648
+              assert outer.testzip() is None
+              with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
+                  entries = inner.infolist()
+                  expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
+                  assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
+                  assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
+                  assert inner.testzip() is None
+                  pathlib.Path('binary').mkdir()
+                  with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
+                      header = source.read(20)
+                      assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
+                      destination.write(header)
+                      shutil.copyfileobj(source, destination)
+          PY
+          chmod 0755 binary/rustfs
+          printf '%s  binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
+
+      - name: Run the real top.disk service job through proxy
+        shell: bash
+        env:
+          SOURCE_SHA: ${{ inputs.source_sha }}
+          CONNECT_E2E_PRODUCER_SOURCE: ${{ inputs.source_sha }}
+          CONNECT_E2E_PRODUCER_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_E2E_PRODUCER_ARTIFACT_ID: ${{ inputs.artifact_id }}
+        run: |
+          set -euo pipefail
+          git -C connect-harness diff --exit-code
+          printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
+          changed=$(git -C connect-harness diff --name-only)
+          [[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
+          npm --prefix connect-harness/web ci
+          connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
+          make -C connect-harness e2e-connected-dispatch-check
+          RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
+            RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
+            CONNECT_E2E_TOP_DISK_PROXY_EVIDENCE="$GITHUB_WORKSPACE/top-disk-proxy-service-job-evidence.json" \
+            make -C connect-harness e2e-connected E2E_SCENARIO=top-disk-proxy
+
+      - name: Bind and validate sanitized evidence
         shell: bash
         env:
           BUILD_RUN_ID: ${{ inputs.build_run_id }}
           ARTIFACT_ID: ${{ inputs.artifact_id }}
           ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
+          BINARY_SHA256: ${{ inputs.binary_sha256 }}
+          CONNECT_SHA: ${{ inputs.connect_sha }}
+          SOURCE_SHA: ${{ inputs.source_sha }}
         run: |
           set -euo pipefail
-          jq \
-            --arg workflowRunId "$GITHUB_RUN_ID" \
-            --arg buildRunId "$BUILD_RUN_ID" \
-            --arg artifactId "$ARTIFACT_ID" \
-            --arg artifactDigest "$ARTIFACT_DIGEST" \
-            '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest}' \
-            top-disk-runtime-evidence.json >top-disk-runtime-evidence.bound.json
-          mv top-disk-runtime-evidence.bound.json top-disk-runtime-evidence.json
+          jq -e --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" --arg artifact "$ARTIFACT_ID" '
+            .scenario == "top-disk-proxy" and .topDisk.status == "PASS"
+            and .topDisk.officialArtifactId == $artifact and .topDisk.producerSource == $source
+            and .topDisk.producerSha256 == $binary
+            and .topDisk.result.provenance.sourceCommit == $source
+            and .topDisk.result.provenance.executableSha256 == $binary
+            and .topology.agentsInternalOnly == true and .topology.proxyOnlyBridge == true
+            and .topology.proxyCredentialsFromFiles == true
+            and .probes.direct.connectStatus == "000"
+            and .probes.noAuthentication.connectStatus == "407"
+            and .probes.deniedTarget.connectStatus == "403"
+            and .probes.wrongCa.connectStatus == "200" and .probes.wrongCa.exitCode == 60
+            and .probes.approved.connectStatus == "200" and .probes.approved.exitCode == 0
+            and .proxyLogCounts.approvedTunnels >= 3
+            and .ingress.jobFetchObserved == true
+            and .ingress.bundleReservationObserved == true
+            and .ingress.boundBundleCompletionObserved == true
+            and .ingress.jobResultObserved == true
+          ' top-disk-proxy-service-job-evidence.json >/dev/null
+          jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
+            --arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
+            --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" \
+            --arg expiry "$(cat artifact-expires-at)" '
+            . + {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry,connectSha:$connect,sourceSha:$source,binarySha256:$binary}
+          ' top-disk-proxy-service-job-evidence.json > top-disk-proxy-service-job-evidence.bound.json
+          mv top-disk-proxy-service-job-evidence.bound.json top-disk-proxy-service-job-evidence.json
 
-      - name: Upload acceptance evidence
+      - name: Upload only verified sanitized evidence
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
-          name: connect-top-disk-evidence-${{ github.run_id }}
-          path: |
-            top-disk-runtime-evidence.json
-            top-disk.zip
+          name: connect-top-disk-proxy-service-job-evidence-${{ github.run_id }}
+          path: top-disk-proxy-service-job-evidence.json
           retention-days: 14
           if-no-files-found: error

.github/workflows/connect-top-locks-acceptance.yml

@@ -49,7 +49,7 @@ permissions:
 jobs:
   top-locks:
     name: Verify native Linux x86_64 top.locks service job
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 45
     steps:
       - name: Checkout acceptance harness
@@ -76,7 +76,7 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 
@@ -107,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/connect-top-rpc-acceptance.yml

@@ -49,7 +49,7 @@ permissions:
 jobs:
   top-rpc:
     name: Verify native Linux x86_64 top.rpc service job
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 45
     steps:
       - name: Checkout acceptance harness
@@ -76,7 +76,7 @@ jobs:
       - name: Set up Node.js
         uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
-          node-version: 22.22.2
+          node-version: 25.8.1
           cache: npm
           cache-dependency-path: connect-harness/web/package-lock.json
 
@@ -107,7 +107,7 @@ jobs:
           [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
           [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
 
-          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
+          expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
           jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
           jq -e --arg name "$expected_job" '
             [.[].jobs[] | select(.name == $name)] as $matches

.github/workflows/coverage.yml

@@ -128,7 +128,7 @@ jobs:
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/docker.yml

@@ -83,7 +83,7 @@ jobs:
       (github.event.workflow_run.conclusion == 'success' &&
        github.event.workflow_run.event == 'push' &&
        github.event.workflow_run.head_branch != 'main')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
@@ -501,7 +501,7 @@ jobs:
     name: Scan Docker Images
     needs: [ build-check, build-docker ]
     if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     permissions:
       contents: read
@@ -559,7 +559,7 @@ jobs:
     name: Sync Docker Hub Description
     needs: [ build-check, build-docker ]
     if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Checkout repository
@@ -604,7 +604,7 @@ jobs:
     name: Docker Build Summary
     needs: [ build-check, build-docker ]
     if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Docker build completion summary

.github/workflows/e2e-distributed.yml

@@ -200,7 +200,7 @@ jobs:
     name: Alert on scheduled failure
     needs: [distributed]
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-replication-nightly.yml

@@ -230,7 +230,7 @@ jobs:
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-s3tests.yml

@@ -371,7 +371,7 @@ jobs:
     name: Upstream HEAD classification canary
     if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
     continue-on-error: true
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 20
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -453,7 +453,7 @@ jobs:
     # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
     # dispatch failures are already watched by a human.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/e2e-upgrade.yml

@@ -90,7 +90,7 @@ jobs:
             cache_key: e2e-multipart-layout-baseline
             test: rc5_baseline_replicates_multipart_layouts
             artifact: multipart-layout-baseline
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"

.github/workflows/functional-chain-health.yml

@@ -11,12 +11,44 @@ concurrency:
   cancel-in-progress: false
 jobs:
   collect:
-    runs-on: ubuntu-latest
+    runs-on: smoke-testing
     timeout-minutes: 15
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
         with:
           persist-credentials: false
+      - name: Ensure GitHub CLI
+        shell: bash
+        run: |
+          if command -v gh >/dev/null 2>&1; then
+            gh --version
+            exit 0
+          fi
+
+          gh_version="2.101.0"
+          case "$(uname -m)" in
+            x86_64) gh_arch="amd64" ;;
+            aarch64|arm64) gh_arch="arm64" ;;
+            *)
+              echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
+              exit 1
+              ;;
+          esac
+
+          gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
+          rm -rf "$gh_install_root"
+          mkdir -p "$gh_install_root/bin"
+          archive="$gh_install_root/gh.tar.gz"
+          curl --fail --location --retry 3 \
+            "https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
+            --output "$archive"
+          tar -xzf "$archive" -C "$gh_install_root"
+          install -m 0755 \
+            "$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
+            "$gh_install_root/bin/gh"
+          echo "$gh_install_root/bin" >> "$GITHUB_PATH"
+          export PATH="$gh_install_root/bin:$PATH"
+          gh --version
       - name: Collect and publish verified chain health
         env:
           GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}

.github/workflows/fuzz.yml

@@ -60,7 +60,7 @@ jobs:
   cancel-closed-pr-runs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Explain cancellation run
@@ -240,7 +240,7 @@ jobs:
     # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
     # ci-8); PR and manual dispatch failures are already watched by a human.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/helm-package.yml

@@ -31,7 +31,7 @@ permissions:
 
 jobs:
   build-helm-package:
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     if: |
       (github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
@@ -48,9 +48,10 @@ jobs:
       chart_version: ${{ steps.version.outputs.chart_version }}
 
     steps:
-      - name: Checkout helm chart repo
+      - name: Checkout release source
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
         with:
+          ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
           persist-credentials: false
 
       # Both inputs reach the shell through env rather than `${{ }}`
@@ -83,6 +84,40 @@ jobs:
 
           ./scripts/helm_chart_version.sh "$RAW_TAG"
 
+      # Docker and Helm share the binary-build trigger. Do not publish a chart
+      # until the image selected by its default appVersion is pullable.
+      - name: Wait for release image
+        timeout-minutes: 20
+        env:
+          APP_VERSION: ${{ steps.version.outputs.app_version }}
+        run: |
+          set -euo pipefail
+          if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
+             [[ "$APP_VERSION" == *-preview* ]]; then
+            echo "Invalid deliverable image version: $APP_VERSION" >&2
+            exit 1
+          fi
+
+          for attempt in {1..60}; do
+            if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
+              'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
+              | jq -er '.token // empty') &&
+              curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
+                --header "Authorization: Bearer $token" \
+                --header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
+                "https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
+              echo "Release image is available: rustfs/rustfs:$APP_VERSION"
+              exit 0
+            fi
+            echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
+            if (( attempt < 60 )); then
+              sleep 15
+            fi
+          done
+
+          echo "Release image is unavailable; refusing to publish Helm chart" >&2
+          exit 1
+
       - name: Replace chart version and app version
         env:
           CHART_VERSION: ${{ steps.version.outputs.chart_version }}
@@ -114,7 +149,7 @@ jobs:
           retention-days: 1
 
   publish-helm-package:
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     needs: [ build-helm-package ]
     if: needs.build-helm-package.result == 'success'

.github/workflows/issue-translator.yml

@@ -32,7 +32,7 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7

.github/workflows/minio-interop.yml

@@ -62,7 +62,7 @@ jobs:
     name: MinIO interop (EC + SSE read parity)
     # Skip on forks: needs the repo's runners and is not a contributor gate.
     if: github.repository == 'rustfs/rustfs'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 40
     env:
       # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
@@ -131,7 +131,7 @@ jobs:
     name: Alert on scheduled failure
     needs: [minio-interop]
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/mint.yml

@@ -114,7 +114,7 @@ jobs:
     # label was abandoned (no docker.sock on the pod its only run landed on).
     # TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
     # run) is added by the ci-8 composite action; do not implement it here.
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 120
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -258,7 +258,7 @@ jobs:
     # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
     # dispatch failures are already watched by a human.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/nightly-gnu.yml

@@ -44,7 +44,7 @@ env:
 jobs:
   resolve-source:
     name: Resolve nightly source
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     outputs:
       source_sha: ${{ steps.source.outputs.sha }}
@@ -360,9 +360,12 @@ jobs:
   # live scripts (AppRole login, three-node Raft leader failover).
   #
   # GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
-  # fleet: the HA failover script needs a working Docker daemon, and the
-  # self-hosted fleet is heterogeneous — a docker-dependent workflow has been
-  # burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
+  # fleet: both lanes need a working Docker daemon, and the sm-standard-2 pool
+  # does not provide one. #8112 rerouted these lanes there and both died in
+  # seconds on "Docker is not available" while the package itself built and
+  # published fine, which kept the nightly chain from firing (its gate
+  # requires a successful build run). See also the banner in e2e-s3tests.yml
+  # and rustfs/backlog#1149.
   kms-vault-lane:
     needs: resolve-source
     name: KMS live Vault lane
@@ -495,7 +498,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/nix-flake-update.yml

@@ -33,7 +33,7 @@ concurrency:
 jobs:
   update-flake:
     name: Update flake.lock
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 90
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"

.github/workflows/nix.yml

@@ -54,7 +54,7 @@ jobs:
   cancel-closed-pr-runs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - name: Explain cancellation run
@@ -63,7 +63,7 @@ jobs:
   nix-validation:
     name: Nix Build & Check
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 60
     permissions:
       contents: read

.github/workflows/oidc-keycloak.yml

@@ -51,7 +51,7 @@ concurrency:
 jobs:
   oidc-keycloak-live:
     name: OIDC Keycloak live gate
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
@@ -95,7 +95,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (needs.oidc-keycloak-live.result == 'failure' || needs.oidc-keycloak-live.result == 'cancelled')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/on-demand-migration-interop.yml

@@ -72,7 +72,7 @@ jobs:
     # Skip on forks: needs this repository's runners and is not a contributor
     # gate.
     if: github.repository == 'rustfs/rustfs'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 90
     env:
       NO_PROXY: 127.0.0.1,localhost
@@ -183,7 +183,7 @@ jobs:
   cloud-source:
     name: ${{ matrix.provider }} source (three-case minimum)
     if: github.repository == 'rustfs/rustfs'
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 45
     strategy:
       fail-fast: false
@@ -300,7 +300,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/package.yml

@@ -77,7 +77,7 @@ jobs:
       (github.event.workflow_run.conclusion == 'success' &&
        github.event.workflow_run.event == 'push' &&
        github.event.workflow_run.head_branch != 'main')
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
     timeout-minutes: 10
     outputs:
       version: ${{ steps.resolve.outputs.version }}
@@ -229,7 +229,7 @@ jobs:
   package:
     name: Package (${{ matrix.arch }}-${{ matrix.libc }})
     needs: resolve
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
     timeout-minutes: 30
     strategy:
       fail-fast: false
@@ -619,7 +619,7 @@ jobs:
     name: Summary
     needs: [ resolve, package ]
     if: always()
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 5
     steps:
       - name: Print summary

.github/workflows/performance-ab.yml

@@ -366,7 +366,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/runner-hygiene.yml

@@ -43,7 +43,7 @@ concurrency:
 jobs:
   check-ephemerality:
     name: Check runner ephemerality
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 15
     steps:
       - name: Checkout repository
@@ -66,7 +66,7 @@ jobs:
     # scheduled runs file a tracking issue, manual dispatch stays quiet so
     # debugging never produces a spurious alert.
     if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/rustfs-fault-tolerance-matrix.yml

@@ -12,7 +12,7 @@
 #                        assert the expected outage
 #   M-<topo>EC<P>-rec    after restoring every node, reads/writes succeed
 #
-# 10 topologies x EC levels = 38 combinations, 320 cases. The full sweep runs
+# 9 topologies x EC levels = 36 combinations, 300 cases. The full sweep runs
 # 3-5 hours and is dispatched on demand (e.g. after a nightly chain goes
 # green), not nightly. Case ids and expectations come from product source
 # (default_parity_count, erasure set sizing) via rustfs/auto-testing.
@@ -32,7 +32,7 @@ on:
         default: main
         type: string
       matrix_topo:
-        description: 'Topology filter like 4x2; empty sweeps all ten topologies'
+        description: 'Topology filter like 4x2; empty sweeps all nine topologies'
         required: false
         default: ''
         type: string
@@ -174,12 +174,12 @@ jobs:
           } > "${REPORT_FILE}"
           cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
 
-          # An unfiltered sweep must land exactly 320 cases (38 topology x EC
+          # An unfiltered sweep must land exactly 300 cases (36 topology x EC
           # combinations at 2*(nodes+1) cases each); filtered sweeps are held
           # to internal consistency. Missing/harness-error rows always fail.
           EXPECTED_TOTAL=""
           if [ -z "${{ inputs.matrix_topo }}" ] && [ -z "${MATRIX_EC}" ]; then
-            EXPECTED_TOTAL=320
+            EXPECTED_TOTAL=300
           fi
           jq -e --arg testing_sha "${AUTO_TESTING_SHA}" --argjson expected_total "${EXPECTED_TOTAL:-null}" '
             .schema_version == 1 and

.github/workflows/rustfs-functional-chain.yml

@@ -42,7 +42,7 @@ concurrency:
 jobs:
   prepare:
     if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
-    runs-on: ubuntu-latest
+    runs-on: smoke-testing
     timeout-minutes: 10
     outputs:
       manifest: ${{ steps.candidate.outputs.manifest }}
@@ -188,7 +188,7 @@ jobs:
   complete-chain:
     needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
     if: ${{ always() && needs.prepare.result != 'skipped' }}
-    runs-on: ubuntu-latest
+    runs-on: smoke-testing
     timeout-minutes: 10
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7

.github/workflows/rustfs-performance-test.yml

@@ -90,7 +90,7 @@ env:
 jobs:
   runner-check:
     if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 5
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7

.github/workflows/rustfs-pool-expand-test.yml

@@ -7,6 +7,11 @@ on:
         description: Verified candidate and chain attempt from the chain driver
         type: string
         required: true
+      pool_timeout_minutes:
+        description: Wall-clock budget (minutes) for the pool run step
+        type: string
+        required: false
+        default: '240'
   workflow_dispatch:
     inputs:
       rustfs_version:
@@ -39,6 +44,10 @@ on:
         description: 'Run the pool decommission step (3-pool topology only)'
         type: boolean
         default: true
+      pool_timeout_minutes:
+        description: 'Wall-clock budget (minutes) for the whole pool run: fill + rebalance wait + decommission'
+        required: false
+        default: '240'
       cleanup_before:
         description: 'Reset the nodes before the test (DESTROYS existing data/config)'
         type: boolean
@@ -82,7 +91,8 @@ jobs:
   pool-expansion-test:
     name: Pool expansion / decommission test
     runs-on: smoke-testing
-    timeout-minutes: 60
+    # Leave setup and finalizer headroom beyond the default 240-minute test.
+    timeout-minutes: 360
     if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
     env:
       RUSTFS_POOL_ADMIN_ENDPOINT: ${{ secrets.RUSTFS_POOL_ADMIN_ENDPOINT || vars.RUSTFS_POOL_ADMIN_ENDPOINT || 'http://rustfs-node1:9000' }}
@@ -251,7 +261,12 @@ jobs:
 
       - name: Run pool expansion & decommission test
         id: pool_test
-        timeout-minutes: 45
+        # The suite budgets up to 24h each for rebalance and decommission
+        # (REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh);
+        # the previous 45m step cap killed four consecutive runs mid-rebalance
+        # with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
+        # in ~21m) plus one decommission pass; override per dispatch.
+        timeout-minutes: ${{ inputs.pool_timeout_minutes || 240 }}
         # Case failures keep the run green: the report and the backlog
         # issue manager carry the product signal.
         continue-on-error: true
@@ -568,6 +583,25 @@ jobs:
             echo "pool test script reported a failure (step FAIL verdict or die() abort)" >&2
             failed=1
           fi
+          # A runner-level kill (step timeout) truncates the suite without a
+          # FAIL verdict or an [ERROR] line, so require every expected
+          # [POOL-STEP] marker to be present with PASS - missing markers mean
+          # the run was truncated, not green. This also closes the gap for
+          # standalone dispatches, where the chain driver's evidence
+          # validation does not run.
+          last_step=6
+          if [ "${{ inputs.pools || '3' }}" = "3" ]; then
+            last_step=8
+            if [ "${{ inputs.run_decommission != 'false' }}" = "true" ]; then
+              last_step=9
+            fi
+          fi
+          for n in $(seq 1 "${last_step}"); do
+            grep -q "^\[POOL-STEP\] ${n} .* PASS" "${POOL_ARTIFACT_DIR}/pool-test.log" 2>/dev/null || {
+              echo "pool test log is missing [POOL-STEP] ${n} PASS (expected through step ${last_step}); the run was truncated" >&2
+              failed=1
+            }
+          done
           [ "${failed}" -eq 0 ] || exit 1
 
       - name: Upload functional report to dashboard

.github/workflows/schedule-failure-alert-drill.yml

@@ -44,7 +44,7 @@ permissions:
 jobs:
   forced-failure:
     name: Forced failure
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 5
     steps:
       - name: Fail on purpose
@@ -58,7 +58,7 @@ jobs:
     # Mirrors the consumer wiring, minus the schedule-event guard (this
     # workflow is dispatch-only by design).
     if: always() && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

.github/workflows/scheduled-validation-freshness.yml

@@ -29,6 +29,9 @@ concurrency:
 jobs:
   check-freshness:
     name: Check scheduled validation freshness
+    # Stay on GitHub-hosted runners: this monitor must keep running when the
+    # self-hosted sm-standard-* pool is down, which is exactly when the
+    # scheduled validations it watches stall.
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/scheduled-validation-watchdog.yml

@@ -42,6 +42,9 @@ jobs:
       github.event.workflow_run.event == 'schedule' &&
       github.event.workflow_run.conclusion != 'success' &&
       github.event.workflow_run.conclusion != 'failure'
+    # Stay on GitHub-hosted runners: this monitor must keep running when the
+    # self-hosted sm-standard-* pool is down, which is exactly when the
+    # scheduled validations it watches stall.
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/stale.yml

@@ -26,7 +26,7 @@ on:
 
 jobs:
   stale:
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 30
     steps:
       - uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0

.github/workflows/star-history.yml

@@ -14,7 +14,7 @@ concurrency:
 
 jobs:
   update:
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     steps:
       - uses: overtrue/repo-visuals-action@ee2c632f6ce617e851fb46ea935ee8af762ebb93 # v1.4.0

.github/workflows/targets-integration.yml

@@ -41,7 +41,7 @@ env:
 jobs:
   targets-live:
     name: PostgreSQL, MySQL, AMQP, and NATS
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 90
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
@@ -177,7 +177,7 @@ jobs:
     if: >-
       always() && github.event_name == 'schedule' &&
       (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-2
     timeout-minutes: 10
     permissions:
       contents: read

CHANGELOG.md

@@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.
 
 ### Fixed
+- **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117.
 - **Vault backends accepted key names that escape the key prefix**: only the Local backend refused a key identifier containing a path separator, so on Vault KV2 `POST /rustfs/admin/v3/kms/keys` with a name such as `bad/name` succeeded and created a nested KV2 path that the key listing then reported as a directory rather than a key, and an identifier containing `..` addressed a record outside the configured key prefix once the HTTP client normalised the URL. Both Vault backends now apply the Local backend's containment rule at the point where the identifier becomes a path or a Transit key name: an empty identifier, one containing `/`, `\\` or NUL, or the dot segments `.` and `..` is refused with `400` (`InvalidKey`) before any request reaches Vault. The AWS backend is unaffected, since it addresses keys by ARN and alias. A Vault deployment that created such a key on an earlier build must re-create it under a plain name; objects encrypted under the old name stay readable only until that key is refused, so migrate them first. Refs rustfs/backlog#2474.
 - **Fresh multi-pool bootstrap with distinct format creators**: a new deployment whose pools have their first endpoint on different nodes (for example two single-node pools) could never publish its initial `pool.bin`: each node held fresh-bootstrap proof only for the pool it formatted, the deployment-wide proof collapsed to none, and every node died with `pool metadata recovery required: no durable bootstrap identity or pool.bin replica is available` after the startup retry budget. The first pool's creator now mints the pending cluster identity on its own pool, every other creator copies that nonce-bound identity onto the pool it formatted first-hand, and the elected writer publishes `pool.bin` once every pool replica carries the same pending identity. Corrupt or disagreeing replicas, pools that merely have a format, expansion pools joining an initialized deployment, and restarts without first-hand proof still fail closed. Non-elected nodes that start before `pool.bin` exists, and the elected writer while it waits for the other creators, no longer latch their pool-metadata write gate for the life of the process. Refs rustfs/backlog#2338, rustfs/backlog#2375.
 - **Lock RPC timeout storms** (#7363): the remote lock client no longer evicts and re-dials the shared internode HTTP/2 channel on every request deadline. A timeout evicts only when the peer has not completed any lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (`RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS`, default 5 s), and a timed-out request is left running instead of being reset (bounded per peer by `RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT`, default 256), so a slow lock endpoint can no longer drive the `RST_STREAM`/`GOAWAY too_many_resets`/reconnect loop. A lock granted after its caller timed out is released immediately, and unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New `rustfs_remote_lock_*` metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer. Operator guide at `docs/operations/lock-rpc-storm-protection.md`.

Cargo.toml

@@ -20,6 +20,7 @@ members = [
     "crates/common", # Shared utilities and data structures
     "crates/data-usage", # Shared data usage models and algorithms
     "crates/config", # Configuration management
+    "crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
     "crates/credentials", # Credential management system
     "crates/crypto", # Cryptography and security features
     "crates/ecstore", # Erasure coding storage implementation
@@ -156,9 +157,9 @@ futures-lite = "2.6.1"
 futures-util = "0.3.34"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.9.0" }
-lapin = { default-features = false, version = "4.11.0" }
+lapin = { default-features = false, version = "4.12.0" }
 hyper = { version = "1.11.1" }
-hyper-rustls = { default-features = false, version = "0.27.9" }
+hyper-rustls = { default-features = false, version = "0.27.10" }
 hyper-util = { version = "0.1.20" }
 http = "1.5.0"
 http-body = "1.1.0"
@@ -244,13 +245,13 @@ atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.12.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-kms = { default-features = false, version = "1.120.0" }
-aws-sdk-s3 = { default-features = false, version = "1.148.0" }
-aws-sdk-sts = { default-features = false, version = "1.116.0" }
+aws-sdk-kms = { default-features = false, version = "1.121.0" }
+aws-sdk-s3 = { default-features = false, version = "1.149.0" }
+aws-sdk-sts = { default-features = false, version = "1.117.0" }
 aws-smithy-async = { version = "1.3.0" }
 aws-smithy-http-client = { default-features = false, version = "1.4.2" }
-aws-smithy-runtime-api = { version = "1.16.2" }
-aws-smithy-types = { version = "1.6.4" }
+aws-smithy-runtime-api = { version = "1.17.0" }
+aws-smithy-types = { version = "1.8.0" }
 base64-simd = "0.8.0"
 brotli = "9.0.0"
 clap = { version = "4.6.7" }
@@ -265,7 +266,7 @@ datafusion = { default-features = false, version = "55.1.0" }
 derive_builder = "0.20.2"
 ed25519-dalek = "3.0.0"
 enumset = "1.1.14"
-faster-hex = "0.10.0"
+faster-hex = "1.0.0"
 flate2 = "1.1.10"
 glob = "0.3.4"
 google-cloud-storage = "1.18.0"
@@ -301,7 +302,7 @@ percent-encoding = "2.3.2"
 qrcode-rs = { version = "2.1.1", default-features = false, features = ["std", "svg"] }
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
-rand = { version = "0.10.2" }
+rand = { version = "0.10.3" }
 ratelimit = "2.0.0"
 rayon = "1.12.0"
 rustfs-erasure-codec = { version = "9.0.0" }
@@ -313,8 +314,8 @@ rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.5" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6", features = ["minio"] }
-s3s-sigv4 = { git = "https://github.com/s3s-project/s3s.git", rev = "50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6" }
+s3s = { version = "0.17.0", features = ["minio"] }
+s3s-sigv4 = "0.17.0"
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
@@ -326,8 +327,8 @@ strum = { version = "0.28.0" }
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
-test-case = "3.3.1"
-thiserror = "2.0.20"
+test-case = "3.4.0"
+thiserror = "2.0.21"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
@@ -351,7 +352,7 @@ zstd = "0.14.0"
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
-dial9 = { version = "0.5.1", default-features = false }
+dial9 = { version = "0.5.2", default-features = false }
 opentelemetry = { version = "0.32.0" }
 opentelemetry-appender-tracing = { version = "0.32.0" }
 opentelemetry-otlp = { version = "0.32.0" }
@@ -364,7 +365,7 @@ pyroscope = { version = "2.1.1" }
 # FTP and SFTP
 libunftp = { version = "0.23.0" }
 unftp-core = "0.1.0"
-suppaftp = { version = "12.0.0" }
+suppaftp = { version = "12.0.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
 russh = { version = "0.63.3" }
 russh-sftp = "3.0.0"
@@ -373,7 +374,7 @@ russh-sftp = "3.0.0"
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-rustfs-mimalloc = { version = "0.5.4" }
+rustfs-mimalloc = { version = "0.5.6" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
 hotpath = { version = "0.26.0", default-features = false }
 # Snapshot testing for output format regression detection

README.md

@@ -141,7 +141,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command

README_ZH.md

@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：

crates/concurrency/src/workload.rs

@@ -96,11 +96,11 @@ pub struct WorkloadAdmissionSnapshot {
     pub class: WorkloadClass,
     /// Current admission state.
     pub state: AdmissionState,
-    /// Active work count when the owner exposes it.
+    /// Active work count or allocated/reserved permit units, as defined by the owner.
     pub active: Option<usize>,
     /// Queued work count when the owner exposes it.
     pub queued: Option<usize>,
-    /// Admission limit when the owner exposes it.
+    /// Admission limit in the same units as `active`, when the owner exposes it.
     pub limit: Option<usize>,
     /// Optional state reason for disabled, throttled, saturated, or unknown states.
     pub reason: Option<String>,

crates/config/README.md

@@ -169,15 +169,17 @@ Scanner cycle budget controls:
 ## Foreground write admission environment variables
 
 Large direct `PutObject` requests and multipart `UploadPart` requests share one
-per-process permit pool that bounds how many bodies are ingested and written
-concurrently. Small direct PUTs stay on the legacy path.
+per-process permit pool that bounds concurrent body ingest and storage writes.
+Small direct PUTs stay on the legacy path.
 
 - `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE`
   - enables the default-on pool; `false` keeps only the soft request counter.
   - default is `true`.
 - `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT`
-  - permits in the pool; `0` derives half of `RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS`, clamped to `32`.
-  - default is `0` (32 permits at stock settings).
+  - a positive value is an exact request-count limit shared by all gated writes.
+  - default is `0`: derive half of `RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS`, clamped to `32`, as large-write slots. Each slot has eight units. A gated direct PUT or unknown-size multipart part uses eight units; known-size parts use one unit per 8 MiB, rounded up, with a minimum of one and maximum of eight.
+  - stock settings therefore share 256 units across at most 32 large/unknown writes or 256 parts of up to 8 MiB. Mixed sizes consume the same budget. Admitted known parts of up to 64 MiB total at most 2 GiB of declared payload; larger streaming writes and queued bodies are separate. This is not a bound on total process or kernel memory.
+  - admission snapshots report allocated or reserved units in `active` and the unit budget in `limit` under automatic sizing. Explicit and strict limits still report request-count permits.
 - `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES`
   - smallest direct `PutObject` that takes a permit; unknown-size requests always do.
   - default is `33554432` (32 MiB).
@@ -193,8 +195,8 @@ concurrently. Small direct PUTs stay on the legacy path.
   - RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. Keep it with margin below the shortest such timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
 - `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING`
   - maximum `UploadPart` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
-  - default is `0`, which derives 16 times the permit limit (512 at stock settings).
-  - each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory), so this depth also bounds that memory. RustFS leaves the receive buffer to kernel autotuning (see `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES` below), which keeps an unread connection at the kernel's initial size (128 KiB on current Linux).
+  - default is `0`, which derives 16 times the large-write slot count, or the explicit request-count limit (512 at stock settings). Automatic subdivision does not enlarge this queue.
+  - each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory). Autotuned receive buffers can remain large on reused connections; queue depth does not imply a fixed per-connection memory cost.
 - `RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE`, `RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT`, `RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
   - experimental strict gate that applies to every foreground write regardless of size and replaces the pool above when enabled.
   - default is disabled; enabling it with limit `0` disables foreground write admission entirely.
@@ -203,9 +205,9 @@ concurrently. Small direct PUTs stay on the legacy path.
 
 - `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES`
   - fixed `SO_RCVBUF` for the API listener, inherited by every accepted socket; `0` leaves the receive buffer to kernel autotuning.
-  - default is `0`. Earlier releases hard-coded 4 MiB, which Linux doubles to 8 MiB and which disables autotuning, so every connection whose body was not being read yet (a multipart part queued for a foreground write permit) could accumulate up to 8 MiB of unread body in kernel memory; at SDK-default multipart concurrency that was enough to push a node into TCP memory pressure.
-  - with autotuning the per-connection receive ceiling is the kernel's (`net.ipv4.tcp_rmem` max, 6 MiB on stock Linux) instead of the former fixed 8 MiB, so a single very high-bandwidth-delay connection may see a somewhat lower ceiling; raise `net.ipv4.tcp_rmem` first, and set this variable only on kernels without receive-buffer autotuning (illumos/Solaris) or where the sysctl cannot be changed.
-  - the send buffer stays fixed at 4 MiB because the stock Linux send autotuning ceiling (`net.ipv4.tcp_wmem` max, 4 MiB) is lower than a GB-level response stream needs.
+  - default is `0`. A fixed buffer disables receive autotuning. Linux doubles the requested value for socket-memory accounting, subject to kernel limits; that capacity is not a measurement of queued payload or allocated memory.
+  - with autotuning the receive ceiling is controlled by the kernel (`net.ipv4.tcp_rmem` on Linux); its defaults vary with kernel version and memory. A connection can retain a buffer enlarged by previous requests while its next request is queued. Tune the kernel ceiling first, and set this variable only on kernels without receive-buffer autotuning (illumos/Solaris) or where the sysctl cannot be changed.
+  - the send buffer remains fixed at 4 MiB; this setting only controls the receive side.
 
 ## Remote tier timeout environment variables
 

crates/config/src/constants/object.rs

@@ -364,11 +364,14 @@ const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
 pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
 pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
 
-/// Maximum automatic foreground write requests admitted concurrently per process.
+/// Explicit maximum foreground write requests admitted concurrently per process.
 ///
-/// `0` derives a conservative default from the local disk-read scheduler cap,
-/// currently clamped to protect the commit path without making ordinary high
-/// throughput uploads single-file.
+/// `0` derives up to 32 large-write slots from the local disk-read scheduler.
+/// Each automatic slot has eight units: a gated direct PUT or unknown-size
+/// part uses all eight, while a known-size part uses one unit per 8 MiB,
+/// rounded up and capped at eight. Thus stock settings share one budget across
+/// at most 32 large/unknown writes or 256 parts of up to 8 MiB. A positive
+/// override retains request-count semantics for every gated write.
 pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
 pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
 

crates/config/src/constants/tls.rs

@@ -164,13 +164,10 @@ pub const DEFAULT_HTTP1_MAX_BUF_SIZE: usize = 64 * 1024; // 64 KB
 /// autotuning.
 ///
 /// A fixed `SO_RCVBUF` is inherited by every accepted socket and disables
-/// receive-buffer autotuning, so a connection whose request body is not being
-/// read yet (a multipart part queued for a foreground write permit) lets up to
-/// the fixed size of unread body accumulate in kernel memory — Linux doubles
-/// the requested value, so the former hard-coded 4 MiB held up to 8 MiB per
-/// queued connection (issue #7385). Autotuning keeps an unread connection at
-/// the kernel's initial size and grows only connections that are being
-/// drained. Set this only on kernels without receive-buffer autotuning
+/// receive-buffer autotuning. Queued multipart bodies can accumulate in kernel
+/// memory with either policy: a reused autotuned connection may already have
+/// a large buffer. Socket capacity is not the same as queued payload or actual
+/// memory allocation. Set this only on kernels without receive-buffer autotuning
 /// (illumos/Solaris) or on very high-bandwidth-delay links where the kernel's
 /// autotuning ceiling (`net.ipv4.tcp_rmem` on Linux) is too low and cannot be
 /// raised.

crates/connect-symbol-catalog/Cargo.toml

@@ -0,0 +1,22 @@
+[package]
+name = "rustfs-connect-symbol-catalog"
+version.workspace = true
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+publish = false
+
+[dependencies]
+hex-simd.workspace = true
+object = { version = "=0.39.1", default-features = false, features = ["read", "std", "elf"] }
+serde = { workspace = true, features = ["derive"] }
+serde_json.workspace = true
+sha2.workspace = true
+# Match pyroscope's pprof-rs frame normalization exactly.
+symbolic-demangle = { version = "=13.9.0", default-features = false, features = ["rust"] }
+
+[dev-dependencies]
+object = { version = "=0.39.1", default-features = false, features = ["write", "elf"] }
+
+[lints]
+workspace = true

crates/connect-symbol-catalog/src/main.rs

@@ -0,0 +1,284 @@
+// Copyright 2024 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+//! Release tooling only: derive reviewed CPU symbol candidates from the final ELF.
+
+use object::{Object, ObjectSymbol};
+use serde::Serialize;
+use sha2::{Digest, Sha256};
+use std::collections::BTreeSet;
+use std::fs::{self, OpenOptions};
+use std::io::{self, Read, Write};
+use std::path::Path;
+
+const MAX_BINARY_BYTES: u64 = 1_073_741_824;
+const MAX_CATALOG_BYTES: usize = 32 * 1024 * 1024;
+const MAX_SYMBOLS: usize = 200_000;
+const NORMALIZATION: &str = "pyroscope-2.1.1-symbolic-demangle-13.9.0";
+
+#[derive(Serialize)]
+#[serde(rename_all = "camelCase")]
+struct Catalog<'a> {
+    schema_version: u8,
+    normalization: &'static str,
+    source_commit: &'a str,
+    executable_sha256: String,
+    os_family: &'static str,
+    architecture: &'static str,
+    build_features: Vec<String>,
+    symbols: BTreeSet<String>,
+}
+
+fn invalid(message: &'static str) -> io::Error {
+    io::Error::new(io::ErrorKind::InvalidData, message)
+}
+
+fn symbol_name(raw: &[u8]) -> Option<String> {
+    // This is the same lossy UTF-8 conversion and demangler used by pprof-rs.
+    let name = symbolic_demangle::demangle(&String::from_utf8_lossy(raw)).into_owned();
+    if name.is_empty()
+        || name.len() > 4096
+        || matches!(name.as_str(), "Unknown" | "<unresolved>")
+        || name.bytes().any(|b| b < 32 || b == 127)
+    {
+        return None;
+    }
+    Some(name)
+}
+
+fn catalog(binary: &[u8], source: &str, target: &str, features: Vec<String>) -> io::Result<Vec<u8>> {
+    if source.len() != 40 || !source.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) {
+        return Err(invalid("source commit must be a lowercase full SHA"));
+    }
+    let expected_arch = match target {
+        "x86_64-unknown-linux-gnu" => object::Architecture::X86_64,
+        "aarch64-unknown-linux-gnu" => object::Architecture::Aarch64,
+        _ => return Err(invalid("only supported Linux GNU CPU profiling targets have catalogs")),
+    };
+    if features.is_empty()
+        || features.len() > 64
+        || !features.iter().any(|f| f == "pyroscope")
+        || features.iter().any(|f| {
+            f.is_empty()
+                || f.len() > 64
+                || !f.as_bytes().first().is_some_and(u8::is_ascii_lowercase)
+                || !f
+                    .bytes()
+                    .all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-' || b == b'_')
+        })
+        || features.windows(2).any(|pair| pair[0] >= pair[1])
+    {
+        return Err(invalid("build features must be sorted, unique and include pyroscope"));
+    }
+    let file = object::File::parse(binary).map_err(|_| invalid("cannot parse executable"))?;
+    if file.format() != object::BinaryFormat::Elf
+        || file.architecture() != expected_arch
+        || !matches!(file.kind(), object::ObjectKind::Executable | object::ObjectKind::Dynamic)
+    {
+        return Err(invalid("the executable does not match the Linux target"));
+    }
+    if file.symbol_table().is_none() {
+        return Err(invalid("the executable is stripped; retain its function symbol table"));
+    }
+    let mut symbols = BTreeSet::new();
+    let mut symbol_bytes = 0;
+    for symbol in file.symbols().chain(file.dynamic_symbols()) {
+        if symbol.kind() != object::SymbolKind::Text || symbol.is_undefined() {
+            continue;
+        }
+        if let Some(name) = symbol.name_bytes().ok().and_then(symbol_name)
+            && !symbols.contains(&name)
+        {
+            symbol_bytes += name.len() + 3;
+            if symbol_bytes > MAX_CATALOG_BYTES {
+                return Err(invalid("symbol catalog exceeds its byte limit"));
+            }
+            symbols.insert(name);
+        }
+        if symbols.len() > MAX_SYMBOLS {
+            return Err(invalid("symbol catalog exceeds its reviewed symbol limit"));
+        }
+    }
+    if symbols.is_empty() {
+        return Err(invalid("the executable has no usable function symbols"));
+    }
+    let catalog = Catalog {
+        schema_version: 1,
+        normalization: NORMALIZATION,
+        source_commit: source,
+        executable_sha256: hex_simd::encode_to_string(Sha256::digest(binary), hex_simd::AsciiCase::Lower),
+        os_family: "LINUX",
+        architecture: if expected_arch == object::Architecture::X86_64 {
+            "x86_64"
+        } else {
+            "aarch64"
+        },
+        build_features: features,
+        symbols,
+    };
+    let json = serde_json::to_vec(&catalog).map_err(io::Error::other)?;
+    if json.len() > MAX_CATALOG_BYTES {
+        return Err(invalid("symbol catalog exceeds its byte limit"));
+    }
+    Ok(json)
+}
+
+fn main() -> io::Result<()> {
+    let args: Vec<_> = std::env::args().skip(1).collect();
+    let [binary_path, source, target, features_json, output] = args.as_slice() else {
+        return Err(invalid(
+            "usage: rustfs-connect-symbol-catalog BINARY SOURCE_SHA TARGET FEATURES_JSON OUTPUT",
+        ));
+    };
+    let input = fs::File::open(binary_path)?;
+    if !input.metadata()?.is_file() || input.metadata()?.len() > MAX_BINARY_BYTES {
+        return Err(invalid("input must be a bounded regular executable"));
+    }
+    let mut binary = Vec::new();
+    input.take(MAX_BINARY_BYTES + 1).read_to_end(&mut binary)?;
+    if u64::try_from(binary.len()).map_err(io::Error::other)? > MAX_BINARY_BYTES {
+        return Err(invalid("executable grew beyond its byte limit"));
+    }
+    let features = serde_json::from_str(features_json).map_err(io::Error::other)?;
+    let json = catalog(&binary, source, target, features)?;
+    // Never overwrite another build's reviewed artifact.
+    let mut file = OpenOptions::new().write(true).create_new(true).open(Path::new(output))?;
+    file.write_all(&json)?;
+    file.sync_all()?;
+    println!("{}", hex_simd::encode_to_string(Sha256::digest(&json), hex_simd::AsciiCase::Lower));
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn executable() -> Vec<u8> {
+        let mut object =
+            object::write::Object::new(object::BinaryFormat::Elf, object::Architecture::X86_64, object::Endianness::Little);
+        let section = object.section_id(object::write::StandardSection::Text);
+        object.append_section_data(section, &[0xc3], 1);
+        object.add_symbol(object::write::Symbol {
+            name: b"_ZN4test4main17h0123456789abcdefE".to_vec(),
+            value: 0,
+            size: 1,
+            kind: object::SymbolKind::Text,
+            scope: object::SymbolScope::Linkage,
+            weak: false,
+            section: object::write::SymbolSection::Section(section),
+            flags: object::SymbolFlags::None,
+        });
+        let mut bytes = object.write().expect("write ELF symbol fixture");
+        // The object writer emits ET_REL; this fixture models the final ET_EXEC header.
+        bytes[16..18].copy_from_slice(&2_u16.to_le_bytes());
+        bytes
+    }
+
+    fn bulk_executable(count: usize, suffix_len: usize) -> Vec<u8> {
+        let mut object =
+            object::write::Object::new(object::BinaryFormat::Elf, object::Architecture::X86_64, object::Endianness::Little);
+        let section = object.section_id(object::write::StandardSection::Text);
+        object.append_section_data(section, &[0xc3], 1);
+        let suffix = "x".repeat(suffix_len);
+        for index in 0..count {
+            object.add_symbol(object::write::Symbol {
+                name: format!("rustfs::profile::function_{index:06}_{suffix}").into_bytes(),
+                value: 0,
+                size: 1,
+                kind: object::SymbolKind::Text,
+                scope: object::SymbolScope::Linkage,
+                weak: false,
+                section: object::write::SymbolSection::Section(section),
+                flags: object::SymbolFlags::None,
+            });
+        }
+        let mut binary = object.write().expect("write bulk ELF fixture");
+        binary[16..18].copy_from_slice(&2_u16.to_le_bytes());
+        binary
+    }
+
+    #[test]
+    fn catalog_binds_the_exact_executable_and_normalizes_function_names() {
+        let binary = executable();
+        let encoded =
+            catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).expect("generate catalog");
+        let value: serde_json::Value = serde_json::from_slice(&encoded).expect("catalog JSON");
+        assert_eq!(value["symbols"], serde_json::json!(["test::main"]));
+        assert_eq!(
+            value["executableSha256"],
+            hex_simd::encode_to_string(Sha256::digest(&binary), hex_simd::AsciiCase::Lower)
+        );
+        assert_eq!(value["normalization"], NORMALIZATION);
+        assert!(catalog(&binary, &"a".repeat(40), "aarch64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
+        assert!(catalog(&binary, "bad", "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
+        assert!(catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into(); 2]).is_err());
+    }
+
+    #[test]
+    fn stripped_executables_and_wrong_abis_are_refused() {
+        let mut binary = executable();
+        // Removing section headers models a fully stripped executable without dynsym.
+        binary[40..48].fill(0);
+        binary[60..64].fill(0);
+        let error = catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()])
+            .expect_err("stripped binary must not yield an empty but trusted catalog");
+        assert!(error.to_string().contains("stripped"));
+        assert!(catalog(&executable(), &"a".repeat(40), "x86_64-unknown-linux-musl", vec!["pyroscope".into()]).is_err());
+    }
+
+    #[test]
+    fn symbol_hash_matches_the_connect_nonce_vector() {
+        let nonce: Vec<u8> = (0..32).collect();
+        let name = symbol_name(b"core::hint::black_box").expect("reviewed function name");
+        let mut digest = Sha256::new();
+        digest.update(b"rustfs-connect-cpu-symbol-v1\0");
+        digest.update(nonce);
+        digest.update(name.as_bytes());
+        assert_eq!(
+            hex_simd::encode_to_string(digest.finalize(), hex_simd::AsciiCase::Lower),
+            "5d5b472a0dd09124bfedf0bd243f88d5e61952b8da15d7463da019137320d77c"
+        );
+    }
+
+    #[test]
+    fn unresolved_and_non_function_material_cannot_become_catalog_entries() {
+        assert!(symbol_name(b"Unknown").is_none());
+        assert!(symbol_name(b"<unresolved>").is_none());
+        assert!(symbol_name(b"secret\nname").is_none());
+        assert!(symbol_name(&vec![b'x'; 4097]).is_none());
+        assert!(catalog(b"not ELF", &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
+    }
+
+    #[test]
+    fn release_sized_catalog_retains_all_function_names() {
+        let binary = bulk_executable(110_000, 160);
+        let encoded = catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()])
+            .expect("retain complete bounded symbol catalog");
+        assert!(encoded.len() > 16 * 1024 * 1024);
+        assert!(encoded.len() <= MAX_CATALOG_BYTES);
+        let value: serde_json::Value = serde_json::from_slice(&encoded).expect("catalog JSON");
+        assert_eq!(value["symbols"].as_array().expect("symbol array").len(), 110_000);
+    }
+
+    #[test]
+    fn catalogs_over_either_resource_limit_are_refused() {
+        let source = "a".repeat(40);
+        let too_many = catalog(
+            &bulk_executable(MAX_SYMBOLS + 1, 0),
+            &source,
+            "x86_64-unknown-linux-gnu",
+            vec!["pyroscope".into()],
+        )
+        .expect_err("too many function names must be refused");
+        assert!(too_many.to_string().contains("reviewed symbol limit"));
+
+        let too_large = catalog(
+            &bulk_executable(10_000, 3_500),
+            &source,
+            "x86_64-unknown-linux-gnu",
+            vec!["pyroscope".into()],
+        )
+        .expect_err("oversized catalog must be refused");
+        assert!(too_large.to_string().contains("byte limit"));
+    }
+}

crates/e2e_test/src/get_object_parts_count_test.rs

@@ -0,0 +1,153 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging};
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+use tracing::info;
+
+const PARTS_COUNT_BUCKET: &str = "parts-count-test-bucket";
+const MULTIPART_KEY: &str = "multipart-object.bin";
+const SINGLE_PUT_KEY: &str = "single-put-object.bin";
+
+/// s3's minimum size for every multipart part but the last.
+const PART_SIZE: usize = 5 * 1024 * 1024;
+const LAST_PART_SIZE: usize = 1024;
+const EXPECTED_PARTS: i32 = 3;
+
+#[tokio::test]
+async fn get_object_reports_parts_count_for_multipart_upload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    init_logging();
+    info!("Starting GetObject x-amz-mp-parts-count regression test");
+
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server(Vec::new()).await?;
+
+    let client = env.create_s3_client();
+    env.create_test_bucket(PARTS_COUNT_BUCKET).await?;
+
+    // A genuine multipart upload of three parts.
+    let created = client
+        .create_multipart_upload()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .send()
+        .await?;
+    let upload_id = created.upload_id().expect("upload id");
+
+    let mut completed_parts = Vec::new();
+    for part_number in 1..=EXPECTED_PARTS {
+        let size = if part_number == EXPECTED_PARTS {
+            LAST_PART_SIZE
+        } else {
+            PART_SIZE
+        };
+        let uploaded = client
+            .upload_part()
+            .bucket(PARTS_COUNT_BUCKET)
+            .key(MULTIPART_KEY)
+            .upload_id(upload_id)
+            .part_number(part_number)
+            .body(ByteStream::from(vec![b'a' + (part_number as u8); size]))
+            .send()
+            .await?;
+        completed_parts.push(
+            CompletedPart::builder()
+                .part_number(part_number)
+                .e_tag(uploaded.e_tag().expect("part etag"))
+                .build(),
+        );
+    }
+
+    client
+        .complete_multipart_upload()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .upload_id(upload_id)
+        .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed_parts)).build())
+        .send()
+        .await?;
+
+    let total_len = (PART_SIZE * 2 + LAST_PART_SIZE) as i64;
+
+    // The header the fix is about.
+    let first_part = client
+        .get_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .part_number(1)
+        .send()
+        .await?;
+    assert_eq!(
+        first_part.parts_count(),
+        Some(EXPECTED_PARTS),
+        "GetObject with partNumber must report the object's part count"
+    );
+    assert_eq!(
+        first_part.content_length(),
+        Some(PART_SIZE as i64),
+        "the response body must be part 1, not the whole object"
+    );
+    let expected_range = format!("bytes 0-{}/{}", PART_SIZE - 1, total_len);
+    assert_eq!(
+        first_part.content_range(),
+        Some(expected_range.as_str()),
+        "Content-Range must state part 1's extent and the whole object's size"
+    );
+
+    // Without partNumber s3 omits the header, so a plain GET is unchanged.
+    let whole = client
+        .get_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .send()
+        .await?;
+    assert_eq!(whole.parts_count(), None, "a GET that does not name a part is owed no part count");
+    assert_eq!(whole.content_length(), Some(total_len), "a plain GET returns the whole object");
+
+    // A single PutObject is not a multipart upload, so it is owed no count
+    // even when a part is named.
+    client
+        .put_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(SINGLE_PUT_KEY)
+        .body(ByteStream::from_static(b"0123456789abcdef"))
+        .send()
+        .await?;
+    let single = client
+        .get_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(SINGLE_PUT_KEY)
+        .part_number(1)
+        .send()
+        .await?;
+    assert_eq!(single.parts_count(), None, "an object stored by a single PutObject is not multipart");
+
+    client
+        .delete_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(MULTIPART_KEY)
+        .send()
+        .await?;
+    client
+        .delete_object()
+        .bucket(PARTS_COUNT_BUCKET)
+        .key(SINGLE_PUT_KEY)
+        .send()
+        .await?;
+    env.delete_test_bucket(PARTS_COUNT_BUCKET).await?;
+    env.stop_server();
+
+    Ok(())
+}

crates/e2e_test/src/lib.rs

@@ -232,6 +232,10 @@ mod tier_stats_cluster_test;
 #[cfg(test)]
 mod checksum_upload_test;
 
+// GetObject with partNumber reporting x-amz-mp-parts-count
+#[cfg(test)]
+mod get_object_parts_count_test;
+
 // Group deletion tests
 #[cfg(test)]
 mod group_delete_test;
@@ -272,6 +276,8 @@ mod storage_class_capability_test;
 // S3 dummy-compat bucket API tests
 #[cfg(test)]
 mod bucket_logging_test;
+#[cfg(test)]
+mod website_hosting_test;
 
 // Multipart control API auth regression tests
 #[cfg(test)]

crates/e2e_test/src/list_objects_duplicates_test.rs

@@ -311,4 +311,233 @@ mod tests {
 
         env.stop_server();
     }
+
+    async fn collect_prefix_pages(
+        client: &Client,
+        bucket: &str,
+        prefix: &str,
+        delimiter: Option<&str>,
+        max_keys: i32,
+        start_after: Option<&str>,
+    ) -> (Vec<aws_sdk_s3::types::Object>, Vec<String>) {
+        let mut objects = Vec::new();
+        let mut prefixes = Vec::new();
+        let mut token = None;
+        let mut seen_tokens = std::collections::HashSet::new();
+        for _ in 0..64 {
+            let page = client
+                .list_objects_v2()
+                .bucket(bucket)
+                .prefix(prefix)
+                .set_delimiter(delimiter.map(ToOwned::to_owned))
+                .max_keys(max_keys)
+                .set_start_after(start_after.map(ToOwned::to_owned))
+                .set_continuation_token(token)
+                .send()
+                .await
+                .expect("prefix pagination request should succeed");
+            let count = page.contents().len() + page.common_prefixes().len();
+            assert!(count <= usize::try_from(max_keys).expect("positive page size"));
+            assert_eq!(page.key_count(), Some(i32::try_from(count).expect("small page")));
+            objects.extend_from_slice(page.contents());
+            prefixes.extend(
+                page.common_prefixes()
+                    .iter()
+                    .map(|item| item.prefix().expect("CommonPrefix must contain a prefix").to_owned()),
+            );
+            if page.is_truncated() == Some(false) {
+                assert!(page.next_continuation_token().is_none(), "final page must not have a next token");
+                return (objects, prefixes);
+            }
+            assert_eq!(page.is_truncated(), Some(true));
+            let next = page.next_continuation_token().expect("truncated page requires a token");
+            assert!(!next.is_empty());
+            assert!(seen_tokens.insert(next.to_owned()), "continuation token must advance");
+            token = Some(next.to_owned());
+        }
+        panic!("prefix pagination exceeded its finite page budget");
+    }
+
+    fn object_keys(objects: &[aws_sdk_s3::types::Object]) -> Vec<&str> {
+        objects
+            .iter()
+            .map(|object| object.key().expect("listed object must have a key"))
+            .collect()
+    }
+
+    /// Issue #8175: the prefix object must occur once, including across page boundaries.
+    #[tokio::test]
+    async fn test_list_objects_v2_prefix_marker_pagination() {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
+        env.start_rustfs_server(vec![]).await.expect("start RustFS");
+        let client = create_s3_client(&env);
+        let bucket = "test-prefix-marker-pagination";
+        create_bucket(&client, bucket).await.expect("create bucket");
+        let mut expected = vec!["content/".to_owned()];
+        expected.extend((0..25).map(|index| format!("content/0123456789abcdef0123456789abcdef/{index:03}")));
+        for key in &expected {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(if key == "content/" { b"" } else { b"x" }))
+                .send()
+                .await
+                .expect("create prefix fixture");
+        }
+        for max_keys in [1000, 1, 2] {
+            let (objects, prefixes) = collect_prefix_pages(&client, bucket, "content/", None, max_keys, None).await;
+            assert_eq!(object_keys(&objects), expected, "all keys must appear once at MaxKeys={max_keys}");
+            assert!(prefixes.is_empty());
+        }
+        let (objects, prefixes) = collect_prefix_pages(&client, bucket, "content/", None, 1, Some("content/")).await;
+        assert_eq!(object_keys(&objects), expected[1..]);
+        assert!(prefixes.is_empty());
+
+        // V1 shares the storage listing path but resumes with a key marker.
+        let mut marker = Some("content/".to_owned());
+        let mut v1_keys = Vec::new();
+        let mut finished = false;
+        for _ in 0..32 {
+            let page = client
+                .list_objects()
+                .bucket(bucket)
+                .prefix("content/")
+                .max_keys(1)
+                .set_marker(marker.clone())
+                .send()
+                .await
+                .expect("list V1 continuation page");
+            assert!(page.contents().len() <= 1);
+            let keys = object_keys(page.contents());
+            v1_keys.extend(keys.iter().map(|key| (*key).to_owned()));
+            if page.is_truncated() == Some(false) {
+                finished = true;
+                break;
+            }
+            assert_eq!(page.is_truncated(), Some(true));
+            let next = page
+                .next_marker()
+                .or_else(|| keys.last().copied())
+                .expect("V1 page must advance");
+            assert!(marker.as_deref().is_none_or(|previous| next > previous));
+            marker = Some(next.to_owned());
+        }
+        assert!(finished, "V1 pagination must terminate");
+        assert_eq!(v1_keys, expected[1..]);
+        env.stop_server();
+    }
+
+    /// An exact prefix match does not establish EOF, including for ordinary keys.
+    #[tokio::test]
+    async fn test_list_objects_v2_exact_prefix_pagination_boundaries() {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
+        env.start_rustfs_server(vec![]).await.expect("start RustFS");
+        let client = create_s3_client(&env);
+        let bucket = "test-exact-prefix-boundaries";
+        create_bucket(&client, bucket).await.expect("create bucket");
+        for key in [
+            "a",
+            "ab",
+            "solo/",
+            "marker/",
+            "marker/file",
+            "marker/subdir/",
+            "marker/subdir/file",
+        ] {
+            client
+                .put_object()
+                .bucket(bucket)
+                .key(key)
+                .body(ByteStream::from_static(b""))
+                .send()
+                .await
+                .expect("create boundary fixture");
+        }
+        let (objects, prefixes) = collect_prefix_pages(&client, bucket, "a", None, 1, None).await;
+        assert_eq!(object_keys(&objects), vec!["a", "ab"]);
+        assert!(prefixes.is_empty());
+        let first = client
+            .list_objects()
+            .bucket(bucket)
+            .prefix("a")
+            .max_keys(1)
+            .send()
+            .await
+            .expect("list V1 exact-prefix first page");
+        assert_eq!(object_keys(first.contents()), vec!["a"]);
+        assert_eq!(first.is_truncated(), Some(true));
+        let last = client
+            .list_objects()
+            .bucket(bucket)
+            .prefix("a")
+            .max_keys(1)
+            .marker(first.next_marker().unwrap_or("a"))
+            .send()
+            .await
+            .expect("list V1 exact-prefix final page");
+        assert_eq!(object_keys(last.contents()), vec!["ab"]);
+        assert_eq!(last.is_truncated(), Some(false));
+
+        let solo = client
+            .list_objects_v2()
+            .bucket(bucket)
+            .prefix("solo/")
+            .max_keys(1)
+            .send()
+            .await
+            .expect("list isolated marker");
+        assert_eq!(object_keys(solo.contents()), vec!["solo/"]);
+        assert_eq!(solo.is_truncated(), Some(false));
+        assert!(solo.next_continuation_token().is_none());
+        let (objects, prefixes) = collect_prefix_pages(&client, bucket, "marker/", Some("/"), 1, None).await;
+        assert_eq!(object_keys(&objects), vec!["marker/", "marker/file"]);
+        assert_eq!(prefixes, vec!["marker/subdir/"]);
+        env.stop_server();
+    }
+
+    /// A directory marker must retain its own metadata when a same-named object exists.
+    #[tokio::test]
+    async fn test_list_objects_v2_prefix_marker_preserves_metadata() {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
+        env.start_rustfs_server(vec![]).await.expect("start RustFS");
+        let client = create_s3_client(&env);
+        let bucket = "test-prefix-marker-metadata";
+        create_bucket(&client, bucket).await.expect("create bucket");
+        let plain = client
+            .put_object()
+            .bucket(bucket)
+            .key("content")
+            .body(ByteStream::from_static(b"plain object body"))
+            .send()
+            .await
+            .expect("put plain object");
+        let directory = client
+            .put_object()
+            .bucket(bucket)
+            .key("content/")
+            .body(ByteStream::from_static(b""))
+            .send()
+            .await
+            .expect("put directory marker");
+        assert_ne!(plain.e_tag(), directory.e_tag(), "fixture ETags must distinguish the objects");
+        for max_keys in [1, 1000] {
+            let (objects, prefixes) = collect_prefix_pages(&client, bucket, "content/", None, max_keys, None).await;
+            assert_eq!(object_keys(&objects), vec!["content/"]);
+            assert!(prefixes.is_empty());
+            assert_eq!(objects[0].size(), Some(0));
+            assert_eq!(objects[0].e_tag(), directory.e_tag());
+            let (objects, prefixes) = collect_prefix_pages(&client, bucket, "content", None, max_keys, None).await;
+            assert_eq!(object_keys(&objects), vec!["content", "content/"]);
+            assert!(prefixes.is_empty());
+            assert_eq!(objects[0].size(), Some(17));
+            assert_eq!(objects[0].e_tag(), plain.e_tag());
+            assert_eq!(objects[1].size(), Some(0));
+            assert_eq!(objects[1].e_tag(), directory.e_tag());
+        }
+        env.stop_server();
+    }
 }

crates/e2e_test/src/namespace_lock_quorum_test.rs

@@ -462,7 +462,10 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
         }
         let write_ready = survivors >= 3;
         let read_quorum = survivors >= 2;
-        let expected_status = if write_ready { 200 } else { 503 };
+        // `/health/ready` keeps a node that can still serve reads in the Service.
+        let node_ready = read_quorum;
+        let node_status = if node_ready { 200 } else { 503 };
+        let cluster_status = if write_ready { 200 } else { 503 };
         for (idx, client) in clients.iter().enumerate().take(survivors) {
             let url = &cluster.nodes[idx].url;
             let deadline = Instant::now() + Duration::from_secs(30);
@@ -472,27 +475,27 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
                 let response = http.get(format!("{url}/health/ready")).send().await?;
                 let status = response.status().as_u16();
                 let payload: serde_json::Value = response.json().await?;
-                if status == expected_status
-                    && payload["ready"] == write_ready
-                    && payload["details"]["storage"]["ready"] == write_ready
+                if status == node_status
+                    && payload["ready"] == node_ready
+                    && payload["details"]["storage"]["ready"] == node_ready
                     && payload["details"]["storage"]["readQuorum"] == read_quorum
                     && payload["details"]["storage"]["writeQuorum"] == write_ready
                     && payload["details"]["poolMetadata"]["ready"] == true
                     && payload["details"]["iam"]["ready"] == true
-                    && payload["details"]["lock"]["ready"] == write_ready
+                    && payload["details"]["lock"]["ready"] == read_quorum
                 {
                     break payload;
                 }
                 assert!(Instant::now() < deadline, "node {idx}, survivors={survivors}: HTTP {status}, {payload}");
                 tokio::time::sleep(Duration::from_millis(200)).await;
             };
-            assert_eq!(payload["details"]["storage"]["readinessScope"], "write_quorum_and_pool_metadata");
+            assert_eq!(payload["details"]["storage"]["readinessScope"], "read_quorum");
             assert_eq!(payload["details"]["storage"]["source"], "local_runtime");
             assert_eq!(
                 payload["details"]["storage"]["status"],
-                if write_ready { "connected" } else { "disconnected" }
+                if node_ready { "connected" } else { "disconnected" }
             );
-            if !write_ready {
+            if !node_ready {
                 assert!(
                     payload["degradedReasons"]
                         .as_array()
@@ -504,10 +507,10 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
 
             for path in ["/health/ready", "/minio/health/ready"] {
                 let head = http.head(format!("{url}{path}")).send().await?;
-                assert_eq!(head.status().as_u16(), expected_status, "HEAD {path}, survivors={survivors}");
+                assert_eq!(head.status().as_u16(), node_status, "HEAD {path}, survivors={survivors}");
                 assert!(head.bytes().await?.is_empty());
                 let response = http.get(format!("{url}{path}")).send().await?;
-                assert_eq!(response.status().as_u16(), expected_status);
+                assert_eq!(response.status().as_u16(), node_status);
                 let body: serde_json::Value = response.json().await?;
                 assert_eq!(body["details"]["storage"], payload["details"]["storage"]);
                 assert_eq!(body["details"]["poolMetadata"], payload["details"]["poolMetadata"]);
@@ -526,7 +529,7 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
                     let response = http.get(format!("{url}{path}")).send().await?;
                     let status = response.status().as_u16();
                     let body: serde_json::Value = response.json().await?;
-                    if status == expected_status
+                    if status == cluster_status
                         && body["details"]["storage"]["ready"] == storage_ready
                         && body["details"]["lock"]["ready"] == write_ready
                     {
@@ -559,11 +562,12 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
             let get = client.get_object().bucket(BUCKET).key(seed_key).send().await;
             let get_status = match get {
                 Ok(object) => {
+                    assert!(read_quorum, "GET must fail once read quorum is lost");
                     assert_eq!(object.body.collect().await?.into_bytes().as_ref(), seed_body);
                     200
                 }
                 Err(error) => {
-                    assert!(!write_ready, "GET must succeed on a ready cluster: {error:?}");
+                    assert!(!read_quorum, "GET must succeed while read quorum holds: {error:?}");
                     let status = error
                         .raw_response()
                         .expect("GET should have an HTTP response")
@@ -576,11 +580,12 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
             let list = client.list_objects_v2().bucket(BUCKET).send().await;
             let list_status = match list {
                 Ok(result) => {
+                    assert!(read_quorum, "listing must fail once read quorum is lost");
                     assert!(result.contents().iter().any(|object| object.key() == Some(seed_key)));
                     200
                 }
                 Err(error) => {
-                    assert!(!write_ready, "listing must succeed on a ready cluster: {error:?}");
+                    assert!(!read_quorum, "listing must succeed while read quorum holds: {error:?}");
                     let status = error
                         .raw_response()
                         .expect("LIST should have an HTTP response")

crates/e2e_test/src/website_hosting_test.rs

@@ -0,0 +1,271 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::common::{RustFSTestEnvironment, init_logging};
+use aws_sdk_s3::presigning::PresigningConfig;
+use aws_sdk_s3::primitives::ByteStream;
+use aws_sdk_s3::types::{
+    BucketVersioningStatus, Condition, ErrorDocument, IndexDocument, Redirect, RedirectAllRequestsTo, RoutingRule,
+    VersioningConfiguration, WebsiteConfiguration,
+};
+use reqwest::header::HOST;
+use std::time::Duration;
+
+type TestResult<T = ()> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
+
+#[tokio::test]
+async fn website_host_serves_pages_without_changing_s3_reads() -> TestResult {
+    init_logging();
+    let mut env = RustFSTestEnvironment::new().await?;
+    env.start_rustfs_server_with_env(vec![], &[("RUSTFS_WEBSITE_DOMAINS", "website.test")])
+        .await?;
+    let s3 = env.create_s3_client();
+    let bucket = "website-hosting";
+    s3.create_bucket().bucket(bucket).send().await?;
+    for (key, body) in [
+        ("index.html", "home"),
+        ("404.html", "error page"),
+        ("docs", "ordinary object"),
+        ("section/index.html", "section page"),
+        ("private.txt", "private"),
+    ] {
+        s3.put_object()
+            .bucket(bucket)
+            .key(key)
+            .body(ByteStream::from(body.as_bytes().to_vec()))
+            .send()
+            .await?;
+    }
+    s3.put_object()
+        .bucket(bucket)
+        .key("move")
+        .website_redirect_location("/new-page")
+        .body(ByteStream::from_static(b"ordinary redirect object"))
+        .send()
+        .await?;
+    s3.put_bucket_website()
+        .bucket(bucket)
+        .website_configuration(
+            WebsiteConfiguration::builder()
+                .index_document(IndexDocument::builder().suffix("index.html").build()?)
+                .error_document(ErrorDocument::builder().key("404.html").build()?)
+                .build(),
+        )
+        .send()
+        .await?;
+    let policy = serde_json::json!({
+        "Version": "2012-10-17",
+        "Statement": [
+            {"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
+            {"Effect":"Deny","Principal":"*","Action":"s3:GetObject","Resource":format!("arn:aws:s3:::{bucket}/private.txt")}
+        ]
+    })
+    .to_string();
+    s3.put_bucket_policy().bucket(bucket).policy(policy).send().await?;
+
+    let http = reqwest::Client::builder()
+        .no_proxy()
+        .redirect(reqwest::redirect::Policy::none())
+        .build()?;
+    let website = |path: &str| format!("{}{path}", env.url);
+    let host = format!("{bucket}.website.test");
+    let base_domain = http
+        .get(website("/website-hosting/index.html"))
+        .header(HOST, "website.test")
+        .send()
+        .await?;
+    assert_eq!(base_domain.status(), 404);
+    assert!(!base_domain.text().await?.contains("<Error>"));
+    let home = http.get(website("/")).header(HOST, &host).send().await?;
+    assert_eq!(home.status(), 200);
+    assert_eq!(home.text().await?, "home");
+    let head = http.head(website("/")).header(HOST, &host).send().await?;
+    assert_eq!(head.status(), 200);
+    assert_eq!(head.text().await?, "");
+    let missing = http.get(website("/missing")).header(HOST, &host).send().await?;
+    assert_eq!(missing.status(), 404);
+    assert_eq!(missing.text().await?, "error page");
+    let denied = http.get(website("/private.txt")).header(HOST, &host).send().await?;
+    assert_eq!(denied.status(), 403);
+    assert_eq!(denied.text().await?, "error page");
+    let collision = http.get(website("/docs")).header(HOST, &host).send().await?;
+    assert_eq!(collision.status(), 200);
+    assert_eq!(collision.text().await?, "ordinary object");
+    let directory = http.get(website("/section")).header(HOST, &host).send().await?;
+    assert_eq!(directory.status(), 302);
+    assert_eq!(directory.headers()[reqwest::header::LOCATION], "/section/");
+    let section = http.get(website("/section/")).header(HOST, &host).send().await?;
+    assert_eq!(section.status(), 200);
+    assert_eq!(section.text().await?, "section page");
+    let redirect = http.get(website("/move")).header(HOST, &host).send().await?;
+    assert_eq!(redirect.status(), 301);
+    assert_eq!(redirect.headers()[reqwest::header::LOCATION], "/new-page");
+    let redirect_with_condition = http
+        .get(website("/move"))
+        .header(HOST, &host)
+        .header(reqwest::header::IF_NONE_MATCH, "*")
+        .send()
+        .await?;
+    assert_eq!(redirect_with_condition.status(), 301);
+    let redirect_with_range = http
+        .get(website("/move"))
+        .header(HOST, &host)
+        .header(reqwest::header::RANGE, "bytes=999999-")
+        .send()
+        .await?;
+    assert_eq!(redirect_with_range.status(), 301);
+    let plain_range_error = http
+        .get(website("/index.html"))
+        .header(HOST, &host)
+        .header(reqwest::header::RANGE, "bytes=999999-")
+        .send()
+        .await?;
+    assert_eq!(plain_range_error.status(), 416);
+    assert!(!plain_range_error.text().await?.contains("<Error>"));
+
+    let ordinary = s3.get_object().bucket(bucket).key("move").send().await?;
+    assert_eq!(ordinary.body.collect().await?.into_bytes(), "ordinary redirect object");
+    let ranged = s3.get_object().bucket(bucket).key("move").range("bytes=0-7").send().await?;
+    assert_eq!(ranged.body.collect().await?.into_bytes(), "ordinary");
+    let head_object = s3.head_object().bucket(bucket).key("move").send().await?;
+    let etag = head_object.e_tag().expect("object ETag");
+    let unchanged = s3.get_object().bucket(bucket).key("move").if_none_match(etag).send().await;
+    assert!(unchanged.is_err(), "ordinary S3 conditional GET should return NotModified");
+    let presigned = s3
+        .get_object()
+        .bucket(bucket)
+        .key("move")
+        .presigned(PresigningConfig::expires_in(Duration::from_secs(300))?)
+        .await?;
+    let presigned_read = http.get(presigned.uri().to_string()).send().await?;
+    assert_eq!(presigned_read.status(), 200);
+    assert_eq!(presigned_read.text().await?, "ordinary redirect object");
+    let listed = s3.list_objects_v2().bucket(bucket).send().await?;
+    assert!(listed.contents().iter().any(|object| object.key() == Some("index.html")));
+    let s3_missing = http.get(format!("{}/{bucket}/missing", env.url)).send().await?;
+    assert!(s3_missing.status().is_client_error());
+    assert!(s3_missing.text().await?.contains("<Error>"));
+    s3.put_bucket_versioning()
+        .bucket(bucket)
+        .versioning_configuration(
+            VersioningConfiguration::builder()
+                .status(BucketVersioningStatus::Enabled)
+                .build(),
+        )
+        .send()
+        .await?;
+    let first = s3
+        .put_object()
+        .bucket(bucket)
+        .key("versioned.txt")
+        .body(ByteStream::from_static(b"first"))
+        .send()
+        .await?;
+    s3.put_object()
+        .bucket(bucket)
+        .key("versioned.txt")
+        .body(ByteStream::from_static(b"second"))
+        .send()
+        .await?;
+    let versioned = s3
+        .get_object()
+        .bucket(bucket)
+        .key("versioned.txt")
+        .version_id(first.version_id().expect("first object version"))
+        .send()
+        .await?;
+    assert_eq!(versioned.body.collect().await?.into_bytes(), "first");
+
+    s3.put_bucket_website()
+        .bucket(bucket)
+        .website_configuration(
+            WebsiteConfiguration::builder()
+                .index_document(IndexDocument::builder().suffix("index.html").build()?)
+                .routing_rules(
+                    RoutingRule::builder()
+                        .condition(Condition::builder().http_error_code_returned_equals("404").build())
+                        .redirect(Redirect::builder().replace_key_with("missing-page").build())
+                        .build(),
+                )
+                .routing_rules(
+                    RoutingRule::builder()
+                        .condition(Condition::builder().key_prefix_equals("docs/").build())
+                        .redirect(Redirect::builder().replace_key_with("docs-page").build())
+                        .build(),
+                )
+                .routing_rules(
+                    RoutingRule::builder()
+                        .condition(Condition::builder().http_error_code_returned_equals("500").build())
+                        .redirect(Redirect::builder().replace_key_with("wrong-status").build())
+                        .build(),
+                )
+                .routing_rules(
+                    RoutingRule::builder()
+                        .condition(Condition::builder().http_error_code_returned_equals("416").build())
+                        .redirect(Redirect::builder().replace_key_with("range-error").build())
+                        .build(),
+                )
+                .build(),
+        )
+        .send()
+        .await?;
+    let rule = http.get(website("/docs/missing")).header(HOST, &host).send().await?;
+    assert_eq!(rule.status(), 301);
+    assert_eq!(
+        rule.headers()[reqwest::header::LOCATION],
+        "http://website-hosting.website.test/missing-page"
+    );
+    let ranged_error = http
+        .get(website("/index.html"))
+        .header(HOST, &host)
+        .header(reqwest::header::RANGE, "bytes=999999-")
+        .send()
+        .await?;
+    assert_eq!(ranged_error.status(), 301);
+    assert_eq!(
+        ranged_error.headers()[reqwest::header::LOCATION],
+        "http://website-hosting.website.test/range-error"
+    );
+
+    s3.put_bucket_website()
+        .bucket(bucket)
+        .website_configuration(
+            WebsiteConfiguration::builder()
+                .redirect_all_requests_to(RedirectAllRequestsTo::builder().host_name("destination.test").build()?)
+                .build(),
+        )
+        .send()
+        .await?;
+    let all = http.get(website("/a%20b?x=1")).header(HOST, &host).send().await?;
+    assert_eq!(all.status(), 301);
+    assert_eq!(all.headers()[reqwest::header::LOCATION], "http://destination.test/a%20b?x=1");
+
+    let dotted = "my.website.bucket";
+    s3.create_bucket().bucket(dotted).send().await?;
+    s3.put_bucket_website()
+        .bucket(dotted)
+        .website_configuration(
+            WebsiteConfiguration::builder()
+                .redirect_all_requests_to(RedirectAllRequestsTo::builder().host_name("destination.test").build()?)
+                .build(),
+        )
+        .send()
+        .await?;
+    let dotted_host = format!("{dotted}.website.test");
+    let dotted_response = http.get(website("/key")).header(HOST, dotted_host).send().await?;
+    assert_eq!(dotted_response.status(), 301);
+    assert_eq!(dotted_response.headers()[reqwest::header::LOCATION], "http://destination.test/key");
+    env.stop_server();
+    Ok(())
+}

crates/ecstore/Cargo.toml

@@ -223,11 +223,13 @@ aws-smithy-http-client = { workspace = true, default-features = false, features
 # Observability and Metrics
 metrics = { workspace = true }
 
-# Runtime-probed io_uring read backend (backlog#1104). Linux-only, from
-# crates.io. The guard scripts/check_no_tokio_io_uring.sh allows an explicit
+# Runtime-probed io_uring read backend (backlog#1104). Linux-only.
+# The guard scripts/check_no_tokio_io_uring.sh allows an explicit
 # io-uring integration; only the tokio "io-uring" runtime feature is banned.
 [target.'cfg(target_os = "linux")'.dependencies]
-rustfs-uring = "0.2.2"
+# SharedReadBudget and the bounded read APIs are available in the 0.3.0
+# registry release; keep the application on the published contract.
+rustfs-uring = "0.3.0"
 
 [target.'cfg(windows)'.dependencies]
 winapi-util.workspace = true

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -409,6 +409,11 @@ pub struct BucketTargetSys {
     /// lock (never held across an await) so the replication worker can record
     /// a verdict from inside its synchronous PUT-response audit.
     version_identity_map: Arc<std::sync::RwLock<HashMap<String, VersionIdentityCapability>>>,
+    /// Target ARNs whose remote rejected a PUT / CreateMultipartUpload
+    /// `versionId` query with `InvalidArgument`; later requests to the same
+    /// ARN omit the query instead of failing again. Reset alongside
+    /// `version_identity_map` — same lifetime, same reasoning.
+    put_version_id_query_rejected: Arc<std::sync::RwLock<HashSet<String>>>,
     pub targets_map: Arc<RwLock<HashMap<String, Vec<BucketTarget>>>>,
     /// Buckets whose persisted `bucket-targets.json` exists but cannot be
     /// decoded (rustfs/backlog#2282). Written under the bucket's update mutex
@@ -458,6 +463,7 @@ impl BucketTargetSys {
             arn_remotes_map: Arc::new(RwLock::new(HashMap::new())),
             ssec_passthrough_map: Arc::new(RwLock::new(HashMap::new())),
             version_identity_map: Arc::new(std::sync::RwLock::new(HashMap::new())),
+            put_version_id_query_rejected: Arc::new(std::sync::RwLock::new(HashSet::new())),
             targets_map: Arc::new(RwLock::new(HashMap::new())),
             unreadable_targets: Arc::new(RwLock::new(HashSet::new())),
             h_mutex: Arc::new(RwLock::new(HashMap::new())),
@@ -782,10 +788,36 @@ impl BucketTargetSys {
                 health_map.remove(&target.arn);
                 ssec_map.remove(&target.arn);
                 self.forget_version_identity_capability(&target.arn);
+                self.forget_put_version_id_query_rejected(&target.arn);
             }
         }
     }
 
+    /// Whether this target ARN is already known to reject a `versionId`
+    /// query on PUT / CreateMultipartUpload.
+    pub fn put_version_id_query_rejected(&self, arn: &str) -> bool {
+        self.put_version_id_query_rejected
+            .read()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .contains(arn)
+    }
+
+    /// Record that this target ARN rejected a `versionId` query on PUT /
+    /// CreateMultipartUpload, so later requests to it omit the query.
+    pub fn record_put_version_id_query_rejected(&self, arn: &str) {
+        self.put_version_id_query_rejected
+            .write()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .insert(arn.to_string());
+    }
+
+    fn forget_put_version_id_query_rejected(&self, arn: &str) {
+        self.put_version_id_query_rejected
+            .write()
+            .unwrap_or_else(|poisoned| poisoned.into_inner())
+            .remove(arn);
+    }
+
     /// Cached version-identity verdict for a target ARN; `Unknown` until a
     /// replication write or a replication-check VersionFidelity probe judged
     /// it since the target was built.
@@ -1252,6 +1284,7 @@ impl BucketTargetSys {
                     .is_none_or(|edited| !same_replication_service(edited, &target))
                 {
                     self.forget_version_identity_capability(&target.arn);
+                    self.forget_put_version_id_query_rejected(&target.arn);
                 }
                 self.update_bandwidth_limit(bucket, &target.arn, 0);
             }
@@ -1449,8 +1482,13 @@ pub fn resolve_delete_api_version_id(version_id: Option<String>, opts: &RemoveOb
 /// "null" (the delete path already maps it via `target_delete_version_id`),
 /// and an empty id means the source object carries no version: send no query
 /// so an unversioned target stays valid.
-fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
-    if source_version_id.is_empty() {
+///
+/// A standard S3 target (AWS S3 included) instead rejects this query on PUT /
+/// CreateMultipartUpload outright with `InvalidArgument: This operation does
+/// not accept a version-id`. Once a target ARN has been observed to reject
+/// it (`arn_rejected_query`), it is never sent to that ARN again.
+fn resolve_put_api_version_id(source_version_id: &str, arn_rejected_query: bool) -> Option<&str> {
+    if arn_rejected_query || source_version_id.is_empty() {
         None
     } else if Uuid::parse_str(source_version_id).is_ok_and(|uuid| uuid.is_nil()) {
         Some(rustfs_filemeta::NULL_VERSION_ID)
@@ -1459,6 +1497,13 @@ fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
     }
 }
 
+/// True when a PUT / CreateMultipartUpload service error is the standard S3
+/// rejection of a `versionId` query on those operations (AWS S3 and other
+/// strict S3-compatible targets; MinIO/RustFS peers accept the query).
+fn is_put_version_id_query_rejected(code: Option<&str>, message: Option<&str>) -> bool {
+    code == Some("InvalidArgument") && message.is_some_and(|m| m.to_ascii_lowercase().contains("version-id"))
+}
+
 /// Resolve the S3 `versionId` for a proxied read against a remote target.
 /// RustFS represents the null version internally as the nil UUID while the S3
 /// API addresses it as the literal "null" (same mapping as
@@ -2251,7 +2296,9 @@ impl TargetClient {
         if !version_id.is_empty() {
             insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
         }
-        let api_version_id = resolve_put_api_version_id(&version_id).map(ToOwned::to_owned);
+        let api_version_id =
+            resolve_put_api_version_id(&version_id, BucketTargetSys::get().put_version_id_query_rejected(&self.arn))
+                .map(ToOwned::to_owned);
 
         // A PUT carrying Object Lock parameters must also carry Content-MD5 or
         // an x-amz-checksum-* header on AWS-compatible targets (rustfs#7082).
@@ -2333,6 +2380,9 @@ impl TargetClient {
                 SdkError::ServiceError(service_err) => {
                     let err = service_err.into_err();
                     let meta = err.meta();
+                    if is_put_version_id_query_rejected(meta.code(), meta.message()) {
+                        BucketTargetSys::get().record_put_version_id_query_rejected(&self.arn);
+                    }
                     let error = match (meta.code(), meta.message()) {
                         (Some(code), Some(message)) => format!("put_object failed: {code}: {message}"),
                         (Some(code), None) => format!("put_object failed: {code}"),
@@ -2369,7 +2419,9 @@ impl TargetClient {
         let version_id = opts.internal.source_version_id.clone();
         // The remote version of a multipart replication is decided at initiate
         // time; CompleteMultipartUpload does not read a versionId.
-        let api_version_id = resolve_put_api_version_id(&version_id).map(ToOwned::to_owned);
+        let api_version_id =
+            resolve_put_api_version_id(&version_id, BucketTargetSys::get().put_version_id_query_rejected(&self.arn))
+                .map(ToOwned::to_owned);
 
         match self
             .client
@@ -2395,7 +2447,15 @@ impl TargetClient {
             .await
         {
             Ok(res) => Ok(res.upload_id.unwrap_or_default()),
-            Err(e) => Err(e.into()),
+            Err(e) => {
+                if let SdkError::ServiceError(service_err) = &e {
+                    let meta = service_err.err().meta();
+                    if is_put_version_id_query_rejected(meta.code(), meta.message()) {
+                        BucketTargetSys::get().record_put_version_id_query_rejected(&self.arn);
+                    }
+                }
+                Err(e.into())
+            }
         }
     }
 
@@ -3597,6 +3657,87 @@ mod tests {
         assert_eq!(sys.version_identity_capability(arn), VersionIdentityCapability::Unknown);
     }
 
+    #[test]
+    fn put_version_id_query_rejection_is_per_arn_and_forgotten_with_the_target() {
+        let sys = BucketTargetSys::default();
+        let arn = "arn:rustfs:replication:us-east-1:bucket:versionid";
+        assert!(!sys.put_version_id_query_rejected(arn));
+        sys.record_put_version_id_query_rejected(arn);
+        assert!(sys.put_version_id_query_rejected(arn));
+        assert!(!sys.put_version_id_query_rejected("other"));
+        // A rebuilt target may point at a different service.
+        sys.forget_put_version_id_query_rejected(arn);
+        assert!(!sys.put_version_id_query_rejected(arn));
+    }
+
+    #[tokio::test]
+    async fn update_all_targets_forgets_put_version_id_rejection_when_endpoint_changes() {
+        // Editing a target's ARN to point at a different endpoint must not
+        // carry over a stale "this remote rejects versionId" verdict from the
+        // service it used to point at (rustfs review follow-up on the
+        // versionId-on-PUT fix): otherwise a peer that legitimately needs the
+        // query silently loses version-id fidelity until the process restarts.
+        let sys = BucketTargetSys::default();
+        let arn = "arn:rustfs:replication:us-east-1:bucket:versionid";
+        let target = |endpoint: &str| BucketTarget {
+            arn: arn.to_string(),
+            endpoint: endpoint.to_string(),
+            target_bucket: "target-bucket".to_string(),
+            region: "us-east-1".to_string(),
+            credentials: Some(Credentials {
+                access_key: "access".to_string(),
+                secret_key: "secret".to_string(),
+                session_token: None,
+                expiration: None,
+            }),
+            ..Default::default()
+        };
+
+        sys.update_all_targets(
+            "bucket",
+            Some(&BucketTargets {
+                targets: vec![target("aws-s3.example:9000")],
+            }),
+        )
+        .await;
+        sys.record_put_version_id_query_rejected(arn);
+        assert!(sys.put_version_id_query_rejected(arn));
+
+        // Same ARN, different endpoint: the update path (not `delete`) is the
+        // only thing that runs here, so the reset must come from
+        // `update_all_targets_locked` itself.
+        sys.update_all_targets(
+            "bucket",
+            Some(&BucketTargets {
+                targets: vec![target("minio-peer.example:9000")],
+            }),
+        )
+        .await;
+
+        assert!(
+            !sys.put_version_id_query_rejected(arn),
+            "editing the target to a different endpoint must forget the stale rejection verdict"
+        );
+    }
+
+    #[test]
+    fn resolve_put_api_version_id_omits_query_once_rejected() {
+        let version_id = Uuid::new_v4().to_string();
+        assert_eq!(resolve_put_api_version_id(&version_id, false), Some(version_id.as_str()));
+        assert_eq!(resolve_put_api_version_id(&version_id, true), None);
+    }
+
+    #[test]
+    fn is_put_version_id_query_rejected_matches_aws_invalid_argument() {
+        assert!(is_put_version_id_query_rejected(
+            Some("InvalidArgument"),
+            Some("This operation does not accept a version-id.")
+        ));
+        assert!(!is_put_version_id_query_rejected(Some("AccessDenied"), Some("access denied")));
+        assert!(!is_put_version_id_query_rejected(Some("InvalidArgument"), Some("bucket name is invalid")));
+        assert!(!is_put_version_id_query_rejected(None, None));
+    }
+
     #[test]
     fn endpoint_health_key_preserves_explicit_port() {
         let url = Url::parse("https://remote.example:9443").expect("url should parse");

crates/ecstore/src/bucket/metadata.rs

@@ -18,9 +18,10 @@ use super::versioning::VersioningApi;
 use super::{quota::BucketQuota, target::BucketTargets};
 use crate::bucket::replication::invalid_replication_config_status_field;
 use crate::bucket::utils::deserialize;
-use crate::config::com::{read_config, read_config_preserve_empty, save_config};
+use crate::config::com::{read_config, read_config_preserve_empty, save_config, save_config_with_opts};
 use crate::disk::BUCKET_META_PREFIX;
 use crate::error::{Error, Result};
+use crate::object_api::WriteCompletion;
 use crate::runtime::sources as runtime_sources;
 use crate::store::ECStore;
 use byteorder::{BigEndian, ByteOrder, LittleEndian};
@@ -437,6 +438,7 @@ pub struct BucketMetadata {
     pub lock_enabled: bool, // While marked as unused, it may need to be retained
     pub bucket_incarnation_id: Uuid,
     pub(crate) bucket_incarnation_sidecar: bool,
+    pub(crate) bucket_creation_committed: bool,
     pub policy_config_json: Vec<u8>,
     pub notification_config_xml: Vec<u8>,
     pub lifecycle_config_xml: Vec<u8>,
@@ -511,6 +513,7 @@ impl Default for BucketMetadata {
             lock_enabled: Default::default(),
             bucket_incarnation_id: Uuid::nil(),
             bucket_incarnation_sidecar: false,
+            bucket_creation_committed: false,
             policy_config_json: Default::default(),
             notification_config_xml: Default::default(),
             lifecycle_config_xml: Default::default(),
@@ -594,6 +597,22 @@ impl BucketMetadata {
         metadata
     }
 
+    /// Whether this generation still needs its creation-commit proof.
+    ///
+    /// The proof is only ever written while the caller holds the bucket
+    /// metadata transaction fence; see
+    /// `BucketMetadataSys::migrate_bucket_creation_commit`.
+    ///
+    /// Metadata without an authoritative incarnation sidecar is owned by the
+    /// legacy migration path, which mints the sidecar first; it must not be
+    /// rewritten by this migration.
+    pub(crate) fn needs_bucket_creation_commit(&self) -> bool {
+        self.lock_enabled
+            && !self.bucket_creation_committed
+            && self.bucket_incarnation_sidecar
+            && !self.bucket_incarnation_id.is_nil()
+    }
+
     pub fn save_file_path(&self) -> String {
         format!("{}/{}/{}", BUCKET_META_PREFIX, self.name.as_str(), BUCKET_METADATA_FILE)
     }
@@ -733,6 +752,7 @@ impl BucketMetadata {
                 "TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
                 "DurabilityConfigUpdatedAt" => self.durability_config_updated_at = read_msgp_time_value(rd)?,
                 "OnDemandMigrationConfigUpdatedAt" => self.on_demand_migration_config_updated_at = read_msgp_time_value(rd)?,
+                "BucketCreationCommitted" => self.bucket_creation_committed = read_msgp_bool(rd)?,
                 other => {
                     tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
                     skip_msgp_value(rd)?;
@@ -745,8 +765,8 @@ impl BucketMetadata {
 
     /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
-        // Map size: MinIO fields (25) + RustFS extensions (21)
-        let map_len: u32 = 46;
+        // Map size: MinIO fields (25) + RustFS extensions (22)
+        let map_len: u32 = 47;
         rmp::encode::write_map_len(wr, map_len)?;
 
         // MinIO field order (same as Go struct)
@@ -827,6 +847,8 @@ impl BucketMetadata {
         write_msgp_time(wr, self.durability_config_updated_at)?;
         rmp::encode::write_str(wr, "OnDemandMigrationConfigUpdatedAt")?;
         write_msgp_time(wr, self.on_demand_migration_config_updated_at)?;
+        rmp::encode::write_str(wr, "BucketCreationCommitted")?;
+        rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
 
         Ok(())
     }
@@ -982,6 +1004,11 @@ impl BucketMetadata {
                 self.object_lock_config = None;
                 if !data.is_empty() {
                     self.lock_enabled = true;
+                    // Enabling Object Lock is only reachable for a bucket that
+                    // already exists: the caller validated physical presence and
+                    // holds the incarnation fence, so this generation is committed
+                    // and must never be mistaken for a pre-physical creation intent.
+                    self.bucket_creation_committed = true;
                 }
                 self.object_lock_config_xml = data;
                 self.object_lock_config_updated_at = updated;
@@ -1086,6 +1113,21 @@ impl BucketMetadata {
     /// server's bucket metadata lands in that server's `.rustfs.sys`, not the
     /// ambient (first) one. [`BucketMetadata::save`] keeps the ambient default.
     pub async fn save_with_store(&mut self, store: std::sync::Arc<crate::store::ECStore>) -> Result<()> {
+        self.save_with_store_completion(store, WriteCompletion::Quorum).await
+    }
+
+    /// Persist a committed creation record and drain the rename fan-out before
+    /// returning. A quorum ACK is insufficient here: later degraded reads may
+    /// have to reconstruct this record from any read quorum of surviving shards.
+    pub(crate) async fn save_with_store_committed(&mut self, store: std::sync::Arc<crate::store::ECStore>) -> Result<()> {
+        self.save_with_store_completion(store, WriteCompletion::TailDrained).await
+    }
+
+    async fn save_with_store_completion(
+        &mut self,
+        store: std::sync::Arc<crate::store::ECStore>,
+        write_completion: WriteCompletion,
+    ) -> Result<()> {
         self.parse_all_configs()?;
         let mut buf: Vec<u8> = vec![0; 4];
 
@@ -1099,7 +1141,17 @@ impl BucketMetadata {
 
         buf.extend_from_slice(&data);
 
-        save_config(store, self.save_file_path().as_str(), buf).await?;
+        save_config_with_opts(
+            store,
+            self.save_file_path().as_str(),
+            buf,
+            &crate::object_api::ObjectOptions {
+                max_parity: true,
+                write_completion,
+                ..Default::default()
+            },
+        )
+        .await?;
 
         Ok(())
     }
@@ -1451,8 +1503,13 @@ pub(crate) async fn load_bucket_metadata_parse_with_presence(
         }
     };
 
-    let incarnation = load_bucket_incarnation(api, bucket).await?;
+    let incarnation = load_bucket_incarnation(api.clone(), bucket).await?;
     if persisted {
+        if bm.name.is_empty() {
+            bm.name = bucket.to_string();
+        } else if bm.name != bucket {
+            return Err(Error::FileCorrupt);
+        }
         if let Some(incarnation) = incarnation {
             if !bm.bucket_incarnation_id.is_nil() && bm.bucket_incarnation_id != incarnation {
                 return Err(Error::other("bucket incarnation sidecar does not match bucket metadata"));
@@ -1622,6 +1679,7 @@ mod test {
         BucketMetadata::check_header(&body).expect("recovered body is a valid .metadata.bin");
         let mut bm = BucketMetadata::unmarshal(&body[4..]).expect("unmarshal recovered blob");
         assert_eq!(bm.name, "interop");
+        assert!(!bm.bucket_creation_committed, "legacy metadata must default to uncommitted");
         bm.parse_all_configs().expect("parse recovered configs");
         assert!(bm.lifecycle_config.is_some());
     }
@@ -1630,13 +1688,15 @@ mod test {
     async fn marshal_msg() {
         // write_time(OffsetDateTime::UNIX_EPOCH).unwrap();
 
-        let bm = BucketMetadata::new("dada");
+        let mut bm = BucketMetadata::new("dada");
+        bm.bucket_creation_committed = true;
 
         let buf = bm.marshal_msg().unwrap();
 
         let new = BucketMetadata::unmarshal(&buf).unwrap();
 
         assert_eq!(bm.name, new.name);
+        assert!(new.bucket_creation_committed);
         assert!(!bm.bucket_incarnation_id.is_nil());
         assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
     }

crates/ecstore/src/bucket/metadata_sys.rs

@@ -527,11 +527,19 @@ pub(crate) async fn set_new_bucket_metadata_in(
     lock.persist_new_and_set(bm).await
 }
 
-pub(crate) async fn cache_bucket_metadata_in(ctx: &crate::runtime::instance::InstanceContext, bm: BucketMetadata) -> Result<()> {
+pub(crate) async fn set_new_bucket_metadata_intent_in(
+    ctx: &crate::runtime::instance::InstanceContext,
+    bm: BucketMetadata,
+) -> Result<()> {
     let sys = bucket_metadata_sys_of(ctx)?;
     let lock = sys.read().await;
-    lock.set(bm.name.clone(), Arc::new(bm)).await;
-    Ok(())
+    lock.persist_new_bucket_metadata_intent(bm).await
+}
+
+pub(crate) async fn commit_bucket_metadata_in(ctx: &crate::runtime::instance::InstanceContext, bm: BucketMetadata) -> Result<()> {
+    let sys = bucket_metadata_sys_of(ctx)?;
+    let lock = sys.read().await;
+    lock.commit_bucket_metadata(bm).await
 }
 
 pub(crate) async fn remove_bucket_metadata_in(ctx: &crate::runtime::instance::InstanceContext, bucket: &str) -> Result<bool> {
@@ -2074,9 +2082,32 @@ impl BucketMetadataSys {
     }
 
     async fn persist_new_and_set(&self, mut bm: BucketMetadata) -> Result<()> {
+        bm.bucket_creation_committed = true;
+        bm.save_with_store_committed(self.object_store()).await?;
+        save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
+        bm.bucket_incarnation_sidecar = true;
+        self.set(bm.name.clone(), Arc::new(bm)).await;
+        Ok(())
+    }
+
+    async fn persist_new_bucket_metadata_intent(&self, mut bm: BucketMetadata) -> Result<()> {
+        bm.bucket_creation_committed = false;
         bm.save_with_store(self.object_store()).await?;
         save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
         bm.bucket_incarnation_sidecar = true;
+        // A pending creation intent must not enter the authoritative cache. The
+        // caller publishes it only after physical creation and commit-marker write.
+        Ok(())
+    }
+
+    async fn commit_bucket_metadata(&self, mut bm: BucketMetadata) -> Result<()> {
+        bm.parse_all_configs()?;
+        bm.bucket_incarnation_sidecar = true;
+        // The caller holds the full bucket creation fence (lifecycle, metadata
+        // transaction, and namespace write) and physical creation already
+        // returned, so this generation is committed.
+        bm.bucket_creation_committed = true;
+        bm.save_with_store_committed(self.object_store()).await?;
         self.set(bm.name.clone(), Arc::new(bm)).await;
         Ok(())
     }
@@ -2192,18 +2223,25 @@ impl BucketMetadataSys {
             )
             .await?;
 
-            let bm = Arc::new(bm);
-
             if persisted {
+                // Object Lock metadata is persisted before physical bucket creation as a
+                // creation intent. The commit marker distinguishes a committed bucket
+                // from an intent left behind by an interrupted create.
+                let require_write_quorum = bm.needs_bucket_creation_commit();
+
                 await_bucket_namespace_operation(
                     Some(&guard),
                     bucket,
                     "lazy bucket metadata existence check",
                     Box::pin(async {
-                        self.object_store()
-                            .get_bucket_info_from_sets(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
-                            .await
-                            .map(|_| ())
+                        let store = self.object_store();
+                        let options = crate::storage_api_contracts::bucket::BucketOptions::default();
+                        let info = if require_write_quorum {
+                            store.get_bucket_info_from_sets(bucket, &options).await
+                        } else {
+                            store.get_bucket_info_from_sets_at_read_quorum(bucket, &options).await
+                        };
+                        info.map(|_| ())
                     }),
                 )
                 .await?;
@@ -2212,6 +2250,7 @@ impl BucketMetadataSys {
                         "bucket namespace lock was lost before lazy bucket metadata publish: {bucket}"
                     )));
                 }
+                let bm = Arc::new(bm);
                 let _publish_guard = self
                     .lock_metadata_publish(bucket, &guard, "lazy bucket metadata publish")
                     .await?;
@@ -2226,6 +2265,7 @@ impl BucketMetadataSys {
                 sync_bucket_target_sys(bucket, &bm).await;
                 sync_bucket_durability(bucket, &bm);
                 sync_on_demand_migration(bucket, &bm);
+                return Ok((bm, true));
             } else {
                 let exists = self
                     .bucket_exists(bucket, &guard, "lazy bucket metadata existence check")
@@ -2245,7 +2285,7 @@ impl BucketMetadataSys {
                 }
             }
 
-            Ok((bm, true))
+            Ok((Arc::new(bm), true))
         }
     }
 
@@ -2430,7 +2470,13 @@ impl BucketMetadataSys {
     }
 
     async fn get_metadata_authority(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
-        if let Some(bm) = self.metadata_map.read().await.get(bucket).cloned() {
+        // Bind before matching: holding the map guard across the migration would
+        // deadlock on the publish write.
+        let cached = self.metadata_map.read().await.get(bucket).cloned();
+        if let Some(bm) = cached {
+            if bm.needs_bucket_creation_commit() {
+                return self.migrate_bucket_creation_commit(bucket).await;
+            }
             return Ok(BucketMetadataAuthority::Authoritative(bm));
         }
         if self.fabricated_metadata.read().await.contains(bucket) {
@@ -2442,8 +2488,13 @@ impl BucketMetadataSys {
 
         self.get_config(bucket).await?;
 
-        if let Some(bm) = self.metadata_map.read().await.get(bucket).cloned() {
-            Ok(BucketMetadataAuthority::Authoritative(bm))
+        let loaded = self.metadata_map.read().await.get(bucket).cloned();
+        if let Some(bm) = loaded {
+            if bm.needs_bucket_creation_commit() {
+                self.migrate_bucket_creation_commit(bucket).await
+            } else {
+                Ok(BucketMetadataAuthority::Authoritative(bm))
+            }
         } else if self.fabricated_metadata.read().await.contains(bucket) {
             Ok(BucketMetadataAuthority::Fabricated)
         } else if self.missing_buckets.get(bucket).await.is_some() {
@@ -2453,6 +2504,70 @@ impl BucketMetadataSys {
         }
     }
 
+    /// Persist the creation-commit proof for an Object Lock bucket created
+    /// before this field existed.
+    ///
+    /// Lock order matches the established config-write order: metadata
+    /// transaction (write), then bucket namespace (read). The authoritative
+    /// metadata is re-read under both fences and the physical namespace is
+    /// revalidated at write quorum, so a stale snapshot can neither revert an
+    /// acknowledged configuration update nor outlive a delete/recreate.
+    async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
+        let transaction_lock = self
+            .object_store()
+            .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
+            .await?;
+        let transaction_guard = transaction_lock
+            .get_write_lock(crate::set_disk::get_lock_acquire_timeout())
+            .await?;
+        let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
+        let namespace_guard = namespace_lock
+            .get_read_lock(crate::set_disk::get_lock_acquire_timeout())
+            .await?;
+
+        // Write quorum must still prove that the physical namespace committed.
+        await_bucket_namespace_operation(
+            Some(&namespace_guard),
+            bucket,
+            "bucket creation commitment existence check",
+            self.object_store()
+                .get_bucket_info_from_sets(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default()),
+        )
+        .await?;
+
+        let (mut metadata, persisted) = await_bucket_namespace_operation(
+            Some(&namespace_guard),
+            bucket,
+            "bucket creation commitment metadata reload",
+            load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
+        )
+        .await?;
+        if !persisted {
+            return Ok(BucketMetadataAuthority::Fabricated);
+        }
+        if metadata.needs_bucket_creation_commit() {
+            metadata.bucket_creation_committed = true;
+            metadata.save_with_store_committed(self.object_store()).await?;
+        }
+
+        if transaction_guard.is_lock_lost() || namespace_guard.is_lock_lost() {
+            // Stable message: formatted per-bucket detail would split quorum
+            // error buckets (backlog#1845).
+            return Err(Error::other("bucket creation commitment fence was lost before publish"));
+        }
+        let metadata = Arc::new(metadata);
+        let _publish_guard = self
+            .lock_metadata_publish(bucket, &namespace_guard, "bucket creation commitment publish")
+            .await?;
+        self.metadata_map
+            .write()
+            .await
+            .insert(bucket.to_string(), Arc::clone(&metadata));
+        self.fabricated_metadata.write().await.remove(bucket);
+        self.missing_buckets.invalidate(bucket).await;
+        Ok(BucketMetadataAuthority::Authoritative(metadata))
+    }
+
     async fn migrate_legacy_metadata(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
         let transaction_lock = self
             .object_store()
@@ -2564,7 +2679,6 @@ impl BucketMetadataSys {
                 .await?;
             }
         }
-
         if namespace_guard.is_lock_lost() {
             return Err(Error::other(format!(
                 "bucket namespace lock was lost before legacy metadata publish: {bucket}"
@@ -2626,6 +2740,9 @@ impl BucketMetadataSys {
             load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
         )
         .await?;
+        if persisted && metadata.lock_enabled && !metadata.bucket_creation_committed {
+            return Err(Error::ErasureWriteQuorum);
+        }
         if persisted {
             Ok(BucketMetadataAuthority::Authoritative(Arc::new(metadata)))
         } else {

crates/ecstore/src/cluster/rpc/network_probe.rs

@@ -14,13 +14,18 @@
 
 //! Bounded, authenticated inter-node network probes.
 
-use std::time::{Duration, Instant};
+use std::future::Future;
+use std::pin::Pin;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::time::Duration;
 
 use bytes::Bytes;
 use rustfs_protos::ChannelClass;
 use rustfs_protos::models::{PingBody, PingBodyBuilder};
 use rustfs_protos::proto_gen::node_service::{PingRequest, PingResponse};
 use thiserror::Error;
+use tokio::time::Instant;
 use tokio_util::sync::CancellationToken;
 use tonic::{Code, Request};
 
@@ -33,6 +38,40 @@ pub const MAX_NETWORK_PROBE_DURATION: Duration = Duration::from_secs(30);
 const PING_PROTOCOL_VERSION: u64 = 1;
 const LATENCY_PAYLOAD: &[u8] = b"network-probe-latency-v1";
 const EXPECTED_RESPONSE_PAYLOAD: &[u8] = b"hello, caller";
+const DIAGNOSTIC_BYTES_PER_SECOND: u64 = 1_048_576;
+const DIAGNOSTIC_CHUNK_BYTES: u64 = 16_384;
+
+#[derive(Debug)]
+struct DiagnosticPacing {
+    started: Instant,
+    deadline: Instant,
+    charged_bytes: AtomicU64,
+}
+
+impl DiagnosticPacing {
+    fn reserve(&self, bytes: u64) -> Result<Instant, NetworkPeerProbeError> {
+        if bytes == 0 {
+            return Err(NetworkPeerProbeError::LimitExceeded);
+        }
+        // Reservation is never refunded: a failed or dropped RPC may have sent some payload.
+        let previous = self
+            .charged_bytes
+            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |charged| {
+                charged.checked_add(bytes).filter(|total| *total <= MAX_NETWORK_PROBE_BYTES)
+            })
+            .map_err(|_| NetworkPeerProbeError::LimitExceeded)?;
+        let charged = previous.checked_add(bytes).ok_or(NetworkPeerProbeError::LimitExceeded)?;
+        let millis = u128::from(charged)
+            .checked_mul(1_000)
+            .ok_or(NetworkPeerProbeError::LimitExceeded)?
+            .div_ceil(u128::from(DIAGNOSTIC_BYTES_PER_SECOND));
+        self.started
+            .checked_add(Duration::from_millis(
+                u64::try_from(millis).map_err(|_| NetworkPeerProbeError::LimitExceeded)?,
+            ))
+            .ok_or(NetworkPeerProbeError::LimitExceeded)
+    }
+}
 
 #[derive(Clone, Debug, PartialEq, Eq)]
 pub struct NetworkPeerTarget {
@@ -66,6 +105,7 @@ pub enum NetworkPeerProbeError {
 #[derive(Clone, Debug)]
 pub struct NetworkPeerProbeClient {
     targets: Vec<NetworkPeerTarget>,
+    diagnostic_pacing: Option<Arc<DiagnosticPacing>>,
 }
 
 impl NetworkPeerProbeClient {
@@ -80,7 +120,25 @@ impl NetworkPeerProbeClient {
                 address,
             })
             .collect();
-        Self { targets }
+        Self {
+            targets,
+            diagnostic_pacing: None,
+        }
+    }
+
+    /// Opt in one diagnostic run; clones share its charged payload and absolute deadline.
+    /// Default/admin probes retain their existing unpaced behavior.
+    pub fn with_diagnostic_pacing(mut self, started: Instant, duration: Duration) -> Result<Self, NetworkPeerProbeError> {
+        if self.diagnostic_pacing.is_some() || duration.is_zero() || duration > MAX_NETWORK_PROBE_DURATION {
+            return Err(NetworkPeerProbeError::LimitExceeded);
+        }
+        let deadline = started.checked_add(duration).ok_or(NetworkPeerProbeError::LimitExceeded)?;
+        self.diagnostic_pacing = Some(Arc::new(DiagnosticPacing {
+            started,
+            deadline,
+            charged_bytes: AtomicU64::new(0),
+        }));
+        Ok(self)
     }
 
     pub fn targets(&self) -> Vec<NetworkPeerTarget> {
@@ -110,7 +168,19 @@ impl NetworkPeerProbeClient {
             .find(|target| target.alias == peer_alias)
             .map(|target| target.address.as_str())
             .ok_or(NetworkPeerProbeError::UnknownPeer)?;
-        let probe = probe_peer(address, traffic_bytes);
+        let probe = probe_peer(address, traffic_bytes, self.diagnostic_pacing.as_deref(), cancel);
+        if let Some(pacing) = &self.diagnostic_pacing {
+            let deadline = Instant::now()
+                .checked_add(max_duration)
+                .ok_or(NetworkPeerProbeError::LimitExceeded)?
+                .min(pacing.deadline);
+            return tokio::select! {
+                biased;
+                () = cancel.cancelled() => Err(NetworkPeerProbeError::Cancelled),
+                () = tokio::time::sleep_until(deadline) => Err(NetworkPeerProbeError::TimedOut),
+                result = probe => result,
+            };
+        }
         tokio::select! {
             () = cancel.cancelled() => Err(NetworkPeerProbeError::Cancelled),
             result = tokio::time::timeout(max_duration, probe) => {
@@ -120,7 +190,12 @@ impl NetworkPeerProbeClient {
     }
 }
 
-async fn probe_peer(address: &str, traffic_bytes: u64) -> Result<NetworkPeerProbeMeasurement, NetworkPeerProbeError> {
+async fn probe_peer(
+    address: &str,
+    traffic_bytes: u64,
+    pacing: Option<&DiagnosticPacing>,
+    cancel: &CancellationToken,
+) -> Result<NetworkPeerProbeMeasurement, NetworkPeerProbeError> {
     let started = Instant::now();
     let mut control = client(address, ChannelClass::Control).await?;
     let latency_started = Instant::now();
@@ -132,15 +207,18 @@ async fn probe_peer(address: &str, traffic_bytes: u64) -> Result<NetworkPeerProb
     validate_ping_response(&latency_response)?;
     let latency = latency_started.elapsed();
 
-    let payload_len = usize::try_from(traffic_bytes).map_err(|_| NetworkPeerProbeError::LimitExceeded)?;
-    let payload = vec![0x5a; payload_len];
     let mut bulk = client(address, ChannelClass::Bulk).await?;
-    let response = bulk
-        .ping(Request::new(ping_request(&payload)))
-        .await
-        .map_err(map_status)?
-        .into_inner();
-    validate_ping_response(&response)?;
+    send_payload(traffic_bytes, pacing, cancel, &mut bulk, |bulk, payload| {
+        Box::pin(async move {
+            let response = bulk
+                .ping(Request::new(ping_request(&payload)))
+                .await
+                .map_err(map_status)?
+                .into_inner();
+            validate_ping_response(&response)
+        })
+    })
+    .await?;
 
     Ok(NetworkPeerProbeMeasurement {
         transferred_bytes: traffic_bytes,
@@ -149,6 +227,59 @@ async fn probe_peer(address: &str, traffic_bytes: u64) -> Result<NetworkPeerProb
     })
 }
 
+type PayloadSendFuture<'a> = Pin<Box<dyn Future<Output = Result<(), NetworkPeerProbeError>> + Send + 'a>>;
+
+async fn send_payload<T, F>(
+    traffic_bytes: u64,
+    pacing: Option<&DiagnosticPacing>,
+    cancel: &CancellationToken,
+    sender: &mut T,
+    mut send: F,
+) -> Result<(), NetworkPeerProbeError>
+where
+    T: Send,
+    F: for<'a> FnMut(&'a mut T, Vec<u8>) -> PayloadSendFuture<'a> + Send,
+{
+    let mut remaining = traffic_bytes;
+    while remaining > 0 {
+        let bytes = if pacing.is_some() {
+            remaining.min(DIAGNOSTIC_CHUNK_BYTES)
+        } else {
+            remaining
+        };
+        if let Some(pacing) = pacing {
+            if cancel.is_cancelled() {
+                return Err(NetworkPeerProbeError::Cancelled);
+            }
+            if Instant::now() >= pacing.deadline {
+                return Err(NetworkPeerProbeError::TimedOut);
+            }
+            let due = pacing.reserve(bytes)?;
+            tokio::select! {
+                biased;
+                () = cancel.cancelled() => return Err(NetworkPeerProbeError::Cancelled),
+                () = tokio::time::sleep_until(pacing.deadline) => return Err(NetworkPeerProbeError::TimedOut),
+                () = tokio::time::sleep_until(due) => {},
+            }
+        }
+        let payload_len = usize::try_from(bytes).map_err(|_| NetworkPeerProbeError::LimitExceeded)?;
+        let payload = vec![0x5a; payload_len];
+        if let Some(pacing) = pacing {
+            // Check before polling the send future, including a cancellation/deadline tie.
+            tokio::select! {
+                biased;
+                () = cancel.cancelled() => return Err(NetworkPeerProbeError::Cancelled),
+                () = tokio::time::sleep_until(pacing.deadline) => return Err(NetworkPeerProbeError::TimedOut),
+                result = send(sender, payload) => result?,
+            }
+        } else {
+            send(sender, payload).await?;
+        }
+        remaining -= bytes;
+    }
+    Ok(())
+}
+
 async fn client(
     address: &str,
     class: ChannelClass,
@@ -208,6 +339,205 @@ mod tests {
     use crate::layout::endpoint::Endpoint;
     use crate::layout::endpoints::{Endpoints, PoolEndpoints};
 
+    fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
+        NetworkPeerProbeClient::from_endpoint_pools(&topology())
+            .with_diagnostic_pacing(Instant::now(), duration)
+            .expect("bounded diagnostic pacing window")
+            .diagnostic_pacing
+            .expect("opt-in pacing is present")
+    }
+
+    #[test]
+    fn pacing_uses_exact_millisecond_ceiling_at_n_and_n_plus_one() {
+        for duration in [Duration::ZERO, MAX_NETWORK_PROBE_DURATION + Duration::from_nanos(1)] {
+            assert!(matches!(
+                NetworkPeerProbeClient::from_endpoint_pools(&topology()).with_diagnostic_pacing(Instant::now(), duration),
+                Err(NetworkPeerProbeError::LimitExceeded)
+            ));
+        }
+        for (bytes, millis) in [(1, 1), (1_047_527, 999), (1_047_528, 1_000), (1_048_576, 1_000)] {
+            let pacing = pacing(Duration::from_secs(2));
+            assert_eq!(
+                pacing.reserve(bytes).expect("bounded reservation") - pacing.started,
+                Duration::from_millis(millis)
+            );
+        }
+        for bytes in [0, MAX_NETWORK_PROBE_BYTES + 1, u64::MAX] {
+            let pacing = pacing(Duration::from_secs(2));
+            assert_eq!(pacing.reserve(bytes), Err(NetworkPeerProbeError::LimitExceeded));
+            assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 0);
+        }
+    }
+
+    #[test]
+    fn cloned_diagnostic_clients_share_one_atomic_budget_without_reset() {
+        let client = NetworkPeerProbeClient::from_endpoint_pools(&topology())
+            .with_diagnostic_pacing(Instant::now(), Duration::from_secs(2))
+            .expect("pacing");
+        let cloned = client.clone();
+        let first = client.diagnostic_pacing.as_ref().expect("first budget");
+        let second = cloned.diagnostic_pacing.as_ref().expect("cloned budget");
+        assert!(Arc::ptr_eq(first, second));
+        let mut due = std::thread::scope(|scope| {
+            let first = scope.spawn(|| first.reserve(MAX_NETWORK_PROBE_BYTES / 2));
+            let second = scope.spawn(|| second.reserve(MAX_NETWORK_PROBE_BYTES / 2));
+            [
+                first.join().expect("first reservation thread").expect("first half"),
+                second.join().expect("second reservation thread").expect("second half"),
+            ]
+        });
+        due.sort();
+        assert_eq!(
+            due,
+            [
+                first.started + Duration::from_millis(500),
+                first.started + Duration::from_secs(1)
+            ]
+        );
+        assert_eq!(first.reserve(1), Err(NetworkPeerProbeError::LimitExceeded));
+        assert_eq!(first.charged_bytes.load(Ordering::Acquire), MAX_NETWORK_PROBE_BYTES);
+        assert!(matches!(
+            cloned.with_diagnostic_pacing(Instant::now(), Duration::from_secs(2)),
+            Err(NetworkPeerProbeError::LimitExceeded)
+        ));
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn failed_peer_payload_stays_charged_before_the_next_peer_send() {
+        let pacing = pacing(Duration::from_secs(2));
+        let cancel = CancellationToken::new();
+        let mut observed_partial_bytes = 0;
+        let failed = send_payload(32_768, Some(&pacing), &cancel, &mut observed_partial_bytes, |partial, payload| {
+            Box::pin(async move {
+                assert_eq!(payload.len(), 16_384);
+                // Model failure after a partial write: the sender cannot safely refund the remainder.
+                *partial += 8_192;
+                Err(NetworkPeerProbeError::Unreachable)
+            })
+        })
+        .await;
+        assert_eq!(failed, Err(NetworkPeerProbeError::Unreachable));
+        assert_eq!(observed_partial_bytes, 8_192);
+        assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 16_384);
+        let started = pacing.started;
+        let mut second_bytes = 0;
+        send_payload(16_384, Some(&pacing), &cancel, &mut second_bytes, move |bytes, payload| {
+            Box::pin(async move {
+                assert!(started.elapsed() >= Duration::from_millis(32));
+                *bytes += payload.len();
+                Ok(())
+            })
+        })
+        .await
+        .expect("second peer uses remaining shared budget");
+        assert_eq!(second_bytes, 16_384);
+        assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 32_768);
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn cancellation_and_deadline_take_priority_over_an_earned_send() {
+        for cancelled in [false, true] {
+            let pacing = pacing(Duration::from_millis(16));
+            let cancel = CancellationToken::new();
+            let mut sends = 0;
+            {
+                let mut sending =
+                    std::pin::pin!(send_payload(8_192, Some(&pacing), &cancel, &mut sends, |sends, _| Box::pin(async move {
+                        *sends += 1;
+                        Ok(())
+                    })));
+                assert!(futures::poll!(&mut sending).is_pending());
+                tokio::time::advance(Duration::from_millis(16)).await;
+                if cancelled {
+                    cancel.cancel();
+                }
+                let expected = if cancelled {
+                    NetworkPeerProbeError::Cancelled
+                } else {
+                    NetworkPeerProbeError::TimedOut
+                };
+                assert_eq!(sending.await, Err(expected));
+            }
+            assert_eq!(sends, 0);
+            assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 8_192);
+        }
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn cancellation_during_a_payload_rpc_retains_its_attempted_charge() {
+        let pacing = pacing(Duration::from_secs(2));
+        let cancel = CancellationToken::new();
+        let mut sends = 0;
+        {
+            let mut sending = std::pin::pin!(send_payload(16_384, Some(&pacing), &cancel, &mut sends, |sends, _| Box::pin(
+                async move {
+                    *sends += 1;
+                    std::future::pending().await
+                }
+            )));
+            assert!(futures::poll!(&mut sending).is_pending());
+            tokio::time::advance(Duration::from_millis(16)).await;
+            assert!(futures::poll!(&mut sending).is_pending());
+            cancel.cancel();
+            assert_eq!(sending.await, Err(NetworkPeerProbeError::Cancelled));
+        }
+        assert_eq!(sends, 1);
+        assert_eq!(pacing.charged_bytes.load(Ordering::Acquire), 16_384);
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn default_payload_send_remains_one_unpaced_rpc() {
+        let started = Instant::now();
+        let mut sends = 0;
+        send_payload(32_768, None, &CancellationToken::new(), &mut sends, move |sends, payload| {
+            Box::pin(async move {
+                assert_eq!(started.elapsed(), Duration::ZERO);
+                assert_eq!(payload.len(), 32_768);
+                *sends += 1;
+                Ok(())
+            })
+        })
+        .await
+        .expect("legacy admin send");
+        assert_eq!(sends, 1);
+        assert!(
+            NetworkPeerProbeClient::from_endpoint_pools(&topology())
+                .diagnostic_pacing
+                .is_none()
+        );
+    }
+
+    #[tokio::test(start_paused = true)]
+    async fn diagnostic_payload_is_charged_before_each_actual_send_poll() {
+        let started = tokio::time::Instant::now();
+        let pacing = DiagnosticPacing {
+            started,
+            deadline: started + Duration::from_secs(2),
+            charged_bytes: AtomicU64::new(0),
+        };
+        let mut attempted = 0_u64;
+        send_payload(
+            32_768,
+            Some(&pacing),
+            &CancellationToken::new(),
+            &mut attempted,
+            move |attempted, payload| {
+                Box::pin(async move {
+                    *attempted += u64::try_from(payload.len()).expect("bounded payload length");
+                    let available = u128::from(MAX_NETWORK_PROBE_BYTES) * started.elapsed().as_millis() / 1_000;
+                    assert!(
+                        u128::from(*attempted) <= available,
+                        "send boundary emitted {attempted} bytes with only {available} bytes earned"
+                    );
+                    Ok(())
+                })
+            },
+        )
+        .await
+        .expect("bounded diagnostic payload");
+        assert_eq!(attempted, 32_768);
+    }
+
     fn endpoint(value: &str, is_local: bool) -> Endpoint {
         let mut endpoint = Endpoint::try_from(value).expect("test endpoint should parse");
         endpoint.is_local = is_local;
@@ -230,6 +560,14 @@ mod tests {
         }])
     }
 
+    #[test]
+    fn probe_future_remains_send() {
+        fn require_send<T: Send>(_: T) {}
+        let client = NetworkPeerProbeClient::from_endpoint_pools(&topology());
+        let cancel = CancellationToken::new();
+        require_send(client.probe("peer-1", 1, Duration::from_secs(1), &cancel));
+    }
+
     #[test]
     fn targets_are_unique_sorted_remote_nodes_with_stable_aliases() {
         let client = NetworkPeerProbeClient::from_endpoint_pools(&topology());

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -993,7 +993,19 @@ impl RemoteDisk {
         // internode transport failure classified as retryable can be safely re-dialed. The
         // classifier is direction-agnostic — it inspects the InternodeHttpError kind — so it is
         // reused here from the write path.
-        err.is_retryable_internode_write_failure()
+        if err.is_retryable_internode_write_failure() {
+            return true;
+        }
+        // Header wait uses the body stall budget. That error is `BodyStalled`, not an
+        // `InternodeHttpError`, and the open has not consumed a shard byte yet.
+        matches!(
+            err,
+            DiskError::Io(error)
+                if error
+                    .get_ref()
+                    .and_then(|source| source.downcast_ref::<rustfs_rio::BodyStalled>())
+                    .is_some()
+        )
     }
 
     pub(crate) async fn new(ep: &Endpoint, opt: &DiskOption, data_transport: Arc<dyn InternodeDataTransport>) -> Result<Self> {
@@ -1102,7 +1114,14 @@ impl RemoteDisk {
         let mut attempt = 1;
         let mut last_retry_classification = None;
         loop {
-            match self.data_transport.open_read(request.clone()).await {
+            // The second attempt bypasses the pool. The first failure is often a
+            // stale kept-alive connection to a peer that just restarted.
+            let opened = if attempt == 1 {
+                self.data_transport.open_read(request.clone()).await
+            } else {
+                self.data_transport.open_read_fresh(request.clone()).await
+            };
+            match opened {
                 Ok(reader) => {
                     if attempt > 1
                         && let Some(classification) = last_retry_classification
@@ -1138,7 +1157,12 @@ impl RemoteDisk {
         let mut attempt = 1;
         let mut last_retry_classification = None;
         loop {
-            match self.data_transport.open_read_chunks(request.clone()).await {
+            let opened = if attempt == 1 {
+                self.data_transport.open_read_chunks(request.clone()).await
+            } else {
+                self.data_transport.open_read_chunks_fresh(request.clone()).await
+            };
+            match opened {
                 Ok(reader) => {
                     if attempt > 1
                         && let Some(classification) = last_retry_classification
@@ -1220,13 +1244,14 @@ impl RemoteDisk {
         {
             return;
         }
+        // Own the flag before spawning: shutdown can drop the task without polling it.
+        let lease = RecoveryMonitorLease {
+            active: Arc::clone(&active),
+        };
         let span = Self::recovery_monitor_span(&addr, &endpoint, handle_id);
         super::spawn_background_monitor(span, async move {
             #[cfg(test)]
             test_state.start_count.fetch_add(1, Ordering::AcqRel);
-            let lease = RecoveryMonitorLease {
-                active: Arc::clone(&active),
-            };
             Self::monitor_remote_disk_recovery(addr.clone(), endpoint.clone(), Arc::clone(&health), cancel_token.clone()).await;
             #[cfg(test)]
             if let Some(hook) = test_state.teardown_hook.lock().await.take() {
@@ -1532,7 +1557,8 @@ impl RemoteDisk {
         };
 
         if evict_cached_connection {
-            evict_failed_connection(addr).await;
+            // Cache contention must not hold the recovery lease indefinitely.
+            let _ = timeout(get_drive_active_check_timeout(), evict_failed_connection(addr)).await;
         }
 
         result
@@ -1691,7 +1717,7 @@ impl RemoteDisk {
         if timeout_duration == Duration::ZERO {
             let operation_result = operation().await;
             if operation_result.is_ok() {
-                self.health.log_success();
+                self.health.record_operation_success(&self.endpoint, "operation_success");
             }
             self.handle_network_like_error(op, timeout_duration, &operation_result, failure_health_action)
                 .await;
@@ -1705,7 +1731,7 @@ impl RemoteDisk {
             Ok(operation_result) => {
                 // Log success; the waiting guard balances every exit path.
                 if operation_result.is_ok() {
-                    self.health.log_success();
+                    self.health.record_operation_success(&self.endpoint, "operation_success");
                 }
                 self.handle_network_like_error(op, timeout_duration, &operation_result, failure_health_action)
                     .await;
@@ -7729,6 +7755,25 @@ mod tests {
         assert_eq!(transport.calls().len(), 2, "read_file_stream should retry exactly once");
     }
 
+    #[tokio::test]
+    async fn test_remote_disk_read_file_stream_retries_header_stall_on_fresh_connection() {
+        let stall = rustfs_rio::BodyStalled {
+            timeout: Duration::from_millis(50),
+        };
+        let transport = RetryingOpenReadInternodeDataTransport::with_steps(vec![
+            OpenWriteTestStep::Error(DiskError::Io(std::io::Error::new(std::io::ErrorKind::TimedOut, stall))),
+            OpenWriteTestStep::Success,
+        ]);
+        let remote_disk = new_remote_disk_with_transport(Arc::new(transport.clone())).await;
+
+        let _reader = remote_disk
+            .read_file_stream("bucket", "object/part.1", 0, 4096)
+            .await
+            .expect("a header stall must be retried once before the shard is failed");
+
+        assert_eq!(transport.calls().len(), 2, "header stall should re-dial exactly once");
+    }
+
     #[tokio::test]
     async fn test_remote_disk_read_file_stream_does_not_retry_non_retryable_open_read_error() {
         let transport = RetryingOpenReadInternodeDataTransport::with_steps(vec![OpenWriteTestStep::Error(DiskError::from(
@@ -8161,6 +8206,97 @@ mod tests {
         assert!(result.is_ok());
     }
 
+    #[tokio::test]
+    async fn successful_operation_recovers_suspect_remote_disk_without_monitor() {
+        let endpoint = Endpoint {
+            url: url::Url::parse("http://remote-recovery:9000/data").expect("valid endpoint"),
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+        let disk = RemoteDisk::new(
+            &endpoint,
+            &DiskOption {
+                cleanup: false,
+                health_check: false,
+            },
+            Arc::new(TcpHttpInternodeDataTransport),
+        )
+        .await
+        .expect("create remote disk");
+
+        for duration in [Duration::ZERO, Duration::from_secs(1)] {
+            disk.health.mark_failure(&endpoint, "read_operation_deadline");
+            assert_eq!(disk.runtime_state(), RuntimeDriveHealthState::Suspect);
+            assert!(!disk.recovery_monitor_is_active());
+            let result = disk
+                .execute_with_timeout(|| async { Err::<(), _>(DiskError::FileNotFound) }, duration)
+                .await;
+            assert!(matches!(result, Err(DiskError::FileNotFound)));
+            assert_eq!(
+                disk.runtime_state(),
+                RuntimeDriveHealthState::Suspect,
+                "a failed operation must not restore readiness"
+            );
+
+            disk.execute_with_timeout(|| async { Ok(()) }, duration)
+                .await
+                .expect("successful disk RPC");
+            assert_eq!(
+                disk.runtime_state(),
+                RuntimeDriveHealthState::Online,
+                "successful traffic must restore readiness without a recovery monitor"
+            );
+            assert!(disk.offline_duration_secs().is_none());
+            assert_eq!(disk.health.waiting.load(Ordering::Acquire), 0);
+        }
+
+        disk.health.mark_offline(&endpoint, "test_offline");
+        let result = disk.execute_with_timeout(|| async { Ok(()) }, Duration::from_secs(1)).await;
+        assert!(
+            matches!(result, Err(DiskError::FaultyDisk)),
+            "offline handles still require recovery probes before data I/O"
+        );
+        assert_eq!(disk.runtime_state(), RuntimeDriveHealthState::Offline);
+    }
+
+    #[test]
+    fn recovery_monitor_releases_lease_when_dropped_before_first_poll() {
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .expect("create test runtime");
+        let endpoint = Endpoint {
+            url: url::Url::parse("http://remote-unpolled:9000/data").expect("valid endpoint"),
+            is_local: false,
+            pool_idx: 0,
+            set_idx: 0,
+            disk_idx: 0,
+        };
+        let active = Arc::new(AtomicBool::new(false));
+        let start_count = Arc::new(AtomicU32::new(0));
+        {
+            let _guard = runtime.enter();
+            RemoteDisk::schedule_recovery_monitor(
+                "http://remote-unpolled:9000".to_string(),
+                endpoint,
+                Uuid::new_v4(),
+                Arc::new(DiskHealthTracker::new()),
+                CancellationToken::new(),
+                Arc::clone(&active),
+                RecoveryMonitorTestState {
+                    start_count: Arc::clone(&start_count),
+                    teardown_hook: Arc::new(tokio::sync::Mutex::new(None)),
+                },
+            );
+            assert!(active.load(Ordering::Acquire));
+        }
+        drop(runtime);
+        assert_eq!(start_count.load(Ordering::Acquire), 0, "task must never be polled");
+        assert!(!active.load(Ordering::Acquire), "dropping an unpolled monitor must release its lease");
+    }
+
     #[tokio::test]
     async fn test_execute_with_timeout_marks_remote_disk_faulty() {
         let url = url::Url::parse("http://remote-timeout:9000").expect("operation should succeed");

crates/ecstore/src/disk/disk_store.rs

@@ -1236,6 +1236,17 @@ impl DiskHealthTracker {
                 record_drive_recovery_class(classify_drive_recovery(duration));
             }
             self.offline_since_unix_secs.store(0, Ordering::Release);
+            info!(
+                event = EVENT_DISK_RECOVERY_PROBE_STATE,
+                component = LOG_COMPONENT_ECSTORE,
+                subsystem = LOG_SUBSYSTEM_DISK,
+                endpoint = %endpoint,
+                state = "recovered",
+                previous_state = current.as_str(),
+                runtime_state = next.as_str(),
+                reason,
+                "Disk recovered"
+            );
         } else if let Some(duration) = self.offline_duration() {
             record_drive_offline_duration(endpoint, duration);
         }

crates/ecstore/src/disk/fd_cache_tests.rs

@@ -0,0 +1,179 @@
+// Copyright 2024 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+use super::{FdCache, FdCacheEntry, FdKey};
+use std::fs::File;
+use std::os::unix::fs::FileExt;
+use std::sync::Arc;
+
+fn key(volume: &str, path: &str, direct: bool) -> FdKey {
+    FdKey {
+        volume: volume.to_owned(),
+        path: path.to_owned(),
+        direct,
+    }
+}
+
+fn fixture() -> (tempfile::TempDir, Arc<File>) {
+    let directory = tempfile::tempdir().expect("create cache fixture directory");
+    let path = directory.path().join("part");
+    std::fs::write(&path, b"old").expect("write original cache fixture");
+    let file = Arc::new(File::open(path).expect("open original cache fixture"));
+    (directory, file)
+}
+
+// Both key variants intentionally use normal files. These tests exercise the
+// real Moka cache's identity/invalidation contract, not native O_DIRECT I/O.
+#[tokio::test]
+async fn uring_fd_cache_exact_invalidation_removes_both_modes_and_preserves_other_paths() {
+    let (_directory, file) = fixture();
+    let cache = FdCache::new();
+    let paths = [("v", "a/b"), ("v", "a/b/part.1"), ("v", "a/bc"), ("other", "a/b")];
+    for (volume, path) in paths {
+        for direct in [false, true] {
+            cache.insert(key(volume, path, direct), Arc::clone(&file)).await;
+        }
+    }
+    assert_eq!(cache.entry_count().await, 8, "all mode/scope fixtures must exist before invalidation");
+    cache.invalidate_exact("v", "a/b").await;
+    for (volume, path) in paths {
+        for direct in [false, true] {
+            let hit = cache.get(&key(volume, path, direct)).await.is_some();
+            assert_eq!(
+                hit,
+                (volume, path) != ("v", "a/b"),
+                "wrong invalidation scope: {volume}/{path}, direct={direct}"
+            );
+        }
+    }
+}
+
+#[tokio::test]
+async fn uring_fd_cache_exact_invalidation_bumps_one_generation_and_rejects_old_epoch_inserts() {
+    let (_directory, file) = fixture();
+    let cache = FdCache::new();
+    let entry = Arc::new(FdCacheEntry { file, len: 3 });
+    let old_generation = cache.generation();
+    cache.invalidate_exact("v", "part").await;
+    let current_generation = cache.generation();
+    assert_eq!(current_generation, old_generation + 1, "one logical invalidation must advance once");
+    for direct in [false, true] {
+        let key = key("v", "part", direct);
+        cache.insert_if_fresh(key.clone(), Arc::clone(&entry), old_generation).await;
+        assert!(cache.get(&key).await.is_none(), "old-epoch insert resurrected direct={direct}");
+        cache
+            .insert_if_fresh(key.clone(), Arc::clone(&entry), current_generation)
+            .await;
+        assert!(cache.get(&key).await.is_some(), "current epoch must allow refill, direct={direct}");
+    }
+}
+
+#[tokio::test]
+async fn uring_fd_cache_exact_invalidation_accepts_missing_or_single_mode_entries() {
+    let (_directory, file) = fixture();
+    for present in [None, Some(false), Some(true)] {
+        let cache = FdCache::new();
+        if let Some(direct) = present {
+            cache.insert(key("v", "part", direct), Arc::clone(&file)).await;
+            assert!(cache.get(&key("v", "part", direct)).await.is_some());
+        }
+        cache.invalidate_exact("v", "part").await;
+        for direct in [false, true] {
+            assert!(cache.get(&key("v", "part", direct)).await.is_none(), "mode survived: {present:?}");
+        }
+        assert_eq!(cache.generation(), 1);
+        cache.invalidate_exact("v", "part").await;
+        assert_eq!(cache.generation(), 2, "repeated invalidation still fences concurrent miss opens");
+    }
+}
+
+#[tokio::test]
+async fn uring_fd_cache_prefix_invalidation_covers_both_modes_without_crossing_components() {
+    let (_directory, file) = fixture();
+    let cache = FdCache::new();
+    let paths = [("v", "a/b"), ("v", "a/b/part.1"), ("v", "a/bc"), ("other", "a/b")];
+    for (volume, path) in paths {
+        for direct in [false, true] {
+            cache.insert(key(volume, path, direct), Arc::clone(&file)).await;
+            assert!(cache.get(&key(volume, path, direct)).await.is_some());
+        }
+    }
+    cache.invalidate_under("v", "a/b/");
+    for (volume, path) in paths {
+        for direct in [false, true] {
+            let hit = cache.get(&key(volume, path, direct)).await.is_some();
+            let removed = volume == "v" && matches!(path, "a/b" | "a/b/part.1");
+            assert_eq!(hit, !removed, "wrong prefix scope: {volume}/{path}, direct={direct}");
+        }
+    }
+}
+
+#[tokio::test]
+async fn uring_fd_cache_volume_and_clear_invalidation_cover_both_modes() {
+    let (_directory, file) = fixture();
+    let cache = FdCache::new();
+    for volume in ["v", "other"] {
+        for direct in [false, true] {
+            cache.insert(key(volume, "part", direct), Arc::clone(&file)).await;
+            assert!(cache.get(&key(volume, "part", direct)).await.is_some());
+        }
+    }
+    cache.invalidate_volume("v");
+    for direct in [false, true] {
+        assert!(cache.get(&key("v", "part", direct)).await.is_none());
+        assert!(cache.get(&key("other", "part", direct)).await.is_some());
+        cache.insert(key("v", "part", direct), Arc::clone(&file)).await;
+        assert!(
+            cache.get(&key("v", "part", direct)).await.is_some(),
+            "new entries must survive the earlier volume predicate"
+        );
+    }
+    cache.clear();
+    for volume in ["v", "other"] {
+        for direct in [false, true] {
+            assert!(cache.get(&key(volume, "part", direct)).await.is_none());
+        }
+    }
+}
+
+#[tokio::test]
+async fn uring_fd_cache_invalidation_keeps_borrowed_old_inode_alive_and_allows_replacement_refill() {
+    let (directory, file) = fixture();
+    let cache = FdCache::new();
+    let mut borrowed = Vec::new();
+    for direct in [false, true] {
+        let key = key("v", "part", direct);
+        cache.insert(key.clone(), Arc::clone(&file)).await;
+        borrowed.push(cache.get(&key).await.expect("borrow original inode before replacement"));
+    }
+    drop(file);
+    let replacement = directory.path().join("replacement");
+    std::fs::write(&replacement, b"replacement").expect("write replacement inode");
+    std::fs::rename(&replacement, directory.path().join("part")).expect("replace cached path with new inode");
+    cache.invalidate_exact("v", "part").await;
+    for (entry, direct) in borrowed.iter().zip([false, true]) {
+        assert!(cache.get(&key("v", "part", direct)).await.is_none());
+        let mut bytes = [0; 3];
+        entry
+            .file
+            .read_exact_at(&mut bytes, 0)
+            .expect("borrowed descriptor remains readable after invalidation");
+        assert_eq!(&bytes, b"old");
+        assert_eq!(entry.len, 3, "borrowed size snapshot belongs to the old inode");
+    }
+    let replacement = Arc::new(FdCacheEntry {
+        file: Arc::new(File::open(directory.path().join("part")).expect("open replacement inode")),
+        len: 11,
+    });
+    for direct in [false, true] {
+        let key = key("v", "part", direct);
+        cache
+            .insert_if_fresh(key.clone(), Arc::clone(&replacement), cache.generation())
+            .await;
+        let fresh = cache.get(&key).await.expect("replacement should populate the current epoch");
+        let mut bytes = [0; 11];
+        fresh.file.read_exact_at(&mut bytes, 0).expect("read replacement inode");
+        assert_eq!(&bytes, b"replacement");
+        assert_eq!(fresh.len, 11);
+    }
+}

crates/ecstore/src/disk/mod.rs

@@ -1365,6 +1365,10 @@ pub struct DiskInfoOptions {
     pub disk_id: String,
     pub metrics: bool,
     pub noop: bool,
+    /// Bypass the ordinary one-second admin cache and return a positive
+    /// acknowledgement only after a fresh capacity syscall completes.
+    #[serde(default)]
+    pub fresh_capacity: bool,
 }
 
 #[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
@@ -1389,6 +1393,10 @@ pub struct DiskInfo {
     pub rotational: bool,
     pub metrics: DiskMetrics,
     pub error: String,
+    /// Older peers omit this field. Callers must treat an absent/false value as
+    /// cached or otherwise unproven capacity evidence.
+    #[serde(default)]
+    pub fresh_capacity: bool,
 }
 
 #[derive(Clone, Debug, Default)]
@@ -1912,11 +1920,40 @@ mod tests {
             disk_id: "test-disk-id".to_string(),
             metrics: true,
             noop: false,
+            fresh_capacity: true,
         };
 
         assert_eq!(opts.disk_id, "test-disk-id");
         assert!(opts.metrics);
         assert!(!opts.noop);
+        assert!(opts.fresh_capacity);
+    }
+
+    #[test]
+    fn disk_info_fresh_capacity_ack_is_additive_for_old_peers() {
+        let old_options: DiskInfoOptions =
+            serde_json::from_str(r#"{"disk_id":"disk","metrics":false,"noop":false}"#).expect("old options");
+        assert!(!old_options.fresh_capacity);
+
+        let mut old_response_json = serde_json::to_value(DiskInfo {
+            total: 10,
+            free: 4,
+            used: 6,
+            ..Default::default()
+        })
+        .expect("response fixture");
+        old_response_json
+            .as_object_mut()
+            .expect("response object")
+            .remove("fresh_capacity");
+        let old_response: DiskInfo = serde_json::from_value(old_response_json).expect("old response");
+        assert!(!old_response.fresh_capacity);
+
+        let requested = DiskInfoOptions {
+            fresh_capacity: true,
+            ..Default::default()
+        };
+        assert_eq!(serde_json::to_value(requested).expect("new options")["fresh_capacity"], true);
     }
 
     /// Test ReadMultipleReq structure

crates/ecstore/src/disk/uring_driver_budget.rs

@@ -0,0 +1,340 @@
+// Copyright 2024 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+use std::ffi::OsStr;
+use std::sync::Arc;
+use tokio::sync::{OwnedSemaphorePermit, Semaphore, TryAcquireError};
+
+/// Optional process budget for the dedicated driver threads (one per shard).
+/// It does not account for io-wq workers, blocking tasks, or buffer allocation.
+pub(super) enum DriverThreadBudget {
+    Unlimited,
+    Limited(Arc<Semaphore>),
+}
+
+#[derive(Debug)]
+pub(super) struct InvalidBudget;
+
+impl std::fmt::Display for InvalidBudget {
+    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        formatter.write_str("expected an unsigned decimal integer within Semaphore::MAX_PERMITS")
+    }
+}
+
+impl std::error::Error for InvalidBudget {}
+
+impl DriverThreadBudget {
+    /// Parse a supplied value so tests do not mutate process-wide environment.
+    /// Missing or zero is the legacy unlimited policy; malformed input fails.
+    pub(super) fn from_env_value(value: Option<&OsStr>) -> Result<Self, InvalidBudget> {
+        let Some(value) = value else { return Ok(Self::Unlimited) };
+        let text = value.to_str().ok_or(InvalidBudget)?;
+        if text.is_empty() || !text.bytes().all(|byte| byte.is_ascii_digit()) {
+            return Err(InvalidBudget);
+        }
+        let limit = text.parse::<usize>().map_err(|_| InvalidBudget)?;
+        if limit > Semaphore::MAX_PERMITS {
+            return Err(InvalidBudget);
+        }
+        Ok(if limit == 0 {
+            Self::Unlimited
+        } else {
+            Self::Limited(Arc::new(Semaphore::new(limit)))
+        })
+    }
+
+    /// Reserve the full configured shard count without waiting for a live
+    /// driver to retire. Insufficient capacity selects the caller's std backend.
+    pub(super) fn try_reserve(&self, shards: usize) -> Result<Option<OwnedSemaphorePermit>, TryAcquireError> {
+        // Production shards are clamped to 1..=16 before this call. Retain a
+        // checked conversion at the semaphore's u32 API boundary.
+        let count = u32::try_from(shards).map_err(|_| TryAcquireError::NoPermits)?;
+        if count == 0 {
+            return Err(TryAcquireError::NoPermits);
+        }
+        match self {
+            Self::Unlimited => Ok(None),
+            Self::Limited(permits) => Arc::clone(permits).try_acquire_many_owned(count).map(Some),
+        }
+    }
+}
+
+/// Driver and its lifetime reservation must share the same Arc, including
+/// transient references held by the stats exporter.
+pub(super) struct BudgetedDriver<T> {
+    // Field drop order is intentional: joining all driver threads finishes
+    // before the reservation is returned. Do not move the permit above driver.
+    driver: T,
+    _threads: Option<OwnedSemaphorePermit>,
+}
+
+impl<T> BudgetedDriver<T> {
+    pub(super) fn new(driver: T, threads: Option<OwnedSemaphorePermit>) -> Self {
+        Self {
+            driver,
+            _threads: threads,
+        }
+    }
+}
+
+impl<T> std::ops::Deref for BudgetedDriver<T> {
+    type Target = T;
+
+    fn deref(&self) -> &T {
+        &self.driver
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::atomic::{AtomicBool, Ordering};
+    use std::task::{Context, Poll, Waker};
+    use tokio::sync::oneshot;
+
+    fn budget(limit: usize) -> DriverThreadBudget {
+        DriverThreadBudget::from_env_value(Some(OsStr::new(&limit.to_string()))).expect("valid budget")
+    }
+
+    fn poll_once<T>(future: std::pin::Pin<&mut impl Future<Output = T>>) -> Poll<T> {
+        future.poll(&mut Context::from_waker(Waker::noop()))
+    }
+
+    #[test]
+    fn configuration_accepts_only_supported_decimal_limits() {
+        for value in [None, Some(OsStr::new("0")), Some(OsStr::new("000"))] {
+            assert!(matches!(DriverThreadBudget::from_env_value(value), Ok(DriverThreadBudget::Unlimited)));
+        }
+        for limit in [1, 16, Semaphore::MAX_PERMITS] {
+            let DriverThreadBudget::Limited(permits) = budget(limit) else {
+                panic!("nonzero budget must be limited")
+            };
+            assert_eq!(permits.available_permits(), limit);
+        }
+        for value in ["", " ", " 4", "4 ", "+4", "-1", "bad", "1.5", "184467440737095516160"] {
+            assert!(
+                DriverThreadBudget::from_env_value(Some(OsStr::new(value))).is_err(),
+                "invalid config accepted: {value}"
+            );
+        }
+        for limit in [Semaphore::MAX_PERMITS + 1, usize::MAX] {
+            assert!(DriverThreadBudget::from_env_value(Some(OsStr::new(&limit.to_string()))).is_err());
+        }
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn non_utf8_configuration_fails_closed() {
+        use std::os::unix::ffi::OsStrExt;
+        assert!(DriverThreadBudget::from_env_value(Some(OsStr::from_bytes(&[0xff]))).is_err());
+    }
+
+    #[test]
+    fn weighted_reservations_reject_excess_without_partial_acquisition() {
+        let budget = budget(5);
+        let first = budget.try_reserve(3).expect("first driver fits");
+        assert!(matches!(budget.try_reserve(3), Err(TryAcquireError::NoPermits)));
+        let second = budget.try_reserve(2).expect("failed reserve must not consume capacity");
+        assert!(matches!(budget.try_reserve(1), Err(TryAcquireError::NoPermits)));
+        drop(first);
+        let replacement = budget.try_reserve(3).expect("retirement returns exact weight");
+        drop(second);
+        drop(replacement);
+        assert!(budget.try_reserve(5).is_ok());
+    }
+
+    #[test]
+    fn unlimited_policy_preserves_legacy_and_invalid_shard_counts_do_not_panic() {
+        let budget = DriverThreadBudget::Unlimited;
+        assert!(budget.try_reserve(16).expect("legacy admission").is_none());
+        assert!(budget.try_reserve(0).is_err());
+        if usize::BITS > u32::BITS {
+            assert!(budget.try_reserve(usize::MAX).is_err());
+        }
+    }
+
+    #[test]
+    fn stats_arc_keeps_budget_until_final_driver_drop() {
+        let budget = budget(2);
+        let driver = Arc::new(BudgetedDriver::new((), budget.try_reserve(2).expect("reserve driver")));
+        let stats = Arc::downgrade(&driver);
+        let exporter_reference = stats.upgrade().expect("stats borrows a live driver");
+        drop(driver);
+        assert!(budget.try_reserve(1).is_err(), "backend drop must not bypass the exporter Arc");
+        drop(exporter_reference);
+        assert!(stats.upgrade().is_none());
+        assert!(budget.try_reserve(2).is_ok(), "a Weak alone must not hold the reservation");
+    }
+
+    #[tokio::test]
+    async fn blocking_driver_drop_retains_budget_until_join_finishes() {
+        struct JoiningDriver {
+            started: Option<oneshot::Sender<()>>,
+            finish: std::sync::mpsc::Receiver<()>,
+        }
+        impl Drop for JoiningDriver {
+            fn drop(&mut self) {
+                self.started
+                    .take()
+                    .expect("single driver drop")
+                    .send(())
+                    .expect("observe drop entry");
+                self.finish
+                    .recv_timeout(std::time::Duration::from_secs(10))
+                    .expect("release simulated join");
+            }
+        }
+        let budget = budget(2);
+        let (started_tx, started_rx) = oneshot::channel();
+        let (finish_tx, finish_rx) = std::sync::mpsc::channel();
+        let driver = BudgetedDriver::new(
+            JoiningDriver {
+                started: Some(started_tx),
+                finish: finish_rx,
+            },
+            budget.try_reserve(2).expect("reserve driver"),
+        );
+        let cleanup = tokio::task::spawn_blocking(move || drop(driver));
+        started_rx.await.expect("driver drop started");
+        let unavailable_during_join = budget.try_reserve(1).is_err();
+        finish_tx.send(()).expect("finish driver join");
+        cleanup.await.expect("cleanup completes");
+        assert!(unavailable_during_join);
+        assert!(budget.try_reserve(2).is_ok());
+    }
+
+    #[tokio::test]
+    async fn failed_and_panicking_probes_return_their_thread_reservation() {
+        let budget = budget(2);
+        let threads = budget.try_reserve(2).expect("reserve failed probe");
+        let failure = super::super::uring_probe::run(move || {
+            let _threads = threads;
+            Err::<(), _>(std::io::Error::from_raw_os_error(5))
+        })
+        .await
+        .expect("probe work returns normally");
+        assert_eq!(failure.expect_err("probe error preserved").raw_os_error(), Some(5));
+        let threads = budget.try_reserve(2).expect("failed probe returned reservation");
+        assert!(
+            super::super::uring_probe::run(move || {
+                let _threads = threads;
+                panic!("injected probe panic");
+            })
+            .await
+            .is_err()
+        );
+        assert!(budget.try_reserve(2).is_ok());
+    }
+
+    #[tokio::test]
+    async fn started_probe_cancellation_retains_thread_reservation_until_completion() {
+        let budget = Arc::new(budget(2));
+        let threads = budget.try_reserve(2).expect("reserve initializing driver");
+        let (entered_tx, entered_rx) = oneshot::channel();
+        let (release_tx, release_rx) = std::sync::mpsc::channel();
+        let (dropped_tx, dropped_rx) = oneshot::channel();
+        struct Driver(Option<oneshot::Sender<()>>);
+        impl Drop for Driver {
+            fn drop(&mut self) {
+                self.0.take().expect("single drop").send(()).expect("observe cleanup");
+            }
+        }
+        let mut probe = Box::pin(super::super::uring_probe::run(move || {
+            entered_tx.send(()).expect("probe entered");
+            release_rx
+                .recv_timeout(std::time::Duration::from_secs(10))
+                .expect("finish probe");
+            BudgetedDriver::new(Driver(Some(dropped_tx)), threads)
+        }));
+        assert!(poll_once(probe.as_mut()).is_pending());
+        entered_rx.await.expect("probe started");
+        drop(probe);
+        let still_reserved = budget.try_reserve(1).is_err();
+        release_tx.send(()).expect("complete detached probe");
+        dropped_rx.await.expect("abandoned driver is cleaned up");
+        // The Drop notification precedes the permit field's destructor. Wait on
+        // the test semaphore only to observe final cleanup, never in production.
+        let DriverThreadBudget::Limited(permits) = budget.as_ref() else { panic!("limited budget") };
+        let returned = permits
+            .clone()
+            .acquire_many_owned(2)
+            .await
+            .expect("cleanup returns all thread slots");
+        assert!(still_reserved);
+        drop(returned);
+    }
+
+    #[tokio::test]
+    async fn canceling_initialization_waiter_releases_reserved_driver_slots() {
+        let budget = budget(2);
+        let threads = budget.try_reserve(2).expect("reserve pending initialization");
+        let initialization = Arc::new(Semaphore::new(0));
+        let started = Arc::new(AtomicBool::new(false));
+        let mark_started = started.clone();
+        let mut probe = Box::pin(super::super::uring_probe::run_with_permits(initialization, move || {
+            mark_started.store(true, Ordering::SeqCst);
+            BudgetedDriver::new((), threads)
+        }));
+        assert!(poll_once(probe.as_mut()).is_pending());
+        assert!(budget.try_reserve(1).is_err());
+        drop(probe);
+        assert!(!started.load(Ordering::SeqCst));
+        assert!(budget.try_reserve(2).is_ok());
+    }
+
+    #[tokio::test]
+    async fn completed_unconsumed_driver_keeps_slots_through_offloaded_cleanup() {
+        struct WakeFinished(Arc<tokio::sync::Notify>);
+        impl std::task::Wake for WakeFinished {
+            fn wake(self: Arc<Self>) {
+                self.0.notify_one();
+            }
+        }
+        struct Driver {
+            budget: Arc<DriverThreadBudget>,
+            dropped: Option<oneshot::Sender<(bool, std::thread::ThreadId)>>,
+        }
+        impl Drop for Driver {
+            fn drop(&mut self) {
+                let reservation_retained = self.budget.try_reserve(1).is_err();
+                self.dropped
+                    .take()
+                    .expect("single driver drop")
+                    .send((reservation_retained, std::thread::current().id()))
+                    .expect("observe cleanup");
+            }
+        }
+        let budget = Arc::new(budget(2));
+        let threads = budget.try_reserve(2).expect("reserve driver");
+        let (drop_tx, drop_rx) = oneshot::channel();
+        let driver = Driver {
+            budget: budget.clone(),
+            dropped: Some(drop_tx),
+        };
+        let (finish_tx, finish_rx) = std::sync::mpsc::channel();
+        let initialization = Arc::new(Semaphore::new(1));
+        let mut probe = Box::pin(super::super::uring_probe::run_with_permits(initialization, move || {
+            finish_rx
+                .recv_timeout(std::time::Duration::from_secs(10))
+                .expect("allow result publication");
+            BudgetedDriver::new(driver, threads)
+        }));
+        let finished = Arc::new(tokio::sync::Notify::new());
+        let waker = Waker::from(Arc::new(WakeFinished(finished.clone())));
+        assert!(probe.as_mut().poll(&mut Context::from_waker(&waker)).is_pending());
+        finish_tx.send(()).expect("complete driver initialization");
+        finished.notified().await;
+        assert!(budget.try_reserve(1).is_err(), "ready but unconsumed driver remains charged");
+        drop(probe);
+        let (retained_during_drop, dropped_on) = drop_rx.await.expect("detached result cleanup");
+        let DriverThreadBudget::Limited(permits) = budget.as_ref() else { panic!("limited budget") };
+        let returned = permits
+            .clone()
+            .acquire_many_owned(2)
+            .await
+            .expect("completed cleanup returns thread slots");
+        assert!(retained_during_drop);
+        assert_ne!(dropped_on, std::thread::current().id());
+        drop(returned);
+    }
+}

crates/ecstore/src/disk/uring_probe.rs

@@ -0,0 +1,261 @@
+// Copyright 2024 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+use std::io;
+use std::sync::{Arc, LazyLock};
+use tokio::sync::{OwnedSemaphorePermit, Semaphore};
+
+// This caps concurrent initialization, not live drivers, io-wq threads or bytes.
+const MAX_CONCURRENT_PROBES: usize = 4;
+static PROBE_PERMITS: LazyLock<Arc<Semaphore>> = LazyLock::new(|| Arc::new(Semaphore::new(MAX_CONCURRENT_PROBES)));
+
+/// Run only the root-independent driver probe here. Backend construction may
+/// depend on a mount-lease fd owned by the async caller and must stay with it.
+pub(super) async fn run<T: Send + 'static>(probe: impl FnOnce() -> T + Send + 'static) -> io::Result<T> {
+    run_with_permits(Arc::clone(&PROBE_PERMITS), probe).await
+}
+
+pub(super) async fn run_with_permits<T: Send + 'static>(
+    permits: Arc<Semaphore>,
+    probe: impl FnOnce() -> T + Send + 'static,
+) -> io::Result<T> {
+    // Acquisition precedes spawn: canceling a waiter cannot enqueue probe work.
+    // The closure owns its permit even after its async JoinHandle is abandoned.
+    let permit = permits.acquire_owned().await.map_err(io::Error::other)?;
+    let mut result = tokio::task::spawn_blocking(move || ProbeResult {
+        value: Some(probe()),
+        permit: Some(permit),
+    })
+    .await
+    .map_err(io::Error::other)?;
+    // Consuming the result ends initialization admission. Abandoned results
+    // retain the permit until their blocking cleanup finishes instead.
+    result
+        .value
+        .take()
+        .ok_or_else(|| io::Error::other("uring probe result was already consumed"))
+}
+
+struct ProbeResult<T: Send + 'static> {
+    value: Option<T>,
+    permit: Option<OwnedSemaphorePermit>,
+}
+
+impl<T: Send + 'static> Drop for ProbeResult<T> {
+    fn drop(&mut self) {
+        if let Some(value) = self.value.take() {
+            let permit = self.permit.take();
+            let cleanup = move || {
+                // Driver destruction can join threads. In particular, dropping
+                // a completed-but-unconsumed JoinHandle may run here on a Tokio
+                // worker, so move that destruction back to blocking work.
+                drop(value);
+                drop(permit);
+            };
+            match tokio::runtime::Handle::try_current() {
+                Ok(runtime) => {
+                    runtime.spawn_blocking(cleanup);
+                }
+                Err(_) => cleanup(),
+            }
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+    use std::task::{Context, Poll, Wake, Waker};
+    use tokio::sync::oneshot;
+
+    fn poll_once<T>(future: std::pin::Pin<&mut impl Future<Output = T>>) -> Poll<T> {
+        future.poll(&mut Context::from_waker(Waker::noop()))
+    }
+
+    #[tokio::test]
+    async fn probe_runs_outside_async_caller_thread() {
+        let caller = std::thread::current().id();
+        let worker = run(std::thread::current).await.expect("probe should finish").id();
+        assert_ne!(worker, caller);
+    }
+
+    #[tokio::test]
+    async fn concurrent_probes_never_exceed_the_initialization_limit() {
+        let permits = Arc::new(Semaphore::new(MAX_CONCURRENT_PROBES));
+        let active = Arc::new(AtomicUsize::new(0));
+        let peak = Arc::new(AtomicUsize::new(0));
+        let (entered_tx, mut entered_rx) = tokio::sync::mpsc::unbounded_channel();
+        let mut releases = Vec::new();
+        let mut probes = Vec::new();
+        for _ in 0..MAX_CONCURRENT_PROBES {
+            let (release_tx, release_rx) = std::sync::mpsc::channel();
+            releases.push(release_tx);
+            let active = active.clone();
+            let peak = peak.clone();
+            let entered = entered_tx.clone();
+            probes.push(tokio::spawn(run_with_permits(permits.clone(), move || {
+                let running = active.fetch_add(1, Ordering::SeqCst) + 1;
+                peak.fetch_max(running, Ordering::SeqCst);
+                entered.send(()).expect("observe started probe");
+                release_rx
+                    .recv_timeout(std::time::Duration::from_secs(10))
+                    .expect("release blocked probe");
+                active.fetch_sub(1, Ordering::SeqCst);
+            })));
+        }
+        for _ in 0..MAX_CONCURRENT_PROBES {
+            entered_rx.recv().await.expect("each admitted probe should start");
+        }
+        let extra_started = Arc::new(AtomicBool::new(false));
+        let mark_started = extra_started.clone();
+        let mut extra = Box::pin(run_with_permits(permits.clone(), move || mark_started.store(true, Ordering::SeqCst)));
+        let was_pending = poll_once(extra.as_mut()).is_pending();
+        let started_early = extra_started.load(Ordering::SeqCst);
+        for release in releases {
+            release.send(()).expect("release active probes");
+        }
+        for probe in probes {
+            probe.await.expect("probe task join").expect("probe execution");
+        }
+        extra.await.expect("queued probe runs after release");
+        assert!(was_pending && !started_early);
+        assert_eq!(peak.load(Ordering::SeqCst), MAX_CONCURRENT_PROBES);
+        assert_eq!(permits.available_permits(), MAX_CONCURRENT_PROBES);
+    }
+
+    #[tokio::test]
+    async fn canceling_before_admission_never_starts_probe() {
+        let permits = Arc::new(Semaphore::new(1));
+        let held = permits.clone().acquire_owned().await.expect("occupy admission");
+        let started = Arc::new(AtomicBool::new(false));
+        let mark_started = started.clone();
+        let mut waiting = Box::pin(run_with_permits(permits.clone(), move || mark_started.store(true, Ordering::SeqCst)));
+        assert!(poll_once(waiting.as_mut()).is_pending());
+        drop(waiting);
+        drop(held);
+        run_with_permits(permits.clone(), || ()).await.expect("next probe should run");
+        assert!(!started.load(Ordering::SeqCst));
+        assert_eq!(permits.available_permits(), 1);
+    }
+
+    #[tokio::test]
+    async fn canceling_started_probe_holds_capacity_until_worker_finishes() {
+        let permits = Arc::new(Semaphore::new(1));
+        let (entered_tx, entered_rx) = oneshot::channel();
+        let (release_tx, release_rx) = std::sync::mpsc::channel();
+        let mut first = Box::pin(run_with_permits(permits.clone(), move || {
+            entered_tx.send(()).expect("notify probe entry");
+            release_rx
+                .recv_timeout(std::time::Duration::from_secs(10))
+                .expect("release running probe");
+        }));
+        assert!(poll_once(first.as_mut()).is_pending());
+        entered_rx.await.expect("probe started");
+        drop(first);
+        let held_after_cancel = permits.available_permits();
+        let mut second = Box::pin(run_with_permits(permits.clone(), || ()));
+        let second_pending = poll_once(second.as_mut()).is_pending();
+        release_tx.send(()).expect("finish detached probe");
+        second.await.expect("capacity returns after real completion");
+        assert_eq!(held_after_cancel, 0);
+        assert!(second_pending);
+        assert_eq!(permits.available_permits(), 1);
+    }
+
+    #[tokio::test]
+    async fn ordinary_probe_errors_remain_unwrapped_and_release_capacity() {
+        let permits = Arc::new(Semaphore::new(1));
+        // Model both restriction and unexpected probe results. The offload layer
+        // must pass them through; only its own Acquire/Join errors are wrapped.
+        for errno in [1, 5] {
+            let result = run_with_permits(permits.clone(), move || Err::<(), _>(io::Error::from_raw_os_error(errno)))
+                .await
+                .expect("blocking work itself should complete");
+            assert_eq!(result.expect_err("probe returned an error").raw_os_error(), Some(errno));
+            assert_eq!(permits.available_permits(), 1);
+        }
+    }
+
+    #[tokio::test]
+    async fn panic_and_closed_admission_are_not_os_restriction_errors() {
+        let permits = Arc::new(Semaphore::new(1));
+        let panic_error = run_with_permits(permits.clone(), || panic!("injected probe panic"))
+            .await
+            .expect_err("worker panic must be returned");
+        assert!(
+            panic_error
+                .get_ref()
+                .and_then(|error| error.downcast_ref::<tokio::task::JoinError>())
+                .is_some()
+        );
+        assert_eq!(panic_error.raw_os_error(), None);
+        assert_eq!(permits.available_permits(), 1);
+        permits.close();
+        let closed_error = run_with_permits(permits, || panic!("closed admission must not start"))
+            .await
+            .expect_err("closed admission must return an error");
+        assert_eq!(closed_error.raw_os_error(), None);
+        assert!(
+            closed_error
+                .get_ref()
+                .and_then(|error| error.downcast_ref::<tokio::sync::AcquireError>())
+                .is_some()
+        );
+    }
+
+    #[tokio::test]
+    async fn completed_unconsumed_result_drops_outside_async_caller_with_capacity_held() {
+        struct WakeFinished(Arc<tokio::sync::Notify>);
+        impl Wake for WakeFinished {
+            fn wake(self: Arc<Self>) {
+                self.0.notify_one();
+            }
+        }
+        struct BlockingDrop {
+            entered: Option<oneshot::Sender<std::thread::ThreadId>>,
+            release: std::sync::mpsc::Receiver<()>,
+        }
+        impl Drop for BlockingDrop {
+            fn drop(&mut self) {
+                self.entered
+                    .take()
+                    .expect("single drop")
+                    .send(std::thread::current().id())
+                    .expect("observe cleanup");
+                self.release
+                    .recv_timeout(std::time::Duration::from_secs(10))
+                    .expect("release cleanup");
+            }
+        }
+        let permits = Arc::new(Semaphore::new(1));
+        let (drop_tx, drop_rx) = oneshot::channel();
+        let (release_tx, release_rx) = std::sync::mpsc::channel();
+        let (produce_tx, produce_rx) = std::sync::mpsc::channel();
+        let result = BlockingDrop {
+            entered: Some(drop_tx),
+            release: release_rx,
+        };
+        let mut probe = Box::pin(run_with_permits(permits.clone(), move || {
+            produce_rx
+                .recv_timeout(std::time::Duration::from_secs(10))
+                .expect("allow probe completion after waiter registration");
+            result
+        }));
+        let finished = Arc::new(tokio::sync::Notify::new());
+        let waker = Waker::from(Arc::new(WakeFinished(finished.clone())));
+        assert!(probe.as_mut().poll(&mut Context::from_waker(&waker)).is_pending());
+        produce_tx.send(()).expect("complete probe");
+        // JoinHandle wakes only after its result is ready. Do not poll the
+        // outer future again: cancel precisely before it consumes that result.
+        finished.notified().await;
+        drop(probe);
+        let dropped_on = drop_rx.await.expect("abandoned result cleanup should run");
+        let capacity_during_drop = permits.available_permits();
+        release_tx.send(()).expect("allow cleanup to finish");
+        let released = permits.clone().acquire_owned().await.expect("cleanup returns permit");
+        assert_ne!(dropped_on, std::thread::current().id());
+        assert_eq!(capacity_during_drop, 0);
+        drop(released);
+    }
+}

crates/ecstore/src/disk/uring_read_budget.rs

@@ -0,0 +1,156 @@
+// Copyright 2024 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+use std::ffi::OsStr;
+
+#[cfg(target_os = "linux")]
+pub(super) const ENV_TOTAL: &str = "RUSTFS_IO_URING_READ_BUDGET_TOTAL_BYTES";
+#[cfg(target_os = "linux")]
+pub(super) const ENV_DRIVER: &str = "RUSTFS_IO_URING_READ_BUDGET_DRIVER_BYTES";
+
+/// Parsed independently of Linux driver construction so configuration checks
+/// never require a kernel probe or process-wide environment mutation.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub(super) enum ReadBudgetConfig {
+    Disabled,
+    Limited { total: usize, per_driver: usize },
+}
+
+#[derive(Debug)]
+pub(super) struct InvalidReadBudget;
+
+impl std::fmt::Display for InvalidReadBudget {
+    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        formatter.write_str(
+            "expected both read-budget settings as positive decimal byte counts, with driver quota no greater than total capacity or Semaphore::MAX_PERMITS",
+        )
+    }
+}
+
+impl std::error::Error for InvalidReadBudget {}
+
+impl ReadBudgetConfig {
+    pub(super) fn from_env_values(total: Option<&OsStr>, per_driver: Option<&OsStr>) -> Result<Self, InvalidReadBudget> {
+        let (total, per_driver) = match (total, per_driver) {
+            (None, None) => return Ok(Self::Disabled),
+            (Some(total), Some(per_driver)) => (total, per_driver),
+            _ => return Err(InvalidReadBudget),
+        };
+        let parse = |raw: &OsStr| {
+            let text = raw.to_str().ok_or(InvalidReadBudget)?;
+            if text.is_empty() || !text.bytes().all(|byte| byte.is_ascii_digit()) {
+                return Err(InvalidReadBudget);
+            }
+            text.parse::<usize>().map_err(|_| InvalidReadBudget)
+        };
+        let total = parse(total)?;
+        let per_driver = parse(per_driver)?;
+        if total == 0 || per_driver == 0 || per_driver > total || per_driver > tokio::sync::Semaphore::MAX_PERMITS {
+            return Err(InvalidReadBudget);
+        }
+        Ok(Self::Limited { total, per_driver })
+    }
+}
+
+/// Clones retain the same process pool, including across disk reconstruction.
+/// The library, rather than the backend wrapper, retains leaked reservations.
+#[cfg(target_os = "linux")]
+#[derive(Clone)]
+pub(super) enum DriverReadBudget {
+    Disabled,
+    Limited {
+        pool: rustfs_uring::SharedReadBudget,
+        per_driver: usize,
+    },
+}
+
+#[cfg(target_os = "linux")]
+impl DriverReadBudget {
+    pub(super) fn from_config(config: ReadBudgetConfig) -> Result<Self, InvalidReadBudget> {
+        match config {
+            ReadBudgetConfig::Disabled => Ok(Self::Disabled),
+            ReadBudgetConfig::Limited { total, per_driver } => Ok(Self::Limited {
+                pool: rustfs_uring::SharedReadBudget::new(total).map_err(|_| InvalidReadBudget)?,
+                per_driver,
+            }),
+        }
+    }
+
+    pub(super) fn start_driver(
+        &self,
+        entries: u32,
+        shards: usize,
+    ) -> Result<rustfs_uring::UringDriver, rustfs_uring::ProbeFailure> {
+        match self {
+            Self::Disabled => rustfs_uring::UringDriver::probe_and_start_sharded(entries, shards),
+            Self::Limited { pool, per_driver } => rustfs_uring::UringDriver::probe_and_start_with_shared_budget(
+                entries,
+                shards,
+                rustfs_uring::ReadLimits {
+                    max_read_len: None,
+                    max_in_flight_bytes: Some(*per_driver),
+                },
+                pool,
+            ),
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn parse(total: Option<&str>, per_driver: Option<&str>) -> Result<ReadBudgetConfig, InvalidReadBudget> {
+        ReadBudgetConfig::from_env_values(total.map(OsStr::new), per_driver.map(OsStr::new))
+    }
+
+    #[test]
+    fn both_unset_preserve_legacy_but_partial_configuration_fails() {
+        assert_eq!(parse(None, None).expect("unset settings"), ReadBudgetConfig::Disabled);
+        assert!(parse(Some("8"), None).is_err());
+        assert!(parse(None, Some("8")).is_err());
+        assert!(parse(Some("0"), Some("0")).is_err(), "zero is not an implicit unlimited mode");
+    }
+
+    #[test]
+    fn valid_pair_keeps_usize_total_and_exact_driver_quota() {
+        for (total, per_driver) in [(1, 1), (16, 8), (usize::MAX, tokio::sync::Semaphore::MAX_PERMITS)] {
+            assert_eq!(
+                parse(Some(&total.to_string()), Some(&per_driver.to_string())).expect("valid quota pair"),
+                ReadBudgetConfig::Limited { total, per_driver }
+            );
+        }
+        assert_eq!(
+            parse(Some("00016"), Some("00008")).expect("leading zeros remain decimal"),
+            ReadBudgetConfig::Limited {
+                total: 16,
+                per_driver: 8
+            }
+        );
+    }
+
+    #[test]
+    fn malformed_or_impossible_pair_never_removes_the_limit() {
+        for value in ["", " ", "0", " 8", "8 ", "+8", "-1", "8MiB", "1.5", "184467440737095516160"] {
+            assert!(parse(Some(value), Some("1")).is_err(), "invalid total accepted");
+            assert!(parse(Some("16"), Some(value)).is_err(), "invalid driver quota accepted");
+        }
+        assert!(parse(Some("7"), Some("8")).is_err());
+        assert!(
+            parse(
+                Some(&usize::MAX.to_string()),
+                Some(&(tokio::sync::Semaphore::MAX_PERMITS + 1).to_string())
+            )
+            .is_err()
+        );
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn non_unicode_configuration_is_rejected_on_either_side() {
+        use std::os::unix::ffi::OsStrExt;
+        let invalid = OsStr::from_bytes(&[0xff]);
+        assert!(ReadBudgetConfig::from_env_values(Some(invalid), Some(OsStr::new("1"))).is_err());
+        assert!(ReadBudgetConfig::from_env_values(Some(OsStr::new("8")), Some(invalid)).is_err());
+    }
+}

crates/ecstore/src/disk/uring_read_chunks.rs

@@ -0,0 +1,95 @@
+// Copyright 2024 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+use std::ffi::OsStr;
+
+const MIN_READ_CHUNK_BYTES: usize = 4096;
+const DEFAULT_READ_CHUNK_BYTES: usize = 128 << 20;
+
+/// Validated logical length cap for one io_uring read operation. Direct-I/O
+/// padding, parallel reads and application result allocations are not included.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub(super) struct ReadChunkSize(usize);
+
+#[derive(Debug)]
+pub(super) struct InvalidReadChunkSize;
+
+impl std::fmt::Display for InvalidReadChunkSize {
+    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        formatter.write_str("expected an unsigned decimal integer between 4096 and 134217728 bytes")
+    }
+}
+
+impl std::error::Error for InvalidReadChunkSize {}
+
+impl ReadChunkSize {
+    pub(super) fn from_env_value(value: Option<&OsStr>) -> Result<Self, InvalidReadChunkSize> {
+        let Some(value) = value else { return Ok(Self(DEFAULT_READ_CHUNK_BYTES)) };
+        let text = value.to_str().ok_or(InvalidReadChunkSize)?;
+        if text.is_empty() || !text.bytes().all(|byte| byte.is_ascii_digit()) {
+            return Err(InvalidReadChunkSize);
+        }
+        let size = text.parse::<usize>().map_err(|_| InvalidReadChunkSize)?;
+        if !(MIN_READ_CHUNK_BYTES..=DEFAULT_READ_CHUNK_BYTES).contains(&size) {
+            return Err(InvalidReadChunkSize);
+        }
+        Ok(Self(size))
+    }
+
+    pub(super) fn get(self) -> usize {
+        self.0
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn missing_configuration_preserves_128_mib_default() {
+        assert_eq!(ReadChunkSize::from_env_value(None).expect("default config").get(), 128 << 20);
+    }
+
+    #[test]
+    fn accepts_range_boundaries_and_non_block_multiple_caps() {
+        for size in [4096, 4097, 65536, 128 << 20] {
+            assert_eq!(
+                ReadChunkSize::from_env_value(Some(OsStr::new(&size.to_string())))
+                    .expect("valid cap")
+                    .get(),
+                size
+            );
+        }
+    }
+
+    #[test]
+    fn invalid_configuration_never_becomes_the_default() {
+        for value in [
+            "",
+            " ",
+            "0",
+            "4095",
+            "134217729",
+            "-4096",
+            "+4096",
+            " 4096",
+            "4096 ",
+            "4KiB",
+            "4096.0",
+            "184467440737095516160",
+        ] {
+            assert!(
+                ReadChunkSize::from_env_value(Some(OsStr::new(value))).is_err(),
+                "invalid cap accepted: {value}"
+            );
+        }
+        assert!(ReadChunkSize::from_env_value(Some(OsStr::new(&usize::MAX.to_string()))).is_err());
+    }
+
+    #[cfg(unix)]
+    #[test]
+    fn non_utf8_configuration_is_rejected() {
+        use std::os::unix::ffi::OsStrExt;
+        assert!(ReadChunkSize::from_env_value(Some(OsStr::from_bytes(&[0xff]))).is_err());
+    }
+}

crates/ecstore/src/disk/uring_result_budget.rs

@@ -0,0 +1,183 @@
+// Copyright 2024 RustFS Team
+// SPDX-License-Identifier: Apache-2.0
+
+use bytes::Bytes;
+use std::ffi::OsStr;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+#[cfg(target_os = "linux")]
+pub(super) const ENV_RESULT: &str = "RUSTFS_IO_URING_READ_RESULT_BUDGET_BYTES";
+
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub(super) enum ResultBudgetConfig {
+    Disabled,
+    Limited { bytes: usize },
+}
+
+#[derive(Debug)]
+pub(super) struct InvalidResultBudget;
+
+impl std::fmt::Display for InvalidResultBudget {
+    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        formatter.write_str("expected a positive decimal result-budget byte count")
+    }
+}
+
+impl std::error::Error for InvalidResultBudget {}
+
+impl ResultBudgetConfig {
+    pub(super) fn from_env_value(value: Option<&OsStr>) -> Result<Self, InvalidResultBudget> {
+        let Some(value) = value else {
+            return Ok(Self::Disabled);
+        };
+        let value = value.to_str().ok_or(InvalidResultBudget)?;
+        if value.is_empty() || !value.bytes().all(|byte| byte.is_ascii_digit()) {
+            return Err(InvalidResultBudget);
+        }
+        let bytes = value.parse::<usize>().map_err(|_| InvalidResultBudget)?;
+        if bytes == 0 {
+            return Err(InvalidResultBudget);
+        }
+        Ok(Self::Limited { bytes })
+    }
+}
+
+#[derive(Clone, Debug)]
+pub(super) enum ProcessResultBudget {
+    Disabled,
+    Limited(Arc<ResultBudgetInner>),
+}
+
+#[derive(Debug)]
+pub(super) struct ResultBudgetInner {
+    available: AtomicUsize,
+}
+
+impl ProcessResultBudget {
+    pub(super) fn from_config(config: ResultBudgetConfig) -> Self {
+        match config {
+            ResultBudgetConfig::Disabled => Self::Disabled,
+            ResultBudgetConfig::Limited { bytes } => Self::Limited(Arc::new(ResultBudgetInner {
+                available: AtomicUsize::new(bytes),
+            })),
+        }
+    }
+
+    pub(super) fn reserve(&self, bytes: usize) -> Result<Option<ResultBudgetReservation>, std::io::Error> {
+        let Self::Limited(inner) = self else {
+            return Ok(None);
+        };
+        if bytes == 0 {
+            return Ok(None);
+        }
+        inner
+            .available
+            .fetch_update(Ordering::AcqRel, Ordering::Acquire, |available| available.checked_sub(bytes))
+            .map(|_| {
+                Some(ResultBudgetReservation {
+                    inner: Arc::clone(inner),
+                    bytes,
+                })
+            })
+            .map_err(|_| std::io::Error::new(std::io::ErrorKind::WouldBlock, ResultBudgetExhausted))
+    }
+
+    #[cfg(test)]
+    pub(super) fn available(&self) -> Option<usize> {
+        match self {
+            Self::Disabled => None,
+            Self::Limited(inner) => Some(inner.available.load(Ordering::Acquire)),
+        }
+    }
+}
+
+#[derive(Debug)]
+pub(super) struct ResultBudgetReservation {
+    inner: Arc<ResultBudgetInner>,
+    bytes: usize,
+}
+
+impl Drop for ResultBudgetReservation {
+    fn drop(&mut self) {
+        self.inner.available.fetch_add(self.bytes, Ordering::Release);
+    }
+}
+
+/// Keeps the result-budget reservation alive until every returned `Bytes` clone
+/// is dropped. `Bytes::from_owner` retains this owner without copying the data.
+pub(super) struct BudgetedBytes {
+    bytes: Bytes,
+    _reservation: Option<ResultBudgetReservation>,
+}
+
+impl BudgetedBytes {
+    pub(super) fn wrap(bytes: Bytes, reservation: Option<ResultBudgetReservation>) -> Bytes {
+        match reservation {
+            Some(reservation) => Bytes::from_owner(Self {
+                bytes,
+                _reservation: Some(reservation),
+            }),
+            None => bytes,
+        }
+    }
+}
+
+impl AsRef<[u8]> for BudgetedBytes {
+    fn as_ref(&self) -> &[u8] {
+        self.bytes.as_ref()
+    }
+}
+
+/// Marker carried by `WouldBlock` so the io_uring backend does not route a
+/// process-budget denial through the unbounded std fallback.
+#[derive(Debug)]
+pub(super) struct ResultBudgetExhausted;
+
+impl std::fmt::Display for ResultBudgetExhausted {
+    fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        formatter.write_str("io_uring result budget exhausted")
+    }
+}
+
+impl std::error::Error for ResultBudgetExhausted {}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn configuration_is_strict_and_unset_is_disabled() {
+        assert_eq!(ResultBudgetConfig::from_env_value(None).unwrap(), ResultBudgetConfig::Disabled);
+        assert_eq!(
+            ResultBudgetConfig::from_env_value(Some(OsStr::new("4096"))).unwrap(),
+            ResultBudgetConfig::Limited { bytes: 4096 }
+        );
+        for value in ["", "0", " 8", "+8", "8 ", "8MiB"] {
+            assert!(ResultBudgetConfig::from_env_value(Some(OsStr::new(value))).is_err());
+        }
+    }
+
+    #[test]
+    fn reservation_is_refunded_only_after_owner_drop() {
+        let budget = ProcessResultBudget::from_config(ResultBudgetConfig::Limited { bytes: 8 });
+        let reservation = budget.reserve(8).unwrap().unwrap();
+        assert_eq!(budget.available(), Some(0));
+        assert!(budget.reserve(1).is_err());
+        drop(reservation);
+        assert_eq!(budget.available(), Some(8));
+    }
+
+    #[test]
+    fn bytes_owner_keeps_reservation_through_clones() {
+        let budget = ProcessResultBudget::from_config(ResultBudgetConfig::Limited { bytes: 4 });
+        let reservation = budget.reserve(4).unwrap().unwrap();
+        let bytes = BudgetedBytes::wrap(Bytes::from_static(b"data"), Some(reservation));
+        let clone = bytes.clone();
+        assert_eq!(budget.available(), Some(0));
+        drop(bytes);
+        assert_eq!(budget.available(), Some(0));
+        drop(clone);
+        assert_eq!(budget.available(), Some(4));
+    }
+}

crates/ecstore/src/erasure/coding/decode.rs

@@ -5267,6 +5267,71 @@ mod tests {
         assert_eq!(DATA_SHARDS + 1, bufs.iter().filter(|buf| buf.is_some()).count());
     }
 
+    /// A peer that sends part of a shard and then stops must not pin the stripe
+    /// for the full read timeout. The other EC 2+2 shards already hold a
+    /// decode-plus-verification quorum, so the lockstep hedge retires the
+    /// stalled reader and the stripe completes.
+    #[tokio::test]
+    async fn test_lockstep_hedges_shard_that_stops_mid_read() {
+        const NUM_SHARDS: usize = 1;
+        const BLOCK_SIZE: usize = 64;
+        const DATA_SHARDS: usize = 2;
+        const PARITY_SHARDS: usize = 2;
+        const SHARD_SIZE: usize = BLOCK_SIZE / DATA_SHARDS;
+
+        let hash_algo = HashAlgorithm::None;
+        let readers = vec![
+            Some(BitrotReader::new(
+                TestShardReader::PartialThenPending {
+                    data: vec![0xab],
+                    emitted: false,
+                },
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![1_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![2_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo.clone(),
+                false,
+            )),
+            Some(BitrotReader::new(
+                TestShardReader::Ready(Cursor::new(vec![3_u8; SHARD_SIZE * NUM_SHARDS])),
+                SHARD_SIZE,
+                hash_algo,
+                false,
+            )),
+        ];
+
+        let erasure = Erasure::new(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE);
+        let mut parallel_reader = ParallelReader::new_with_metrics_path_read_costs_timeout_and_reconstruction_verification(
+            readers,
+            erasure,
+            0,
+            NUM_SHARDS * BLOCK_SIZE,
+            None,
+            vec![ShardReadCost::Unknown; 4],
+            Duration::from_secs(60),
+            true,
+        );
+
+        let (bufs, errs) = tokio::time::timeout(Duration::from_millis(500), parallel_reader.read())
+            .await
+            .expect("a shard that stops mid-read must be hedged instead of waiting out the 60s read timeout");
+
+        assert!(matches!(&errs[0], Some(DiskError::Io(err)) if err.kind() == ErrorKind::TimedOut));
+        assert!(parallel_reader.readers[0].is_none());
+        assert!(bufs[0].is_none());
+        assert_eq!(DATA_SHARDS + 1, bufs.iter().filter(|buf| buf.is_some()).count());
+    }
+
     #[tokio::test(start_paused = true)]
     async fn lockstep_reopens_hedged_shard_after_later_peer_loss() {
         const DATA_SHARDS: usize = 2;

crates/ecstore/src/erasure/coding/encode.rs

@@ -25,6 +25,7 @@ use bytes::{Bytes, BytesMut};
 use futures::StreamExt;
 use futures::stream::FuturesUnordered;
 use rustfs_utils::HashAlgorithm;
+use std::pin::Pin;
 use std::sync::Arc;
 use std::time::Instant;
 use std::vec;
@@ -325,6 +326,80 @@ impl Default for WriteProgressPolicy {
     }
 }
 
+/// After write quorum is already in hand, how long a slower shard may still
+/// finish before it is dropped. The per-shard stall budget (30s by default)
+/// is what a black-hole peer is allowed to consume *before* quorum exists.
+/// Once quorum exists, waiting out that whole budget pins the caller for as
+/// long as the client request timeout, so the client gives up on a write the
+/// remaining disks have already accepted.
+const WRITE_QUORUM_STRAGGLER_GRACE: std::time::Duration = std::time::Duration::from_secs(1);
+
+fn post_quorum_straggler_grace(stall_timeout: Option<std::time::Duration>) -> Option<std::time::Duration> {
+    stall_timeout.map(|stall| stall.min(WRITE_QUORUM_STRAGGLER_GRACE))
+}
+
+/// Drive shard ops until write quorum has succeeded and stragglers have had
+/// [`post_quorum_straggler_grace`] to finish. Returns `true` for each slot
+/// whose future completed; `false` means the caller must fail that slot.
+async fn await_write_quorum<F>(
+    write_quorum: usize,
+    stall_timeout: Option<std::time::Duration>,
+    slot_count: usize,
+    mut futures: FuturesUnordered<F>,
+) -> Vec<bool>
+where
+    F: std::future::Future<Output = (usize, bool)>,
+{
+    let mut saw = vec![false; slot_count];
+    let mut successes = 0usize;
+    let mut grace: Option<Pin<Box<tokio::time::Sleep>>> = None;
+    loop {
+        if futures.is_empty() {
+            break;
+        }
+        let completed = match grace.as_mut() {
+            Some(sleep) => {
+                tokio::select! {
+                    biased;
+                    item = futures.next() => item,
+                    () = sleep.as_mut() => break,
+                }
+            }
+            None => futures.next().await,
+        };
+        let Some((index, succeeded)) = completed else {
+            break;
+        };
+        if let Some(slot) = saw.get_mut(index) {
+            *slot = true;
+        }
+        if succeeded {
+            successes = successes.saturating_add(1);
+            if successes >= write_quorum
+                && grace.is_none()
+                && let Some(delay) = post_quorum_straggler_grace(stall_timeout)
+            {
+                grace = Some(Box::pin(tokio::time::sleep(delay)));
+            }
+        }
+    }
+    saw
+}
+
+fn abandon_unfinished_writers(writers: &mut [Option<BitrotWriterWrapper>], errs: &mut [Option<Error>], saw: &[bool]) {
+    for (index, seen) in saw.iter().enumerate() {
+        if *seen || errs.get(index).and_then(Option::as_ref).is_some() {
+            continue;
+        }
+        if let Some(err) = errs.get_mut(index) {
+            *err = Some(Error::Timeout);
+        }
+        if let Some(writer) = writers.get_mut(index) {
+            *writer = None;
+        }
+    }
+}
+
 pub(crate) struct MultiWriter<'a> {
     writers: &'a mut [Option<BitrotWriterWrapper>],
     integrity: Option<&'a mut IntegrityBuilder>,
@@ -418,9 +493,12 @@ impl<'a> MultiWriter<'a> {
         assert_eq!(shards.len(), self.writers.len());
 
         let budget = self.next_progress_budget();
-        {
-            let mut futures = FuturesUnordered::new();
-            for ((writer_opt, err), shard) in self.writers.iter_mut().zip(self.errs.iter_mut()).zip(shards) {
+        let stall_timeout = self.policy.stall_timeout;
+        let write_quorum = self.write_quorum;
+        let slot_count = self.writers.len();
+        let saw = {
+            let futures = FuturesUnordered::new();
+            for (index, ((writer_opt, err), shard)) in self.writers.iter_mut().zip(self.errs.iter_mut()).zip(shards).enumerate() {
                 if err.is_some() {
                     continue; // Skip if we already have an error for this writer
                 }
@@ -428,7 +506,9 @@ impl<'a> MultiWriter<'a> {
                 // failed and its disk dropped, so a stalled peer cannot pin an
                 // otherwise-healthy write quorum (rustfs/backlog#1319). `budget`
                 // is recomputed per block, so it bounds a stall — not the total
-                // transfer time — and a slow-but-honest writer is never killed.
+                // transfer time — and a slow-but-honest writer is never killed
+                // while quorum is still open. Once quorum has succeeded, a
+                // straggler only gets `WRITE_QUORUM_STRAGGLER_GRACE` more time.
                 futures.push(async move {
                     match budget {
                         Some(budget) => match tokio::time::timeout(budget, Self::write_shard(writer_opt, err, shard)).await {
@@ -440,10 +520,13 @@ impl<'a> MultiWriter<'a> {
                         },
                         None => Self::write_shard(writer_opt, err, shard).await,
                     }
+                    let succeeded = err.is_none() && writer_opt.is_some();
+                    (index, succeeded)
                 });
             }
-            while let Some(()) = futures.next().await {}
-        }
+            await_write_quorum(write_quorum, stall_timeout, slot_count, futures).await
+        };
+        abandon_unfinished_writers(self.writers, &mut self.errs, &saw);
 
         let nil_count = self.errs.iter().filter(|&e| e.is_none()).count();
         if nil_count >= self.write_quorum {
@@ -490,9 +573,12 @@ impl<'a> MultiWriter<'a> {
     pub async fn shutdown(&mut self) -> std::io::Result<()> {
         crate::hp_guard!("MultiWriter::shutdown");
         let budget = self.next_progress_budget();
-        {
-            let mut futures = FuturesUnordered::new();
-            for (writer_opt, err) in self.writers.iter_mut().zip(self.errs.iter_mut()) {
+        let stall_timeout = self.policy.stall_timeout;
+        let write_quorum = self.write_quorum;
+        let slot_count = self.writers.len();
+        let saw = {
+            let futures = FuturesUnordered::new();
+            for (index, (writer_opt, err)) in self.writers.iter_mut().zip(self.errs.iter_mut()).enumerate() {
                 if err.is_some() {
                     continue;
                 }
@@ -501,6 +587,8 @@ impl<'a> MultiWriter<'a> {
                 // here forever for a small object whose bytes were fully buffered
                 // (so `write` never blocked). Bound it with the same progress
                 // budget and drop the stalled writer before the quorum check.
+                // After quorum has shut down, do not keep waiting out the rest of
+                // that budget for a peer that has stopped answering.
                 futures.push(async move {
                     match budget {
                         Some(budget) => match tokio::time::timeout(budget, Self::shutdown_writer(writer_opt, err)).await {
@@ -512,10 +600,13 @@ impl<'a> MultiWriter<'a> {
                         },
                         None => Self::shutdown_writer(writer_opt, err).await,
                     }
+                    let succeeded = err.is_none() && writer_opt.is_some();
+                    (index, succeeded)
                 });
             }
-            while let Some(()) = futures.next().await {}
-        }
+            await_write_quorum(write_quorum, stall_timeout, slot_count, futures).await
+        };
+        abandon_unfinished_writers(self.writers, &mut self.errs, &saw);
 
         let nil_count = self.errs.iter().filter(|&e| e.is_none()).count();
         if nil_count >= self.write_quorum {
@@ -2134,6 +2225,68 @@ mod tests {
         assert!(writers[1].is_some() && writers[2].is_some() && writers[3].is_some());
     }
 
+    /// Quorum is met by the three healthy writers at t=0. The black-hole peer
+    /// must be abandoned after the post-quorum grace, not after the full 5s
+    /// stall budget. Waiting out the stall makes a 30s default stall consume
+    /// the whole client request budget.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_quorum_abandons_black_hole_after_straggler_grace() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..3).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(StallingWriter::stalls_on_write(0), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[0].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[2].clone()), 64)),
+        ];
+
+        let started = tokio::time::Instant::now();
+        {
+            let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            mw.write(four_shards())
+                .await
+                .expect("quorum must succeed without waiting out the black-hole stall");
+        }
+        let elapsed = started.elapsed();
+        assert!(
+            elapsed < Duration::from_secs(2),
+            "post-quorum grace must cut the black-hole wait short of the 5s stall, elapsed {elapsed:?}"
+        );
+        assert!(
+            elapsed >= WRITE_QUORUM_STRAGGLER_GRACE,
+            "the healthy quorum must still give the straggler its grace window, elapsed {elapsed:?}"
+        );
+        assert!(writers[0].is_none(), "the black-hole writer must be dropped once grace expires");
+        assert!(writers[1].is_some() && writers[2].is_some() && writers[3].is_some());
+    }
+
+    /// A slow-but-honest shard that finishes inside the post-quorum grace keeps
+    /// its writer. The grace exists to drop peers that have stopped, not to
+    /// discard a disk that is merely behind the fastest quorum.
+    #[tokio::test(start_paused = true)]
+    async fn multi_writer_quorum_keeps_straggler_that_finishes_within_grace() {
+        let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..4).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
+        let mut writers = vec![
+            Some(bitrot_writer_plain(SlowWriter::new(Duration::from_millis(200)), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[1].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[2].clone()), 64)),
+            Some(bitrot_writer_plain(DeferredCommitWriter::new(committed[3].clone()), 64)),
+        ];
+
+        {
+            let policy = WriteProgressPolicy::new(Duration::from_secs(5), Duration::ZERO);
+            let mut mw = MultiWriter::with_policy(&mut writers, 3, policy);
+            mw.write(four_shards())
+                .await
+                .expect("a straggler inside the grace window must still satisfy the write");
+        }
+
+        assert!(
+            writers.iter().all(Option::is_some),
+            "a shard that finishes within the grace window must not be dropped"
+        );
+    }
+
     // Two black-hole writers drop the healthy count to 2/4, below the quorum of
     // 3, so the write must fail cleanly (not hang).
     #[tokio::test(start_paused = true)]

crates/ecstore/src/services/notification_sys.rs

@@ -23,7 +23,6 @@ use crate::error::{Error, Result};
 use crate::layout::endpoints::EndpointServerPools;
 use crate::runtime::sources as runtime_sources;
 use crate::services::metrics_realtime::{CollectMetricsOpts, MetricType};
-use crate::services::rebalance::RebalSaveOpt;
 use crate::storage_api_contracts::admin::StorageAdminApi;
 use bytes::Bytes;
 use futures::future::join_all;
@@ -2455,30 +2454,11 @@ impl NotificationSys {
             None => store.current_rebalance_id().await,
         };
         match store.stop_rebalance_for_id(local_rebalance_id.as_deref()).await {
-            Ok(_) => {
-                let save_result = match local_rebalance_id.as_deref() {
-                    Some(expected_id) => {
-                        store
-                            .save_rebalance_stats_for_id(usize::MAX, RebalSaveOpt::StoppedAt, expected_id)
-                            .await
-                    }
-                    None => Ok(()),
-                };
-                if let Err(err) = save_result {
-                    error!(
-                        event = EVENT_NOTIFICATION_PEER_PROPAGATION,
-                        component = LOG_COMPONENT_ECSTORE,
-                        subsystem = LOG_SUBSYSTEM_NOTIFICATION,
-                        action = "stop_rebalance",
-                        result = "local_save_failed",
-                        error = %err,
-                        "notification peer propagation"
-                    );
-                    return Err(Error::other(format!(
-                        "local stop_rebalance save_rebalance_stats(stopped_at) failed: {err}"
-                    )));
-                }
-            }
+            // `stop_rebalance_for_id` durably saves the terminal snapshot after
+            // activation readers drain. Do not acquire the same metadata lock
+            // again for an equivalent `StoppedAt` save while cleanup is still
+            // releasing its own metadata work.
+            Ok(_) => {}
             Err(err) => {
                 error!(
                     event = EVENT_NOTIFICATION_PEER_PROPAGATION,

crates/ecstore/src/services/rebalance/control.rs

@@ -630,7 +630,7 @@ impl ECStore {
     pub async fn prepare_rebalance_stop(&self) -> Result<Option<String>> {
         let _start_guard = self.start_gate.lock().await;
 
-        {
+        let (local_stop_id, activation_gate) = {
             let mut rebalance_meta = self.rebalance_meta.write().await;
             if let Some(meta) = rebalance_meta.as_mut()
                 && is_rebalance_conflicting_with_decommission(meta)
@@ -641,7 +641,22 @@ impl ECStore {
                     .cancel();
                 #[cfg(any(test, feature = "test-util"))]
                 observe_rebalance_stop_wait_attempt(Some(meta.id.as_str()));
+                (Some(meta.id.clone()), Some(Arc::clone(&meta.activation_gate)))
+            } else {
+                (None, None)
             }
+        };
+
+        // An in-flight entry already owns the activation read gate. Admission
+        // has been cancelled above, so do not wait on that same gate merely to
+        // refresh a stop target: the eventual stop operation will take the
+        // write gate after the entry drains. If the gate is currently free,
+        // retain the historical reload behaviour (which observes a worker that
+        // reached a terminal state just before this request).
+        if let Some(gate) = activation_gate
+            && gate.try_write_owned().is_err()
+        {
+            return Ok(local_stop_id);
         }
 
         self.load_rebalance_meta_under_start_gate().await?;

crates/ecstore/src/store/bucket.rs

@@ -694,14 +694,13 @@ impl ECStore {
         let lock_newly_enabled = opts.lock_enabled && !meta.lock_enabled;
         if opts.lock_enabled {
             meta.lock_enabled = true;
-            meta.object_lock_config_xml =
-                crate::bucket::utils::serialize::<ObjectLockConfiguration>(&ENABLED_OBJECT_LOCK_CONFIG)?;
+            meta.object_lock_config_xml = crate::bucket::utils::serialize(&*crate::store::ENABLED_OBJECT_LOCK_CONFIG)?;
             meta.versioning_config_xml = crate::bucket::utils::serialize::<VersioningConfiguration>(&ENABLED_VERSIONING_CONFIG)?;
         }
 
         let metadata_persisted_before_physical = confirmed_missing && !is_meta_bucketname(bucket) && opts.lock_enabled;
         if metadata_persisted_before_physical {
-            metadata_sys::set_new_bucket_metadata_in(&self.ctx, meta.clone()).await?;
+            metadata_sys::set_new_bucket_metadata_intent_in(&self.ctx, meta.clone()).await?;
             if bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
                 || metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
                 || ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
@@ -759,12 +758,12 @@ impl ECStore {
 
         let metadata_result = async {
             if metadata_persisted_before_physical {
-                return Ok(());
+                return metadata_sys::commit_bucket_metadata_in(&self.ctx, meta).await;
             }
             if is_meta_bucketname(bucket) {
                 metadata_sys::set_bucket_metadata_in(&self.ctx, meta).await
             } else if existing_incarnation_is_authoritative && !lock_newly_enabled {
-                metadata_sys::cache_bucket_metadata_in(&self.ctx, meta).await
+                metadata_sys::commit_bucket_metadata_in(&self.ctx, meta).await
             } else {
                 metadata_sys::set_new_bucket_metadata_in(&self.ctx, meta).await
             }
@@ -910,6 +909,9 @@ impl ECStore {
                 // permission to serve degraded reads.
                 return Err(Error::ErasureReadQuorum);
             }
+            if metadata.lock_enabled && !metadata.bucket_creation_committed {
+                return Err(Error::ErasureWriteQuorum);
+            }
             if metadata.name != bucket {
                 return Err(Error::FileCorrupt);
             }
@@ -2393,6 +2395,435 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    #[serial]
+    async fn lazy_metadata_load_uses_read_quorum_for_non_lock_bucket() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("lazy-read-quorum-{}", Uuid::new_v4().simple());
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("healthy namespace should accept bucket creation");
+
+        // Replace the instance metadata system with a cold cache while keeping the
+        // persisted metadata and physical bucket in place.
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let set = &store.pools[0].disk_set[0];
+        let offline = take_set_disks_offline(&store, set, &[0, 1]).await;
+
+        metadata_sys::get_on_demand_migration_config_in(&store, &bucket)
+            .await
+            .expect("cold metadata load should admit the exact namespace read quorum");
+        assert!(
+            metadata_sys::get_in(&store.ctx, &bucket).await.is_ok(),
+            "the lazy load should publish authoritative metadata"
+        );
+        assert_eq!(
+            store
+                .get_bucket_info_from_sets(&bucket, &BucketOptions::default())
+                .await
+                .expect_err("bucket mutations must retain their majority namespace check"),
+            StorageError::ErasureWriteQuorum
+        );
+
+        restore_set_disks(&store, set, offline).await;
+    }
+
+    /// EC 2+2 with two disks offline still has object read quorum and has lost
+    /// write quorum. A node that has not cached the bucket must still serve
+    /// HEAD bucket, GET, HEAD object, and List; a write must fail.
+    #[tokio::test]
+    #[serial]
+    async fn cold_bucket_reads_succeed_below_write_quorum() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("cold-read-quorum-{}", Uuid::new_v4().simple());
+        let object = "seed-object";
+        let body = b"cold bucket reads must survive lost write quorum".to_vec();
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("healthy namespace should accept bucket creation");
+        store
+            .put_object(&bucket, object, &mut PutObjReader::from_vec(body.clone()), &ObjectOptions::default())
+            .await
+            .expect("healthy erasure set should accept the seed object");
+
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        assert!(
+            metadata_sys::get_in(&store.ctx, &bucket).await.is_err(),
+            "the replacement metadata system must start without this bucket"
+        );
+        let set = &store.pools[0].disk_set[0];
+        let lock = set
+            .new_ns_lock(&bucket, object)
+            .await
+            .expect("seed namespace lock should resolve");
+        drop(
+            lock.get_write_lock(Duration::from_secs(30))
+                .await
+                .expect("seed physical fanout must finish before taking disks offline"),
+        );
+        let offline = take_set_disks_offline(&store, set, &[0, 1]).await;
+
+        let info = store
+            .get_bucket_info(&bucket, &BucketOptions::default())
+            .await
+            .expect("HEAD bucket must succeed at read quorum before the bucket is cached");
+        assert_eq!(info.name, bucket);
+
+        let mut reader = store
+            .get_object_reader(&bucket, object, None, Default::default(), &ObjectOptions::default())
+            .await
+            .expect("GET must succeed for a bucket that was not loaded at startup");
+        let mut restored = Vec::new();
+        reader
+            .stream
+            .read_to_end(&mut restored)
+            .await
+            .expect("quorum read should reconstruct the body");
+        assert_eq!(restored, body);
+        drop(reader);
+
+        let object_info = store
+            .get_object_info(&bucket, object, &ObjectOptions::default())
+            .await
+            .expect("HEAD object must succeed at read quorum");
+        assert_eq!(object_info.name, object);
+        assert_eq!(object_info.size, i64::try_from(body.len()).expect("body length fits i64"));
+
+        let listed = store
+            .clone()
+            .list_objects_v2(&bucket, "", None, None, 100, false, None, false)
+            .await
+            .expect("List must succeed at read quorum");
+        assert!(
+            listed.objects.iter().any(|item| item.name == object),
+            "list should include the seed object, got {:?}",
+            listed.objects.iter().map(|item| item.name.clone()).collect::<Vec<_>>()
+        );
+        assert!(
+            metadata_sys::get_in(&store.ctx, &bucket).await.is_ok(),
+            "the cold load should publish authoritative metadata"
+        );
+
+        let write_error = store
+            .put_object(&bucket, "rejected-object", &mut PutObjReader::from_vec(body), &ObjectOptions::default())
+            .await
+            .expect_err("writes must still fail without write quorum");
+        assert!(
+            matches!(
+                write_error,
+                StorageError::InsufficientWriteQuorum(_, _) | StorageError::ErasureWriteQuorum
+            ),
+            "writes must fail with a write-quorum error, got {write_error}"
+        );
+
+        restore_set_disks(&store, set, offline).await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn lazy_metadata_load_rejects_below_read_quorum() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("lazy-below-quorum-{}", Uuid::new_v4().simple());
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("healthy namespace should accept bucket creation");
+
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let set = &store.pools[0].disk_set[0];
+        let offline = take_set_disks_offline(&store, set, &[0, 1, 2]).await;
+
+        let error = metadata_sys::get_on_demand_migration_config_in(&store, &bucket)
+            .await
+            .expect_err("cold metadata load must reject read quorum minus one");
+        assert!(
+            matches!(error, StorageError::ErasureReadQuorum | StorageError::InsufficientReadQuorum(_, _)),
+            "below read quorum must fail with a read-quorum error, got {error}"
+        );
+        assert!(
+            metadata_sys::get_in(&store.ctx, &bucket).await.is_err(),
+            "a below-quorum load must not publish metadata"
+        );
+
+        restore_set_disks(&store, set, offline).await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn lazy_metadata_load_keeps_lock_creation_intent_fail_closed() {
+        let (temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("lazy-lock-intent-{}", Uuid::new_v4().simple());
+        let mut intent = BucketMetadata::new(&bucket);
+        intent.lock_enabled = true;
+        metadata_sys::set_new_bucket_metadata_intent_in(&store.ctx, intent)
+            .await
+            .expect("Object Lock creation intent should persist metadata and its incarnation sidecar");
+
+        let set = &store.pools[0].disk_set[0];
+        let offline = take_set_disks_offline(&store, set, &[0, 1]).await;
+        assert_eq!(
+            store
+                .make_bucket_on_sets(&bucket, &MakeBucketOptions::default())
+                .await
+                .expect_err("physical creation must not commit below write quorum"),
+            StorageError::ErasureWriteQuorum
+        );
+        assert!(!temp_dir.path().join("pool0-disk0").join(&bucket).exists());
+        assert!(!temp_dir.path().join("pool0-disk1").join(&bucket).exists());
+        assert!(temp_dir.path().join("pool0-disk2").join(&bucket).is_dir());
+        assert!(temp_dir.path().join("pool0-disk3").join(&bucket).is_dir());
+
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let error = metadata_sys::get_on_demand_migration_config_in(&store, &bucket)
+            .await
+            .expect_err("Object Lock intent must not become visible at read quorum");
+        assert_eq!(error, StorageError::ErasureWriteQuorum);
+        assert!(
+            metadata_sys::get_in(&store.ctx, &bucket).await.is_err(),
+            "the rejected Object Lock intent must not enter the metadata cache"
+        );
+        assert_eq!(
+            store
+                .get_bucket_info(&bucket, &BucketOptions::default())
+                .await
+                .expect_err("bucket-info read fallback must also reject a pending Object Lock intent"),
+            StorageError::ErasureWriteQuorum
+        );
+
+        restore_set_disks(&store, set, offline).await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn lazy_metadata_load_admits_committed_lock_bucket_at_read_quorum() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("lazy-committed-lock-{}", Uuid::new_v4().simple());
+        store
+            .make_bucket(
+                &bucket,
+                &MakeBucketOptions {
+                    lock_enabled: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("healthy namespace should accept Object Lock bucket creation");
+        let (metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+            .await
+            .expect("read committed Object Lock metadata");
+        assert!(persisted && metadata.bucket_creation_committed);
+
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let set = &store.pools[0].disk_set[0];
+        let offline = take_set_disks_offline(&store, set, &[0, 1]).await;
+
+        metadata_sys::get_on_demand_migration_config_in(&store, &bucket)
+            .await
+            .expect("committed Object Lock metadata should load at read quorum");
+        assert!(
+            metadata_sys::get_in(&store.ctx, &bucket).await.is_ok(),
+            "committed Object Lock metadata should be published"
+        );
+
+        restore_set_disks(&store, set, offline).await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn creation_commit_backfill_runs_under_the_config_write_fence() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("lazy-commit-backfill-{}", Uuid::new_v4().simple());
+        let mut intent = BucketMetadata::new(&bucket);
+        intent.lock_enabled = true;
+        metadata_sys::set_new_bucket_metadata_intent_in(&store.ctx, intent)
+            .await
+            .expect("persist pending Object Lock intent");
+        store
+            .make_bucket_on_sets(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("physical creation should commit at full write quorum");
+
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket)
+            .await
+            .expect("the fenced authority path should backfill the creation commit");
+        let (metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+            .await
+            .expect("read backfilled creation commit");
+        assert!(persisted && metadata.bucket_creation_committed);
+
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let set = &store.pools[0].disk_set[0];
+        let offline = take_set_disks_offline(&store, set, &[0, 1]).await;
+        metadata_sys::get_on_demand_migration_config_in(&store, &bucket)
+            .await
+            .expect("backfilled Object Lock bucket should load at read quorum");
+        restore_set_disks(&store, set, offline).await;
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn enabling_object_lock_on_existing_bucket_establishes_commitment() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("enable-lock-legacy-{}", Uuid::new_v4().simple());
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create a versioned non-lock bucket");
+
+        // Simulate a bucket persisted by a build before the commit field existed.
+        let mut legacy = metadata_sys::get_in(&store.ctx, &bucket)
+            .await
+            .expect("cached metadata")
+            .as_ref()
+            .clone();
+        legacy.lock_enabled = false;
+        legacy.object_lock_config_xml = Vec::new();
+        legacy.object_lock_config = None;
+        legacy.bucket_creation_committed = false;
+        metadata_sys::set_new_bucket_metadata_intent_in(&store.ctx, legacy)
+            .await
+            .expect("persist legacy-shaped metadata");
+
+        let lock_xml = crate::bucket::utils::serialize(&*crate::store::ENABLED_OBJECT_LOCK_CONFIG)
+            .expect("serialize Object Lock configuration");
+        metadata_sys::update_in(&store.ctx, &bucket, "object-lock.xml", lock_xml)
+            .await
+            .expect("enabling Object Lock on an existing bucket must succeed");
+
+        assert!(
+            matches!(
+                metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket)
+                    .await
+                    .expect("read the freshly enabled Object Lock state"),
+                metadata_sys::ObjectLockConfigState::Configured { .. }
+            ),
+            "a healthy cluster must not report ErasureWriteQuorum after enabling Object Lock"
+        );
+        let (metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+            .await
+            .expect("read persisted metadata");
+        assert!(persisted && metadata.bucket_creation_committed);
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn creation_commit_backfill_preserves_concurrent_config_update() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("commit-backfill-pack-{}", Uuid::new_v4().simple());
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create bucket");
+
+        // The cached snapshot on the migrating node still predates another
+        // node's acknowledged configuration update.
+        let mut stale = metadata_sys::get_in(&store.ctx, &bucket)
+            .await
+            .expect("cached metadata")
+            .as_ref()
+            .clone();
+        stale.bucket_creation_committed = false;
+        metadata_sys::set_new_bucket_metadata_intent_in(&store.ctx, stale.clone())
+            .await
+            .expect("persist the stale pre-upgrade snapshot");
+
+        // A newer, still-uncommitted copy carrying an acknowledged Object Lock
+        // update lands on disk after the migrating node read its snapshot.
+        let mut newer = stale.clone();
+        newer.lock_enabled = true;
+        newer.object_lock_config_xml = crate::bucket::utils::serialize(&*crate::store::ENABLED_OBJECT_LOCK_CONFIG)
+            .expect("serialize Object Lock configuration");
+        newer
+            .save_with_store(store.clone())
+            .await
+            .expect("persist the newer configuration");
+
+        // A cold node must re-read the newest authoritative metadata under the
+        // fence instead of rewriting its stale snapshot.
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket)
+            .await
+            .expect("fenced commit migration");
+
+        let (metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+            .await
+            .expect("read persisted metadata");
+        assert!(persisted && metadata.bucket_creation_committed);
+        assert!(
+            metadata.lock_enabled && !metadata.object_lock_config_xml.is_empty(),
+            "the acknowledged Object Lock update must survive the backfill"
+        );
+    }
+
+    #[tokio::test]
+    #[serial]
+    async fn creation_commit_backfill_keeps_recreated_generation() {
+        let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        let bucket = format!("commit-backfill-recreate-{}", Uuid::new_v4().simple());
+        store
+            .make_bucket(&bucket, &MakeBucketOptions::default())
+            .await
+            .expect("create bucket");
+        let old_incarnation = store.bucket_incarnation_id_from_disk(&bucket).await.expect("old incarnation");
+
+        // A node that still holds the pre-recreation snapshot must never let it
+        // outlive the replacement generation.
+        let mut stale = metadata_sys::get_in(&store.ctx, &bucket)
+            .await
+            .expect("cached metadata")
+            .as_ref()
+            .clone();
+        stale.lock_enabled = true;
+        stale.object_lock_config_xml = crate::bucket::utils::serialize(&*crate::store::ENABLED_OBJECT_LOCK_CONFIG)
+            .expect("serialize Object Lock configuration");
+        stale.bucket_creation_committed = false;
+        stale.save_with_store(store.clone()).await.expect("persist stale metadata");
+
+        store
+            .delete_bucket(&bucket, &crate::storage_api_contracts::bucket::DeleteBucketOptions::default())
+            .await
+            .expect("delete bucket");
+        store
+            .make_bucket(
+                &bucket,
+                &MakeBucketOptions {
+                    lock_enabled: true,
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("recreate bucket with Object Lock");
+        let new_incarnation = store.bucket_incarnation_id_from_disk(&bucket).await.expect("new incarnation");
+        assert_ne!(old_incarnation, new_incarnation);
+
+        metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        metadata_sys::get_object_lock_config_state_in(&store.ctx, &bucket)
+            .await
+            .expect("fenced backfill on the replacement generation");
+
+        let (metadata, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&store.ctx, &bucket)
+            .await
+            .expect("read persisted metadata");
+        assert!(persisted);
+        assert_eq!(
+            metadata.bucket_incarnation_id, new_incarnation,
+            "the replacement generation must stay authoritative"
+        );
+        assert!(metadata.bucket_creation_committed);
+    }
+
     #[tokio::test]
     #[serial]
     async fn bucket_info_read_quorum_is_scoped_to_each_erasure_set() {

crates/ecstore/src/store/list_objects.rs

@@ -33,8 +33,7 @@ use crate::core::sets::Sets;
 use crate::disk::error::DiskError;
 use crate::disk::{DiskAPI, DiskInfo, DiskStore, RUSTFS_META_BUCKET, WalkDirOptions};
 use crate::error::{
-    Error, Result, StorageError, is_all_disk_not_found, is_all_not_found, is_all_volume_not_found, is_err_bucket_not_found,
-    to_object_err,
+    Error, Result, StorageError, is_all_disk_not_found, is_all_not_found, is_all_volume_not_found, to_object_err,
 };
 use crate::object_api::{ObjectInfo, ObjectOptions};
 use crate::set_disk::SetDisks;
@@ -4023,32 +4022,6 @@ impl ECStore {
             return Ok(result);
         }
 
-        // Optimization: use get for single object lookup with exact prefix
-        if !opts.prefix.is_empty() && max_keys == 1 && opts.marker.is_none() && !incl_deleted {
-            match self
-                .get_object_info(
-                    &opts.bucket,
-                    &opts.prefix,
-                    &ObjectOptions {
-                        no_lock: true,
-                        ..Default::default()
-                    },
-                )
-                .await
-            {
-                Ok(res) if !res.delete_marker && res.version_purge_status.is_empty() => {
-                    return Ok(ListObjectsInfo {
-                        objects: vec![res],
-                        ..Default::default()
-                    });
-                }
-                Err(err) if is_err_bucket_not_found(&err) => {
-                    return Err(err);
-                }
-                _ => {}
-            };
-        };
-
         let mut list_result = self
             .clone()
             .list_path(&opts)
@@ -5534,31 +5507,6 @@ impl Sets {
         // (notably `forward_past`) — see backlog#1047.
         opts.parse_marker();
 
-        if !opts.prefix.is_empty() && max_keys == 1 && opts.marker.is_none() && !incl_deleted {
-            match self
-                .get_object_info(
-                    &opts.bucket,
-                    &opts.prefix,
-                    &ObjectOptions {
-                        no_lock: true,
-                        ..Default::default()
-                    },
-                )
-                .await
-            {
-                Ok(res) if !res.delete_marker && res.version_purge_status.is_empty() => {
-                    return Ok(ListObjectsInfo {
-                        objects: vec![res],
-                        ..Default::default()
-                    });
-                }
-                Err(err) if is_err_bucket_not_found(&err) => {
-                    return Err(err);
-                }
-                _ => {}
-            };
-        }
-
         let mut list_result = self
             .list_path(&opts)
             .await
@@ -6448,33 +6396,6 @@ impl SetDisks {
         // (notably `forward_past`) — see backlog#1047.
         opts.parse_marker();
 
-        if !opts.prefix.is_empty() && max_keys == 1 && opts.marker.is_none() {
-            match self
-                .get_object_info(
-                    &opts.bucket,
-                    &opts.prefix,
-                    &ObjectOptions {
-                        no_lock: true,
-                        ..Default::default()
-                    },
-                )
-                .await
-            {
-                Ok(res) if !res.delete_marker && res.version_purge_status.is_empty() => {
-                    return Ok(ListObjectsInfo {
-                        objects: vec![res],
-                        ..Default::default()
-                    });
-                }
-                Ok(_) => {}
-                Err(err) => {
-                    if is_err_bucket_not_found(&err) {
-                        return Err(err);
-                    }
-                }
-            };
-        }
-
         let mut list_result = self
             .list_path_result(&opts)
             .await
@@ -9637,6 +9558,88 @@ mod test {
         }
     }
 
+    #[tokio::test]
+    async fn list_objects_exact_prefix_paginates_across_storage_layers() {
+        use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
+        use crate::object_api::{ObjectOptions, PutObjReader};
+        use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
+        use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _};
+
+        let (_dirs, store) = isolated_store_over_temp_disks().await;
+        let bucket = "exact-prefix-pagination-bucket";
+        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+        store
+            .make_bucket(bucket, &MakeBucketOptions::default())
+            .await
+            .expect("exact-prefix pagination bucket should be created");
+        for name in ["a", "ab"] {
+            store.pools[0]
+                .put_object(
+                    bucket,
+                    name,
+                    &mut PutObjReader::from_vec(b"pagination fixture".to_vec()),
+                    &ObjectOptions {
+                        no_lock: true,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("pagination fixture object should be written");
+            let info = store
+                .get_object_info(
+                    bucket,
+                    name,
+                    &ObjectOptions {
+                        no_lock: true,
+                        ..Default::default()
+                    },
+                )
+                .await
+                .expect("exact-prefix fixture must be readable by object lookup");
+            assert!(!info.delete_marker && info.version_purge_status.is_empty());
+        }
+
+        for layer in 0..3 {
+            for (prefix, expected) in [("a", vec!["a", "ab"]), ("ab", vec!["ab"]), ("missing", vec![])] {
+                let mut marker = None;
+                for page in 0..expected.len().max(1) {
+                    let result = match layer {
+                        0 => {
+                            store
+                                .clone()
+                                .list_objects_generic(bucket, prefix, marker.clone(), None, 1, false)
+                                .await
+                        }
+                        1 => {
+                            store.pools[0]
+                                .clone()
+                                .list_objects_generic(bucket, prefix, marker.clone(), None, 1, false)
+                                .await
+                        }
+                        _ => {
+                            store.pools[0].disk_set[0]
+                                .clone()
+                                .list_objects_generic(bucket, prefix, marker.clone(), None, 1, false)
+                                .await
+                        }
+                    }
+                    .expect("exact-prefix page should list successfully");
+                    let names: Vec<_> = result.objects.iter().map(|object| object.name.as_str()).collect();
+                    let expected_page: Vec<_> = expected.get(page).copied().into_iter().collect();
+                    assert_eq!(names, expected_page, "layer {layer}, prefix {prefix}, page {page}");
+                    assert!(result.prefixes.is_empty(), "recursive listing should not return common prefixes");
+                    let has_more = page + 1 < expected.len();
+                    assert_eq!(result.is_truncated, has_more, "layer {layer}, prefix {prefix}, page {page}");
+                    assert_eq!(result.next_marker.is_some(), has_more, "only non-final pages should carry a marker");
+                    if has_more {
+                        assert_ne!(result.next_marker, marker, "pagination marker must advance");
+                    }
+                    marker = result.next_marker;
+                }
+            }
+        }
+    }
+
     #[tokio::test]
     async fn list_objects_hides_pending_version_purge_across_walk_and_exact_prefix() {
         use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
@@ -9685,7 +9688,7 @@ mod test {
             .expect("exact-prefix listing should succeed");
         assert!(
             exact.objects.is_empty(),
-            "exact-prefix max_keys=1 shortcut should hide pending version-purge entries"
+            "exact-prefix max_keys=1 listing should hide pending version-purge entries"
         );
         assert!(exact.prefixes.is_empty());
     }

crates/heal/src/heal/mrf_queue.rs

@@ -1188,6 +1188,14 @@ async fn run_mrf_consumer(
     // must be re-persisted by the next flush before replay can delete the
     // startup anchor.
     runtime.dirty = runtime.queue.depth() > 0 || runtime.partial_writes.depth() > 0;
+    // Publish the first successor immediately instead of waiting for the
+    // periodic tick. Replay is a crash-recovery boundary: a restarted process
+    // must establish its pending successor before any admission retry or
+    // cleanup can make progress. A failed publication leaves `dirty` set, so
+    // the normal timer path still retries it.
+    if runtime.dirty {
+        runtime.flush().await;
+    }
 
     let mut flush_tick = tokio::time::interval(runtime.config.flush_interval);
     flush_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);

crates/rio/src/http_reader.rs

@@ -1065,7 +1065,26 @@ impl HttpReader {
         }
 
         let request_started = Instant::now();
-        let resp = request.send().await.map_err(|e| {
+        // `send()` resolves at the response headers, before the body stall timer
+        // in `poll_read` can run. A restarted peer (or a pooled connection left
+        // half-open when its pod network namespace disappeared) accepts the TCP
+        // connection and then never sends headers. Without this bound the shard
+        // open waits out kernel retransmits, long after the client has given up
+        // on the GET. The body stall budget is the same deadline: a header
+        // black hole is the same failure as a body that stops mid-shard.
+        let send_result = match stall_timeout {
+            Some(stall_timeout) => match time::timeout(stall_timeout, request.send()).await {
+                Ok(result) => result,
+                Err(_elapsed) => {
+                    record_internode_operation_duration(track_internode_metrics, internode_operation, request_started.elapsed());
+                    record_internode_stall_timeout(track_internode_metrics, internode_operation);
+                    record_internode_error(track_internode_metrics, internode_operation);
+                    return Err(body_stalled_error(stall_timeout));
+                }
+            },
+            None => request.send().await,
+        };
+        let resp = send_result.map_err(|e| {
             record_internode_operation_duration(track_internode_metrics, internode_operation, request_started.elapsed());
             record_internode_error(track_internode_metrics, internode_operation);
             record_internode_classified_error(track_internode_metrics, internode_operation, classify_reqwest_error(&e));
@@ -2672,6 +2691,52 @@ mod tests {
         handle.abort();
     }
 
+    /// A peer that accepts the connection and then never sends response headers.
+    /// This is the restarted-pod case: the pooled TCP connection stays open, so
+    /// connect timeout does not fire, and the body stall timer has not started
+    /// because `send()` has not returned. The open itself must fail as
+    /// `BodyStalled` inside the stall budget.
+    #[tokio::test]
+    async fn http_reader_header_stall_fails_open_within_stall_budget() {
+        let listener = match tokio::net::TcpListener::bind("127.0.0.1:0").await {
+            Ok(listener) => listener,
+            Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return,
+            Err(err) => panic!("test listener should bind: {err}"),
+        };
+        let addr = listener.local_addr().expect("listener local address should be available");
+        let app = Router::new().route(
+            "/hang-headers",
+            axum::routing::get(|| async {
+                std::future::pending::<()>().await;
+                StatusCode::OK
+            }),
+        );
+        let server_handle = tokio::spawn(async move {
+            axum::serve(listener, app).await.unwrap();
+        });
+
+        let url = format!("http://{addr}/hang-headers");
+        let stall = Duration::from_millis(50);
+        let opened = tokio::time::timeout(
+            Duration::from_secs(2),
+            HttpReader::new_with_stall_timeout(url, Method::GET, HeaderMap::new(), None, Some(stall)),
+        )
+        .await
+        .expect("header stall must fail the open instead of hanging until the test deadline");
+        let err = match opened {
+            Ok(_reader) => panic!("a peer that never sends headers must fail the reader open"),
+            Err(err) => err,
+        };
+        assert_eq!(err.kind(), io::ErrorKind::TimedOut);
+        let stalled = err
+            .get_ref()
+            .and_then(|source| source.downcast_ref::<BodyStalled>())
+            .expect("header stall should retain the typed body-stalled source");
+        assert_eq!(stalled.timeout, stall);
+
+        server_handle.abort();
+    }
+
     #[tokio::test]
     async fn http_chunk_reader_stall_timeout_retains_typed_source() {
         let state = TestState::default();

crates/scanner/src/raw_page_index.rs

@@ -16,6 +16,14 @@ use serde::{Deserialize, Serialize};
 use sha2::{Digest as _, Sha256};
 use thiserror::Error;
 
+mod writer;
+pub(crate) use writer::RawEnumerationPageWriter;
+
+#[cfg(test)]
+thread_local! {
+    pub(crate) static RAW_PAGE_DIGEST_ENTRIES: std::cell::Cell<usize> = const { std::cell::Cell::new(0) };
+}
+
 const RAW_PAGE_INDEX_VERSION: u16 = 1;
 const RAW_PAGE_ENTRY_MAX_BYTES: usize = 16 * 1024;
 
@@ -145,6 +153,13 @@ impl RawEnumerationPageIndex {
         }
     }
 
+    pub(crate) fn parent(&self) -> Option<&str> {
+        match &self.state {
+            RawEnumerationPageIndexState::Unsupported => None,
+            RawEnumerationPageIndexState::Supported(inner) => Some(&inner.parent),
+        }
+    }
+
     pub fn status(&self) -> RawEnumerationPageOwnerStatus {
         match &self.state {
             RawEnumerationPageIndexState::Unsupported => RawEnumerationPageOwnerStatus::Unsupported,
@@ -295,25 +310,7 @@ impl RawEnumerationPageIndex {
             return Err(RawEnumerationPageIndexError::StaleGeneration);
         }
         let entries_start = inner.validated_committed_entries()?.len();
-        let Some(building) = inner.building.as_ref() else {
-            return Err(RawEnumerationPageIndexError::EmptyCommit);
-        };
-        if building.entries.is_empty() {
-            return Err(RawEnumerationPageIndexError::EmptyCommit);
-        }
-        building.validate(
-            u64::try_from(inner.pages.len()).unwrap_or(u64::MAX),
-            u64::try_from(entries_start).unwrap_or(u64::MAX),
-            inner.page_entry_limit,
-        )?;
-        let Some(building) = inner.building.take() else {
-            return Err(RawEnumerationPageIndexError::EmptyCommit);
-        };
-        let page = RawEnumerationPage::new(&inner.parent, building);
-        inner.complete = page.terminal;
-        inner.pages.push(page.clone());
-        inner.generation = inner.generation.saturating_add(1);
-        Ok(page)
+        inner.commit_building_page(entries_start)
     }
 
     pub fn page(&self, page_index: u64, expected_digest: [u8; 32]) -> Result<&RawEnumerationPage, RawEnumerationPageIndexError> {
@@ -349,6 +346,30 @@ impl RawEnumerationPageIndex {
 }
 
 impl RawEnumerationPageIndexInner {
+    // Callers validate the committed prefix at the restore/API boundary. The
+    // live writer maintains its entry count without revisiting that prefix.
+    fn commit_building_page(&mut self, entries_start: usize) -> Result<RawEnumerationPage, RawEnumerationPageIndexError> {
+        let Some(building) = self.building.as_ref() else {
+            return Err(RawEnumerationPageIndexError::EmptyCommit);
+        };
+        if building.entries.is_empty() {
+            return Err(RawEnumerationPageIndexError::EmptyCommit);
+        }
+        building.validate(
+            u64::try_from(self.pages.len()).unwrap_or(u64::MAX),
+            u64::try_from(entries_start).unwrap_or(u64::MAX),
+            self.page_entry_limit,
+        )?;
+        let Some(building) = self.building.take() else {
+            return Err(RawEnumerationPageIndexError::EmptyCommit);
+        };
+        let page = RawEnumerationPage::new(&self.parent, building);
+        self.complete = page.terminal;
+        self.pages.push(page.clone());
+        self.generation = self.generation.saturating_add(1);
+        Ok(page)
+    }
+
     fn status(&self) -> RawEnumerationPageOwnerStatus {
         let indexed_entries = u64::try_from(self.indexed_entries()).unwrap_or(u64::MAX);
         if let Some(building) = &self.building {
@@ -528,6 +549,8 @@ fn entry_sets_match(left: &[String], right: &[String]) -> bool {
 }
 
 fn raw_page_digest(parent: &str, building: &RawEnumerationPageBuilder) -> [u8; 32] {
+    #[cfg(test)]
+    RAW_PAGE_DIGEST_ENTRIES.with(|count| count.set(count.get().saturating_add(building.entries.len())));
     let mut digest = Sha256::new();
     update_digest(&mut digest, b"version", &RAW_PAGE_INDEX_VERSION.to_le_bytes());
     update_digest(&mut digest, b"parent", parent.as_bytes());

crates/scanner/src/raw_page_index/writer.rs

@@ -0,0 +1,149 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::{
+    RawEnumerationPageBuilder, RawEnumerationPageIndex, RawEnumerationPageIndexError, RawEnumerationPageIndexInner,
+    RawEnumerationPageIndexState, owner_entry_is_valid,
+};
+use std::collections::{BTreeSet, HashSet};
+
+/// Exclusive, validated owner of an index during one directory enumeration.
+/// The auxiliary sets are never serialized: restored indexes must cross the
+/// validation boundary again before any pages can be trusted or extended.
+pub(crate) struct RawEnumerationPageWriter {
+    inner: RawEnumerationPageIndexInner,
+    indexed: HashSet<String>,
+    pending: BTreeSet<String>,
+    observed_complete: HashSet<String>,
+    complete_source_changed: bool,
+    observations: usize,
+    revalidate_after_entries: usize,
+    committed_entries: usize,
+}
+
+impl RawEnumerationPageWriter {
+    pub(crate) fn new(index: RawEnumerationPageIndex) -> Result<Self, RawEnumerationPageIndexError> {
+        let RawEnumerationPageIndexState::Supported(inner) = index.state else {
+            return Err(RawEnumerationPageIndexError::Unsupported);
+        };
+        if inner.parent.is_empty() || inner.page_entry_limit == 0 {
+            return Err(RawEnumerationPageIndexError::CorruptIndex);
+        }
+        let entries = inner.validated_indexed_entries()?;
+        let entry_count = entries.len();
+        let indexed: HashSet<_> = entries.into_iter().collect();
+        if indexed.len() != entry_count
+            || inner.pages.iter().any(|page| page.entries.len() > inner.page_entry_limit)
+            || (inner.complete && inner.building.is_some())
+            || inner.pages.last().is_some_and(|page| page.terminal != inner.complete)
+        {
+            return Err(RawEnumerationPageIndexError::CorruptIndex);
+        }
+        let committed_entries = entry_count.saturating_sub(inner.building.as_ref().map_or(0, |page| page.entries.len()));
+        Ok(Self {
+            inner,
+            indexed,
+            pending: BTreeSet::new(),
+            observed_complete: HashSet::new(),
+            complete_source_changed: false,
+            observations: 0,
+            revalidate_after_entries: entry_count,
+            committed_entries,
+        })
+    }
+
+    pub(crate) fn indexed_entry_count(&self) -> usize {
+        self.indexed.len()
+    }
+
+    pub(crate) fn record_entry(&mut self, entry: &str) -> Result<(), RawEnumerationPageIndexError> {
+        if !owner_entry_is_valid(entry) {
+            return Err(RawEnumerationPageIndexError::InvalidEntry);
+        }
+        self.observations = self.observations.saturating_add(1);
+        if self.inner.complete || self.inner.building.as_ref().is_some_and(|page| page.terminal) {
+            if self.indexed.contains(entry) {
+                self.observed_complete.insert(entry.to_owned());
+            } else {
+                self.complete_source_changed = true;
+            }
+        }
+        if self.inner.complete {
+            if self.observations >= self.revalidate_after_entries
+                && (self.observed_complete.len() != self.indexed.len() || self.complete_source_changed)
+            {
+                return Err(RawEnumerationPageIndexError::IdentityMismatch);
+            }
+            return Ok(());
+        }
+        if !self.indexed.contains(entry) {
+            self.pending.insert(entry.to_owned());
+        }
+        // A restored directory can be enumerated in a different order. Keep
+        // the old observation floor and append at most one new entry per call,
+        // in the same order as normalization of the cumulative source.
+        if self.observations < self.revalidate_after_entries {
+            return Ok(());
+        }
+        if self
+            .inner
+            .building
+            .as_ref()
+            .is_some_and(|page| page.entries.len() >= self.inner.page_entry_limit)
+        {
+            return self.commit_page();
+        }
+        if let Some(entry) = self.pending.pop_first() {
+            let building = self.inner.building.get_or_insert_with(|| RawEnumerationPageBuilder {
+                page_index: u64::try_from(self.inner.pages.len()).unwrap_or(u64::MAX),
+                entries_start: u64::try_from(self.committed_entries).unwrap_or(u64::MAX),
+                entries: Vec::new(),
+                terminal: false,
+            });
+            self.indexed.insert(entry.clone());
+            building.entries.push(entry);
+            building.entries.sort();
+            building.terminal = false;
+            self.inner.generation = self.inner.generation.saturating_add(1);
+        }
+        if self
+            .inner
+            .building
+            .as_ref()
+            .is_some_and(|page| page.terminal || page.entries.len() >= self.inner.page_entry_limit)
+        {
+            self.commit_page()?;
+        }
+        Ok(())
+    }
+
+    fn commit_page(&mut self) -> Result<(), RawEnumerationPageIndexError> {
+        self.inner.commit_building_page(self.committed_entries)?;
+        self.committed_entries = self.indexed.len();
+        Ok(())
+    }
+
+    pub(crate) fn checkpoint(&self) -> Result<RawEnumerationPageIndex, RawEnumerationPageIndexError> {
+        let mut inner = self.inner.clone();
+        if inner.building.is_some() {
+            inner.commit_building_page(self.committed_entries)?;
+        }
+        Ok(RawEnumerationPageIndex {
+            state: RawEnumerationPageIndexState::Supported(inner),
+        })
+    }
+}
+
+#[cfg(test)]
+mod tests;

crates/scanner/src/raw_page_index/writer/tests.rs

@@ -0,0 +1,223 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::*;
+use crate::raw_page_index::RAW_PAGE_DIGEST_ENTRIES;
+
+fn checkpoint(mut index: RawEnumerationPageIndex) -> RawEnumerationPageIndex {
+    if let RawEnumerationPageIndexState::Supported(inner) = &index.state
+        && inner.building.is_some()
+    {
+        index
+            .commit_building_page(index.generation().expect("supported index generation"))
+            .expect("reference checkpoint must commit its partial page");
+    }
+    index
+}
+
+fn compare_with_cumulative_ingestion(initial: RawEnumerationPageIndex, source: &[&str]) {
+    let mut reference = initial.clone();
+    let floor = reference.indexed_entries().expect("reference entries").len();
+    let mut writer = RawEnumerationPageWriter::new(initial).expect("validated writer");
+    let mut observed = Vec::new();
+    for entry in source {
+        observed.push((*entry).to_owned());
+        let expected = if observed.len() < floor {
+            Ok(())
+        } else {
+            reference
+                .ingest_partial_owner_entries(observed.clone(), 1, reference.generation().expect("reference generation"))
+                .and_then(|outcome| {
+                    if outcome.ready_to_commit {
+                        reference.commit_building_page(reference.generation().expect("commit generation"))?;
+                    }
+                    Ok(())
+                })
+        };
+        let actual = writer.record_entry(entry);
+        assert_eq!(actual, expected, "observation {observed:?}");
+        if actual.is_err() {
+            return;
+        }
+        let expected = checkpoint(reference.clone());
+        let actual = writer.checkpoint().expect("incremental checkpoint");
+        assert_eq!(actual, expected, "page identity and generation at {observed:?}");
+        assert_eq!(
+            rmp_serde::to_vec(&actual).expect("encode incremental index"),
+            rmp_serde::to_vec(&expected).expect("encode reference index"),
+            "persisted format must remain unchanged"
+        );
+        assert_eq!(writer.indexed_entry_count(), actual.indexed_entries().expect("validate output").len());
+    }
+}
+
+#[test]
+fn raw_enumeration_writer_matches_cumulative_pages_with_duplicates_and_restarts() {
+    for limit in [1, 2, 3, 128] {
+        let source = ["c", "a", "b", "b", "f", "d", "e", "g"];
+        let empty = RawEnumerationPageIndex::new("bucket/metadata", limit).expect("empty index");
+        compare_with_cumulative_ingestion(empty.clone(), &source);
+        let mut writer = RawEnumerationPageWriter::new(empty).expect("writer");
+        for entry in source {
+            writer.record_entry(entry).expect("entry before interruption");
+            let saved = writer.checkpoint().expect("save interrupted enumeration");
+            let encoded = rmp_serde::to_vec(&saved).expect("encode checkpoint");
+            let restored = rmp_serde::from_slice(&encoded).expect("restore checkpoint");
+            compare_with_cumulative_ingestion(restored, &["h", "g", "f", "e", "d", "c", "b", "a", "i", "j"]);
+        }
+    }
+}
+
+#[test]
+fn raw_enumeration_writer_preserves_restored_building_and_terminal_pages() {
+    for complete in [false, true] {
+        for budget in [1, 2] {
+            let mut index = RawEnumerationPageIndex::new("bucket", 2).expect("index");
+            let source = ["a".to_owned(), "b".to_owned()];
+            if complete {
+                index.ingest_owner_entries(source, budget, 0).expect("complete source");
+            } else {
+                index.ingest_partial_owner_entries(source, budget, 0).expect("partial source");
+            }
+            compare_with_cumulative_ingestion(index.clone(), &["b", "a", "a", "c", "d"]);
+            compare_with_cumulative_ingestion(checkpoint(index), &["b", "a", "a", "c", "d"]);
+        }
+    }
+}
+
+#[test]
+fn raw_enumeration_writer_keeps_identity_check_when_terminal_builder_commits() {
+    for limit in [2, 3] {
+        let mut index = RawEnumerationPageIndex::new("bucket", limit).expect("index");
+        index
+            .ingest_owner_entries(["a".to_owned(), "b".to_owned()], 2, 0)
+            .expect("terminal building page");
+        compare_with_cumulative_ingestion(index.clone(), &["b", "a", "b", "c"]);
+        compare_with_cumulative_ingestion(index, &["foreign", "a", "b", "c"]);
+    }
+}
+
+#[test]
+fn raw_enumeration_writer_rejects_complete_source_drift_after_observation_floor() {
+    let mut index = RawEnumerationPageIndex::new("bucket", 2).expect("index");
+    index
+        .ingest_owner_entries(["a".to_owned(), "b".to_owned()], 2, 0)
+        .expect("complete source");
+    let index = checkpoint(index);
+    for source in [["foreign", "a"], ["a", "foreign"], ["a", "a"], ["b", "a"]] {
+        compare_with_cumulative_ingestion(index.clone(), &source);
+    }
+}
+
+#[test]
+fn raw_enumeration_writer_rejects_corruption_at_restore() {
+    let mut writer = RawEnumerationPageWriter::new(RawEnumerationPageIndex::new("bucket", 2).expect("index")).expect("writer");
+    for entry in ["a", "b", "c"] {
+        writer.record_entry(entry).expect("valid entry");
+    }
+    let valid = writer.checkpoint().expect("checkpoint");
+    for case in 0..7 {
+        let mut corrupt = valid.clone();
+        let RawEnumerationPageIndexState::Supported(inner) = &mut corrupt.state else {
+            panic!("supported checkpoint");
+        };
+        match case {
+            0 => inner.pages[0].digest[0] ^= 1,
+            1 => inner.pages[1].entries_start = 0,
+            2 => inner.pages[1] = inner.pages[0].clone(),
+            3 => inner.page_entry_limit = 0,
+            4 => inner.complete = true,
+            5 => inner.parent.clear(),
+            6 => {
+                inner.building = Some(RawEnumerationPageBuilder {
+                    page_index: 2,
+                    entries_start: 3,
+                    entries: vec!["a".to_owned()],
+                    terminal: false,
+                });
+            }
+            _ => unreachable!(),
+        }
+        let bytes = rmp_serde::to_vec(&corrupt).expect("encode damaged checkpoint");
+        let restored = rmp_serde::from_slice(&bytes).expect("decode untrusted checkpoint");
+        assert!(
+            matches!(RawEnumerationPageWriter::new(restored), Err(RawEnumerationPageIndexError::CorruptIndex)),
+            "corruption case {case} must not become trusted runtime state"
+        );
+    }
+}
+
+#[test]
+fn raw_enumeration_writer_rejects_invalid_names_without_publishing_them() {
+    for entry in ["", ".", "..", "nested/name", &"x".repeat(16 * 1024 + 1)] {
+        let mut writer =
+            RawEnumerationPageWriter::new(RawEnumerationPageIndex::new("bucket", 128).expect("index")).expect("writer");
+        let before = writer.checkpoint().expect("empty checkpoint");
+        assert_eq!(writer.record_entry(entry), Err(RawEnumerationPageIndexError::InvalidEntry));
+        assert_eq!(writer.checkpoint().expect("checkpoint after rejection"), before);
+        assert_eq!(writer.indexed_entry_count(), 0);
+    }
+}
+
+#[test]
+fn raw_enumeration_writer_hashes_only_new_pages_after_restore() {
+    let count = 4096;
+    let mut writer = RawEnumerationPageWriter::new(RawEnumerationPageIndex::new("bucket", 128).expect("index")).expect("writer");
+    RAW_PAGE_DIGEST_ENTRIES.set(0);
+    for i in (0..count).rev() {
+        writer.record_entry(&format!("entry-{i:06}")).expect("append entry");
+    }
+    assert_eq!(
+        RAW_PAGE_DIGEST_ENTRIES.get(),
+        count,
+        "each committed entry is hashed once during enumeration"
+    );
+    let saved = writer.checkpoint().expect("checkpoint");
+    assert_eq!(saved.indexed_entries().expect("validated checkpoint").len(), count);
+    let mut resumed = RawEnumerationPageWriter::new(saved).expect("restore large checkpoint");
+    RAW_PAGE_DIGEST_ENTRIES.set(0);
+    for i in 0..count * 2 {
+        resumed.record_entry(&format!("entry-{i:06}")).expect("resume and append");
+    }
+    assert_eq!(resumed.indexed_entry_count(), count * 2);
+    assert_eq!(
+        RAW_PAGE_DIGEST_ENTRIES.get(),
+        count,
+        "restored pages must not be rehashed for each entry or commit"
+    );
+}
+
+#[test]
+fn raw_enumeration_writer_keeps_snapshot_independent_of_later_appends() {
+    let mut writer = RawEnumerationPageWriter::new(RawEnumerationPageIndex::new("bucket", 128).expect("index")).expect("writer");
+    for i in 0..127 {
+        writer.record_entry(&format!("entry-{i:03}")).expect("append before boundary");
+    }
+    let partial = writer.checkpoint().expect("partial page checkpoint");
+    writer.record_entry("entry-127").expect("full page");
+    let full = writer.checkpoint().expect("full page checkpoint");
+    writer.record_entry("entry-128").expect("next page");
+    assert_eq!(partial.indexed_entries().expect("old partial snapshot").len(), 127);
+    assert_eq!(full.indexed_entries().expect("old full snapshot").len(), 128);
+    assert_eq!(
+        writer
+            .checkpoint()
+            .expect("new snapshot")
+            .indexed_entries()
+            .expect("new entries")
+            .len(),
+        129
+    );
+    assert_ne!(partial, full);
+}